SSPHub https://ssphub-test.netlify.app/ Network of data scientists working for the French administration quarto-1.9.37 Tue, 17 Mar 2026 00:00:00 GMT sndsTools, a R package for extracting healthcare utilization in SNDS health data https://ssphub-test.netlify.app/project/2026_sndsTools/

Project summary

sndsTools, an R package for extracting healthcare utilization in SNDS health data
Project details sndsTools is an R package designed to simplify the extraction of healthcare utilization from the Système National de Données de Santé (SNDS) health data hosted on the Health Insurance portal. It streamlines the data extraction steps from the SNDS for data used in the majority of SNDS studies.
Players Insee, Institut du Cerveau, Inria, AP-HM
Project results The R package sndsTools is in production and available for download from GitHub .
Products and project documentation R package documentation site
Project code - The code is available on GitHub https://github.com/SNDStoolers/sndsTools

Similar projects

]]> Insee package data extraction R administrative data https://ssphub-test.netlify.app/project/2026_sndsTools/ Tue, 17 Mar 2026 00:00:00 GMT Comparison of forecasts between nowcasting and bottom-up approach https://ssphub-test.netlify.app/project/2025_nowcasting/

Project summary

Comparison of continuous GDP forecasting methods with a bottom-up method
Project details INSEE bases its GDP growth forecasts on a so-called bottom-up model, which involves reproducing the mechanism for constructing quarterly accounts. The question arises, however, as to the performance of such an approach to forecasting GDP compared with that of a direct method such as nowcasting. A direct approach makes it possible to forecast GDP growth directly from business cycle series, without using quarterly accounts. In order to study the relative performance of the two approaches, we developed a new bottom-up forecasting model for GDP growth in France, inspired by the Federal Reserve Bank of Atlanta’s “GDPNow” model.
Players Insee
Project results At the beginning of the quarter, the direct approach performs slightly better than the bottom-up approach: few quantitative indicators are available at that time, and the added value represented by the use of a complete accounting framework is limited compared with a direct model, which is naturally more parsimonious. From the end of the second month onwards, on the other hand, more quantitative indicators become available, and the bottom-up approach performs slightly better than the direct approach estimated in this study, particularly at the time of publication of the Business Review. Finally, the difference in performance between the two approaches is clear between the end of the quarter and the publication of the first estimates of the quarterly accounts thirty days later: during this period, it is much better to use the information available via a bottom-up approach rather than a direct approach.
Project products and documentation - How do the direct and bottom-up approaches compare in forecasting GDP for the current quarter? INSEE Economic Outlook - September 2025

Similar projects

]]> study nowcasting forecasts Insee https://ssphub-test.netlify.app/project/2025_nowcasting/ Mon, 01 Sep 2025 00:00:00 GMT Use of banking data for INSEE economic forecasts https://ssphub-test.netlify.app/project/2025_comptes_bancaires_conj/

Project summary

Use of banking data for INSEE economic forecasts
Project details Bank account data is an advanced source of information on household consumption and savings in 2020, at both microeconomic and infra-annual levels. Using anonymised data made available by Crédit Mutuel Alliance Fédérale, it is possible to study how the health crisis may have affected the financial situation of the bank’s household customers, depending on their income level, age or socio-professional category.
The banking data used does not allow us to identify household incomes directly, but we can nevertheless use it to deduce an approximation, thanks to all the transfers and cheques entering the accounts. These inflows fell during the first containment period before rebounding in June. On average, the second containment would not have caused a drop in these incoming flows.
Players Insee
Project results These studies are an extension of the work carried out on households using bank account data.

During the two confinements of 2020, all the household groups studied, whatever their income level, are thought to have reduced their consumption, which has refocused, particularly in April, on essential goods. Households that consumed the most before the crisis, essentially executives and high-income earners, would therefore have reduced their consumption more.
The fall in consumption triggered a surge in savings, which fed into household current accounts and passbook accounts. Household gross financial assets (liquid savings, securities accounts and life insurance, excluding loans) are estimated to have risen sharply in 2020. This increase can be seen across all household groups, whatever their level of financial wealth. It is higher in euros for households with high financial wealth, who have been able to save more by reducing their consumption. Households with low wealth also put money aside, particularly during the first period of confinement. However, the amounts involved for these households, generally a few tens or hundreds of euros, remain small even though they represent a significant proportion of their initial wealth. Among working households, some would have been more affected than others by a fall in their income and would therefore have increased their savings less: this is the case for craftsmen and shopkeepers, or employees in the private sector, in contrast to those in the public sector.

The analysis conducted for the June 2025 economic outlook shows that changes in income, consumption, and savings rate aggregates derived from these bank account data are consistent with those in the national accounts. While the average savings rate increased in almost all groups between 2023 and 2024, it was among the oldest that the increase in the savings rate in 2023 and 2024 was the strongest: people aged 65 and over accounted for around two-thirds of the increase in the savings rate between 2023 and 2024.
Project products and documentation - In 2020, the drop in consumption fuelled savings, with a particular increase in the financial wealth of the most well-off: some results obtained by analysing banking data Insee Economic Outlook - March 2021 ;
- In 2024, in 2024, the income of retired customers of La Banque Postale rose sharply but their consumption did not keep pace, which contributed to two-thirds of the increase in the savings ratio Insee Economic Outlook - June 2025

Similar projects

An assessment of cross-border tobacco purchases and associated tax losses in France
experiment
mobile phone data
credit card data
Insee

Using the closing of borders in 2020 as a natural experiment to measure cross-border tobacco purchases

1 Jan 2024

Methodological work on the Family Budget survey
automatic coding
data extraction
scanner data

Modernisation of the family budget survey using automatic classification tools

1 Jan 2022

Using credit card data and mobile phone data to forecast economic activity
forecasts
experiment
Insee
credit card data
mobile phone data

The 2020 health crisis required a review of forecasting processes to be more responsive to events. INSEE used credit card transaction data to forecast economic activity.

1 Dec 2020

What do the electricity production and consumption data say about economic activity during the containment period?
Insee
forecasts
private data
experiment

Using electricity production and consumption data to forecast economic activity

1 Dec 2020

Population movements around the March 2020 containment using mobile phone network operators data
datavisualisation
machine learning
Insee
experiment
open-data
mobile phone data

INSEE has had access to mobile telephony data as part of the monitoring of the 2020 health crisis. These data were used to produce the following statistics on population…

1 Nov 2020

Classification of checkout data using machine learning
Python
automatic coding
scanner data
COICOP
CPI
in production

Using machine learning to classify scanner data in the COICOP nomenclature to calculate the CPI

1 Jan 2020

Urban segregation: insights from mobile phone data
Insee
experiment
mobile phone data
administrative data

Merging administrative data and MNO data to estimate urban segregation at a local level

1 Jan 2018
No matching items

Other studies using bank data

Insee has also carried out other studies using bank account data. They are available on INSEE website:

]]> Insee forecasts bank account data experiment https://ssphub-test.netlify.app/project/2025_comptes_bancaires_conj/ Sun, 01 Jun 2025 00:00:00 GMT An assessment of cross-border tobacco purchases and associated tax losses in France https://ssphub-test.netlify.app/project/2024_cb_mno_tabac/

Project summary

An assessment of cross-border tobacco purchases and associated tax losses in France
Project details Smoking is a major public health problem, causing many preventable diseases worldwide. In recent decades, increasing the price of tobacco has become the main strategy used by governments to reduce smoking. However, price differences between certain neighbouring countries are likely to limit the effectiveness of this measure by allowing some consumers to buy tobacco at a lower price in a neighbouring country. Although the problem of cross-border shopping is not new, the extent of the phenomenon remains poorly understood and is still the subject of regular debate. This study contributes to its assessment in France by exploiting an unprecedented natural experiment : the closure of land borders between March 2020 and June 2020 as part of the fight against the Covid-19 pandemic.
Players Insee
Project results Results show that the closure of the borders led to a 9.5 % surplus in tobacco purchases in mainland France compared to the counterfactual situation in which the borders had remained open. This result probably underestimates cross-border purchases. In fact, some tobacco consumption abroad may have continued during the first lockdown, as the borders were not completely closed, in particular for cross-border workers. Extrapolating the consumption observed in the rest of the country to border regions with identical characteristics, the revenue generated in France would be about 13.5 % higher if there were no cheaper alternatives abroad.
Project products and documentation - An assessment of cross-border tobacco purchases and associated tax losses in France Insee Working Papers No. 2024-06, April 2024

Similar projects

Use of banking data for INSEE economic forecasts
Insee
forecasts
bank account data
experiment
1 Jun 2025

Methodological work on the Family Budget survey
automatic coding
data extraction
scanner data

Modernisation of the family budget survey using automatic classification tools

1 Jan 2022

Using credit card data and mobile phone data to forecast economic activity
forecasts
experiment
Insee
credit card data
mobile phone data

The 2020 health crisis required a review of forecasting processes to be more responsive to events. INSEE used credit card transaction data to forecast economic activity.

1 Dec 2020

What do the electricity production and consumption data say about economic activity during the containment period?
Insee
forecasts
private data
experiment

Using electricity production and consumption data to forecast economic activity

1 Dec 2020

Population movements around the March 2020 containment using mobile phone network operators data
datavisualisation
machine learning
Insee
experiment
open-data
mobile phone data

INSEE has had access to mobile telephony data as part of the monitoring of the 2020 health crisis. These data were used to produce the following statistics on population…

1 Nov 2020

Classification of checkout data using machine learning
Python
automatic coding
scanner data
COICOP
CPI
in production

Using machine learning to classify scanner data in the COICOP nomenclature to calculate the CPI

1 Jan 2020

Urban segregation: insights from mobile phone data
Insee
experiment
mobile phone data
administrative data

Merging administrative data and MNO data to estimate urban segregation at a local level

1 Jan 2018
No matching items
]]> experiment mobile phone data credit card data Insee https://ssphub-test.netlify.app/project/2024_cb_mno_tabac/ Mon, 01 Jan 2024 00:00:00 GMT scanR, an application for monitoring France’s research and innovation landscape https://ssphub-test.netlify.app/project/2024_scanr/

Project summary

Explore the world of French Research and Innovation with scanR
Project details scanR is a web application to help characterise public structures (research units of all types, public institutions) and private structures (companies) involved in research and innovation in France. scanR also helps to identify the the direction of the work of researchers active in France since the early 1990s.

scanR combines structured data under an open licence (publications and theses, participation in collaborative research projects, spin-offs, patents, etc.) and open information extracted directly from the websites of research and innovation players. This information comes from around 13 different sources (theses.fr, the open science barometer HAL, European Commission, INPI, ANR, European Patent Office, etc.) and is obtained by webscraping, pdf-mining or using APIs. The resources are then identified and linked together, in particular using AI methods, and then enriched.
A feedback loop has been set to improve the quality of the data produced. It allows for requesting data corrections from the scanR site.
An engine based on ElasticSearch allows you to search for themes, structures or authors.
Finally, it is now possible to visually display interactions between different structures or themes using a network analysis.
Players Statistical Service of the Ministry of Higher Education and Research (SIES)
Project results The scanR project has been in production since 2016 and records around 50,000 monthly visits.

- The project provides files by structure and authors presenting their organisation, activity, areas of research and source of funding.
- It also offers a search engine on research structures, authors, funding, publications and patents in France.
- It also provides tools for analysing results including graph visualisations.

The project also provides several APIs to retrieve data and data.
Project products and documentation scanR is available at scanr.enseignementsup-recherche.gouv.fr. This site includes :
- the documentation to access the four APIs available ;
- the various data sources used for the project.

Methodology and technical details :
- Detailed technical presentation of scanR scanR - Explore public data on French research and innovation euroCRIS conference, November 2024;
- Mapping scientific communities at scale
Project code - The code is available on GitHub https://github.com/dataesr/scanr-ui

Similar projects

]]> API in production open-data SIES datavisualisation ElasticSearch https://ssphub-test.netlify.app/project/2024_scanr/ Mon, 01 Jan 2024 00:00:00 GMT Visualisations of SARS-Cov2 test data https://ssphub-test.netlify.app/project/2022_drees_dataviz/
Automatic publication and data-visualisation of Covid-19 test data
Project details In recent years, the DREES (the statistical department of the Ministry of Health), has been making available, as part of its strategy open-data as part of its open-data strategy, is making available interactive visualisations of data on a variety of topics (Covid, of course, but also the number of healthcare professionals, pensions and retirement ages, etc.). These visualisations are based on Shiny and give users the freedom to play with the data and update exploratory graphs.
Between October 2020 and June 2023,3 DREES provided data on the volume and validation times for RT-PCR and antigen tests on a weekly basis. These results are based on data from the SI-DEP information system.
Players DREES
Project results The project published automatically updated information on Covid-19 tests and deadlines between 2020 and mid-2023. The data is still available on the https://drees.shinyapps.io/delais_test_app/ but has not been updated since June 2023.
]]> R datavisualisation experiment stopped DREES open-data https://ssphub-test.netlify.app/project/2022_drees_dataviz/ Thu, 01 Jun 2023 00:00:00 GMT Doremifasol https://ssphub-test.netlify.app/project/2023_doremifasol/

Project summary

Package to retrieve data from the Insee website
Project details doremifasol (data with R made available by Insee and easily retrievable in French) is a R package mainly aiming at showing off data available on Insee’s website, helping the user to put them on stage and extract the information they carry. So it is about analysing data, creating maps, quantifying phenomenons and in general using the data without the painful effort to retrieve them on the website, as well as import them into R’s memory. The name of the package stands for the five first notes of music, and pushing the metaphore, underlines its aim at helping the users to easily pratice their solfège in .
Players Insee
Project results The DoReMiFaSol package is open source and can be downloaded from GitHub. The documentation is also published here.
Project code - Code is available on GitHub repo https://github.com/InseeFrLab/DoReMIFaSol

Similar projects

]]> package R in production Insee https://ssphub-test.netlify.app/project/2023_doremifasol/ Sun, 01 Jan 2023 00:00:00 GMT pynsee, a Python package for retrieving INSEE data https://ssphub-test.netlify.app/project/2023_pynsee/

Project summary

pynsee, a Python package to retrieve data from INSEE
Project details pynsee provides rapid access to more than 150,000 macroeconomic series, around ten local data sets, numerous sources available on insee.fr as well as key metadata and a SIRENE database containing data on all French businesses. Visit the API details page.
This package is a contribution to reproducible research and to the transparency of public data. It benefits from developments made by the teams working on APIs at INSEE and IGN.
Players Insee
Project results The pynsee package is in production and can be downloaded from GitHub. The documentation is also published here.
Project code - Repo GitHub
]]> in production Insee package Python https://ssphub-test.netlify.app/project/2023_pynsee/ Sun, 01 Jan 2023 00:00:00 GMT Using satellite images for official statistics https://ssphub-test.netlify.app/project/2022_satellites/

Project summary

Using satellite images for official statistics
Project details At the end of 2022, an experimental working group on the use of satellite data for official statistics was launched. The aim of this initiative was to improve the organisation of mapping surveys in overseas departments particularly in French Guiana and Mayotte. These surveys are used to update the individual/housing register (RIL) each year, and make it easier to identify new spontaneous or temporary housing areas in advance of the surveys. Algorithms applied to satellite photographs are used to identify such areas, which in a given year will require more surveyors than in the previous year.
The project is presented in more detail here.
Actors Insee (DG and Réunion-Mayotte Interregional Directorate)
Project results The results of the project are presented in more detail on the dedicated website. The experimentation phase ended with:
- The transmission of building evolution statistics in Mayotte. These statistics were used to organise the 2024 complementary mapping survey in Mayotte;
- The development of a visualisation application integrating background maps built from Pleiades tiles and buildings detected on these tiles using our algorithms, as well as change detection layers;
- The creation of thePython package for processing satellite images Astrovision
Project code - https://github.com/InseeFrLab/astrovision : Python package package for working with satellite data:
- https://inseefrlab.github.io/satellite-images-webapp/ code of theviewing application
- https://github.com/InseeFrLab/satellite-images-preprocess code of data pre-processing
- https://github.com/InseeFrLab/satellite-images-train code for themodel training
- https://github.com/InseeFrLab/satellite-images-inference code for theinference

Documents relating to the Insee satellite data project

No matching items

Similar projects

]]> Python package deep learning satellite imagery https://ssphub-test.netlify.app/project/2022_satellites/ Sat, 01 Oct 2022 00:00:00 GMT GDP Tracker: a tool for continuous economic forecasting https://ssphub-test.netlify.app/project/2019_gdp_tracker/

Project summary

Nowcasting GDP with GDP Tracker
Details of the project The project GDP Tracker consists of building a continuous economic forecasting tool, fed by all the recent economic indicators available at any given time. t. The aim of this project is to make the best possible use of the most recent sources to estimate the dynamics of the major macroeconomic aggregates over the last few weeks. These sources are diverse in nature: business and consumer surveys, quarterly national accounts, hard-copy statistical data, as well as data from less traditional sources (internet searches, etc.).

Using this heterogeneous set of data, the tool produces a growth forecast for the macroeconomic aggregates that are most closely scrutinised for French economic conditions (GDP, household consumption, investment, etc.). This forecast is made for the current quarter or the following quarters, based on models of machine learning (LASSO, random forests, etc.).

The tool was initially developed for a Business conditions in 2019. Following the health crisis, it was taken up again for more systematic use. Initially focused on France and only on GDP for the current quarter, it has been extended to other countries (Germany, Italy, Spain) and is currently being extended to aggregates other than GDP and to more distant forecasting horizons.
Stakeholders Insee
Project products and documentation - Continuous forecasting of French growth1, Conjoncture in France - December 2019

Similar projects

Footnotes

  1. Regarding nowcasting, an experiment was also conducted in 2022 to impute unpublished variables and forecast growth, see presentation GDP nowcasting: imputation of variables not yet published at the 2022 Statistical Methodology Days (journées de méthodologie statistique).↩︎

]]> machine learning nowcasting experiment stopped forecasts Insee https://ssphub-test.netlify.app/project/2019_gdp_tracker/ Sat, 01 Jan 2022 00:00:00 GMT Curiexplore, the platform for comparing national education and research policies https://ssphub-test.netlify.app/project/2022_Curiexplore/

Project summary

Curiexplore, the platform for comparing national education and research policies
Project details The CurieXplore project, developed by SIES (the statistical department of the French Ministry of Higher Education and Research) offers an interactive visualisation of the teaching and research environment in different countries.
The platform combines textual content collected by SIES from the French diplomatic network and regularly webscrapped data from international statistical sources (UNESCO, World Bank, OECD, Eurostat, etc.) or private sources (ranking of HE institutions such as Shanghai, UMULTIRANK or THE).
Players Statistical Service of the Ministry of Higher Education and Research (SIES)
Project results The project is now in production, with a dedicated website available at curiexplore.enseignementsup-recherche.gouv.fr (in French).
Project code - The code is available on GitHub https://github.com/dataesr/curiexplore-ui.

Similar projects

]]> webscraping in production datavisualisation SIES https://ssphub-test.netlify.app/project/2022_Curiexplore/ Sat, 01 Jan 2022 00:00:00 GMT Methodological work on the Family Budget survey https://ssphub-test.netlify.app/project/2022_Enquete_Budget_Famille/

Project summary

Methodological work on the Family Budget survey
Details of the project The collection of the next Family Budget survey in 2026 will be very different from that of the previous survey (2017).
- In fact, some of the household consumption data will be collected through a mobile application made available to the households surveyed.
- In addition, the products derived from the consumption data will no longer be recorded in the database by the interviewers, but by a data entry or automatic extraction service provider.
There are two main strands to this experiment:
- The first involves testing open-source solutions fromautomatic extraction of the contents of till receipts in order to be able to work more effectively with partners such as Teklia and potentially carry out some of the extraction work in-house;
- The second (main) axis is around the automatic coding of consumer products in the COICOP nomenclature nomenclature: training of a classification model on data from the previous survey and development of an optimal recovery strategy.
Stakeholders Insee
Project results Project in progress
Project code - https://git.lab.sspcloud.fr/ssplab/bdf
- https://git.lab.sspcloud.fr/ssplab/experimentation-bdf

Similar projects

]]> automatic coding data extraction scanner data https://ssphub-test.netlify.app/project/2022_Enquete_Budget_Famille/ Sat, 01 Jan 2022 00:00:00 GMT Jocas, webscraping online job offers https://ssphub-test.netlify.app/project/2022_JOCAS/

Project summary

Online job vacancies, a new source of labour market data
Project details In just a few years, the Internet has become a new source of information on the job market. According to the Dares Job Offer and Recruitment survey (Ofer), 95% of job advertisements were published on the Internet in 2016, compared with 53% in 2005. With this in mind, Dares decided to collect online job offers published on around fifteen websites to create a new database of job offers: Jocas (Job offers collection and analysis system). Various tools are being used to build this new database: webscraping, automatic text classification algorithms and de-duplication.
For 2019, the Jocas database can be compared with the usual sources of public statistics on job vacancies, whether administrative sources, such as vacancies advertised by Pôle emploi and Declarations Préliminaires à l’embauche (DPAE) from Urssaf, or data from surveys such as Pôle emploi’s “Besoins en main-d’œuvre” (BMO), INSEE’s “Emploi” survey, and DARES’ “Activité et conditions d’emploi de la main-d’œuvre” (Acemo) survey. The results show that Jocas covers occupations unevenly. Occupations with a high proportion of managerial staff or that recruit a lot of people online tend to be over-represented. Conversely, those with a high proportion of multiple recruitments or using informal recruitment channels tend to be under-represented.
Players DARES
Project results Online vacancy data has been used to calculate tensions on the labour market. They were also used to produce the table monitoring the labour market situation in 2020-2021 during the Covid-19 crisis. Jocas data is freely accessible to students, researchers and civil servants. Access to the data may also be granted for statistical and non-commercial use, on request from DARES. The database is accessible on the INSEE’s SSPCloud pl by following the path ‘projet-jocas-prod/diffusion/JOCAS’.
Project products and documentation - Description on the DARES website
- Working document
- Hackathon in March 2023 on the duplication of job offers
- Project news
- Training to use the JOCAS database
Project code - Code repo is available on GitHub https://github.com/OnlineJobVacanciesESSnetBigData/JobTitleProcessing_FR

Similar projects

Curiexplore, the platform for comparing national education and research policies

Interactive visualisation of the teaching environment and research environment in different countries.

1 Jan 2022

Methodological work on the Family Budget survey

Modernisation of the family budget survey using automatic classification tools

1 Jan 2022

Open Science Monitor

To be able to monitor the opening up of scientific publications (the objective of the national plan for open science), the statistical service of the Ministry of Higher…

1 Jan 2022

Automatic coding of companies’ main activity

Develop a machine learning algorithm to automate the classification of companies’ main activities and put it into production

1 Jan 2022

Consumer price indices for hotel overnight stays: the experience of webscraping an online booking platform

Exploring webscraping tools to fetch hotel overnight stays in CPI

1 Jun 2021

Predicting growth by reading the newspaper

Use continuous press articles to build an indicator to help forecast growth

1 Mar 2021

Automatic coding of occupations in the PCS 2020 nomenclature

Automatically code occupations as part of the switch to the new PCS nomenclature (PCS 2020)

1 Jan 2021

Webscrape product characteristics to improve inflation measurement

Collect product characteristics on the web to improve the way quality effects are taken into account in the consumer price index.

1 Jun 2020

Classification of checkout data using machine learning

Using machine learning to classify scanner data in the COICOP nomenclature to calculate the CPI

1 Jan 2020

Automatic coding of association activity

Automatic coding of association activity using machine learning methods

1 Jun 2019
No matching items
]]> webscraping in production automatic coding DARES https://ssphub-test.netlify.app/project/2022_JOCAS/ Sat, 01 Jan 2022 00:00:00 GMT Open Science Monitor https://ssphub-test.netlify.app/project/2022_bso/

Project summary

French Open Science Monitor
Project details The generalisation of open access to scientific publications is one of the axes of the French national open science strategy, with the objective of a 100% open access rate in 2030. It facilitates, broadens and accelerates the dissemination of the results of research to scientific communities and to society in general: teachers, students, companies, associations, public policy actors, etc.

The French Open Science Monitor thus constructs open science indicators for publications, data, software and codes in order to measure the achievement of this objective.
As far as publications are concerned, as much metadata as possible is collected. It is then filtered to keep only French affiliations and enriched to track the open status of publications by identified discipline. The data is then published as open data. This method makes it possible to create the most exhaustive database of French publications in the world.
For data, software and codes, the scientific production is first listed. The text of the publication is then analysed to detect the availability of open access data, software or codes. Finally, monitoring indicators are produced and published.
Players Statistical Service of the Ministry of Higher Education and Research (SIES)
Project results The project is presented on the dedicated website : https://frenchopensciencemonitor.esr.gouv.fr/. It includes a methodology
Project code The code files are available on GitHub
]]> webscraping datavisualisation open-data in production SIES https://ssphub-test.netlify.app/project/2022_bso/ Sat, 01 Jan 2022 00:00:00 GMT Automatic coding of companies’ main activity https://ssphub-test.netlify.app/project/2022_codif_ape/

Project summary

Automatic coding of companies’ main activity
Project details The coding of companies’ main activity (APE) on the basis of activity descriptions (in the form of free text) in the Sirene register was previously carried out using 6 deterministic coding environments mobilising a huge number of decision rules. The aim of the experiment is to test the performance of statistical learning models in predicting the category of APE category as part of the overhaul of the Sirene register and the introduction of a one-stop shop.
Stakeholders Insee
Project results The model developed presents similar performance to previous models by automating them, and also offers decision support. The model has also been put into production, applying MLOps principles where possible.
Presentations and written materials relating to the project can be accessed at this site.
Project code Accessible code repositories on Github . They include:
- Code for annotating data using Label Studio ;
- Code for a coding web API deployed on the SSP Cloud ;
- Code implementing a visualisation dashboard for monitoring the activity of a coding model in production and accessible via a web API;
- Code for training APE classification models.

Project documents

No matching items

Similar projects

sndsTools, a R package for extracting healthcare utilization in SNDS health data

The R package sndsTools facilitates the extraction of healthcare utilization from the Système National de Données de Santé (SNDS) health data hosted on the National Health…

17 Mar 2026

Doremifasol

The package Doremifasol makes it easier for data scientists to retrieve Insee data. The library is open source.

1 Jan 2023

pynsee, a Python package for retrieving INSEE data

The package pynsee package makes it easier for data scientists to retrieve INSEE data. The library is open source.

1 Jan 2023

Using satellite images for official statistics

Using satellite images to improve population censuses in the French overseas territories

1 Oct 2022

Methodological work on the Family Budget survey

Modernisation of the family budget survey using automatic classification tools

1 Jan 2022

Jocas, webscraping online job offers

The project Jocas (Job offers collection and analysis system) project enables the DARES (Ministerial Statistical Office for Labour) to automatically collect job offers…

1 Jan 2022

Automatic coding of occupations in the PCS 2020 nomenclature

Automatically code occupations as part of the switch to the new PCS nomenclature (PCS 2020)

1 Jan 2021

Classification of checkout data using machine learning

Using machine learning to classify scanner data in the COICOP nomenclature to calculate the CPI

1 Jan 2020

Automatic coding of association activity

Automatic coding of association activity using machine learning methods

1 Jun 2019
No matching items
]]> Python automatic coding package in production MLFlow https://ssphub-test.netlify.app/project/2022_codif_ape/ Sat, 01 Jan 2022 00:00:00 GMT Detecting cybercrime in procedures https://ssphub-test.netlify.app/project/2021_detection_cyber/
Detecting cybercrime in procedures
Project details Cybercrime is a new and fast-growing form of criminality, and the fight against it is a major challenge for the police and gendarmerie. However, it is proving particularly difficult to quantify. It covers all criminal offences attempted or committed against or mainly using information and communication systems. The nomenclature of offences allows only part of this type of crime to be identified. To obtain a better statistical measure, it is necessary to use information provided in textual areas of procedures. The aim of the experiment is to explore the potential of textual analysis methods on these data to measure the extent of this type of crime.
Players INSEE and the Ministerial Statistical Office (MSO) for Internal Security (SSMSI)
Project results The work was taken up by the SSMSI to derive a two-dimensional nomenclature of cyber-related offences. The algorithm predicts more than 95% of offences as falling within the definition of cybercrime. Similarly, it predicts less than 1% of offences as not falling within the scope of cybercrime as defined.
Project products and documentation - Detection of cybercrime offences based on textual analysis of modus operandi, 2022 Statistical Methodology Days (Journées de méthodologie statistique 2022)
]]> in production SSMSI deep learning https://ssphub-test.netlify.app/project/2021_detection_cyber/ Tue, 01 Jun 2021 00:00:00 GMT Modelling road vehicle fleet ownership and use https://ssphub-test.netlify.app/project/2021_vehicules/

Project summary

Modelling road vehicle fleet ownership and use
Project details In 2019, the Ministerial Statistical Office of the French Ministry of Ecological Transition (SDES) will be overhauling the statistical register of road vehicles (RSVero). The main innovation of this project is the use of technical inspection data to ensure that registered vehicles are still in circulation and to determine their annual use, thanks to the odometer reading taken at each visit. However, the integration of this data raises methodological issues: inspections take place on variable dates, sometimes with a delay, and we cannot be sure of the status of vehicles whose last inspection preceded the date on which we wish to determine the fleet. The project will enable us to estimate, for all the cars in the inventory, the probability that they are still in circulation on a given date and, where appropriate, the annual distance travelled. More broadly, this project is part of the drive to develop methodologies for processing administrative sources.
Players Insee, MSO of the French Ministry of Ecological Transition (SDES)
Project results The methodology is now in production and is used to determine the vehicles on the road each year, their characteristics and their use.
Project products and documentation - Methodology for estimating vehicle fleets and distances travelled (abstract in English) Working document, March 2024
- Modelling road vehicle fleet ownership and use Journées de méthodologie statistique 2022
Project code - Repo GitHub

Similar projects

]]> in production matching SDES administrative data https://ssphub-test.netlify.app/project/2021_vehicules/ Tue, 01 Jun 2021 00:00:00 GMT Consumer price indices for hotel overnight stays: the experience of webscraping an online booking platform https://ssphub-test.netlify.app/project/2021_webscraping_ipc_hotel/

Project summary

Webscraping of the prices of hotel nights to construct the consumer price index
Project details The consumer price index for overnight stays in hotels is calculated from data collected in the field by INSEE surveyors, who record the price of a room for one night on the same day for 2 people, including breakfast. In order to improve the index and overcome some of the limitations of the current method, this project is exploring an innovative collection method, the webscraping from a booking site.
Once the data has been collected online, it is raw and needs to be cleaned up: for example, the value for a characteristic is not necessarily described in the same way between two observations. To overcome the problems associated with a fixed basket index, the final index is constructed from homogeneous classes. Finally, the results of the index calculated using data from the online booking platform are compared with the published index.
Players Insee
Project results Since 2026, the hotel price index has been based half on field data and half on the booking platform.
Project products and documentation - Consumer price indices for hotel overnight stays: the experience of webscraping an online booking platform, 2022 Statistical Methodology Days (Journées de méthodologie statistique 2022)

Similar projects

An assessment of cross-border tobacco purchases and associated tax losses in France
experiment
mobile phone data
credit card data
Insee

Using the closing of borders in 2020 as a natural experiment to measure cross-border tobacco purchases

1 Jan 2024

Curiexplore, the platform for comparing national education and research policies
webscraping
in production
datavisualisation
SIES

Interactive visualisation of the teaching environment and research environment in different countries.

1 Jan 2022

Methodological work on the Family Budget survey
automatic coding
data extraction
scanner data

Modernisation of the family budget survey using automatic classification tools

1 Jan 2022

Jocas, webscraping online job offers
webscraping
in production
automatic coding
DARES

The project Jocas (Job offers collection and analysis system) project enables the DARES (Ministerial Statistical Office for Labour) to automatically collect job offers…

1 Jan 2022

Open Science Monitor
webscraping
datavisualisation
open-data
in production
SIES

To be able to monitor the opening up of scientific publications (the objective of the national plan for open science), the statistical service of the Ministry of Higher…

1 Jan 2022

Predicting growth by reading the newspaper
Insee
machine learning
experiment
forecasts
webscraping

Use continuous press articles to build an indicator to help forecast growth

1 Mar 2021

Using credit card data and mobile phone data to forecast economic activity
forecasts
experiment
Insee
credit card data
mobile phone data

The 2020 health crisis required a review of forecasting processes to be more responsive to events. INSEE used credit card transaction data to forecast economic activity.

1 Dec 2020

What do the electricity production and consumption data say about economic activity during the containment period?
Insee
forecasts
private data
experiment

Using electricity production and consumption data to forecast economic activity

1 Dec 2020

Population movements around the March 2020 containment using mobile phone network operators data
datavisualisation
machine learning
Insee
experiment
open-data
mobile phone data

INSEE has had access to mobile telephony data as part of the monitoring of the 2020 health crisis. These data were used to produce the following statistics on population…

1 Nov 2020

Webscrape product characteristics to improve inflation measurement
Insee
CPI
webscraping
random forest

Collect product characteristics on the web to improve the way quality effects are taken into account in the consumer price index.

1 Jun 2020

Classification of checkout data using machine learning
Python
automatic coding
scanner data
COICOP
CPI
in production

Using machine learning to classify scanner data in the COICOP nomenclature to calculate the CPI

1 Jan 2020

Urban segregation: insights from mobile phone data
Insee
experiment
mobile phone data
administrative data

Merging administrative data and MNO data to estimate urban segregation at a local level

1 Jan 2018
No matching items
]]> in production webscraping CPI Insee https://ssphub-test.netlify.app/project/2021_webscraping_ipc_hotel/ Tue, 01 Jun 2021 00:00:00 GMT Predicting growth by reading the newspaper https://ssphub-test.netlify.app/project/2021_gdp_media/

Project summary

Construction of an economic sentiment indicator based on press articles to forecast economic growth
Project details The information produced by the media is reactive, abundant and covers a wide range of economic fields. In addition, text analysis (text mining) and automated online data collection (webscraping) techniques can be used to develop indicators that summarise this wealth of information. The aim of the experiment is to estimate the added value of a media indicator associated with economic activity by complementing existing business surveys. This experiment complements pre-existing work, which is seeking to construct an indicator to help predict GDP on the basis of the archives of the newspaper Le Monde, by mobilising complementary journalistic sources and reviewing the models used.
Players Insee
Project results An indicator of current media sentiment was constructed on the basis of the tone of press articles.
Project products and documentation - How to forecast employment figures by reading the newspaper Economic outlook, March 2017 ;
- Nowcasting GDP Growth by Reading Newspapers Economics and Statistics n° 505-506 - 2018 ;
- Information gleaned from press articles can help predict economic activity in real time Economic outlook, December 2020 ;
- French economic activity through press articles Economic Outlook, March 2021 ;
- Predicting economic activity from press articles (abstract in English), 2022 Statistical Methodology Days (Journées de méthodologie statistique 2022)

Similar projects

]]> Insee machine learning experiment forecasts webscraping https://ssphub-test.netlify.app/project/2021_gdp_media/ Mon, 01 Mar 2021 00:00:00 GMT Comparison of matching methods and the contribution of machine learning https://ssphub-test.netlify.app/project/2021_Appariement/
Comparison of matching methods and the contribution of machine learning
Project details The Resil programme aims to build a sustainable and scalable system of directories of individuals, households and residential premises, updated from a variety of administrative sources. It requires the aggregation of several data sources without a common direct identifier.
The aim of the experiment is to test and compare different matching methods in order to draw up recommendations for the work needed to build the directories. These will be based on performance criteria (quality of matching) but also on operational considerations (ease of deployment, calculation time, etc.). In particular, the aim is to assess the contribution and constraints of probabilistic methods and machine learning in matching tasks. This work will be accompanied by a reflection on the prior normalisation of data and the evaluation of matching results.
Stakeholders Insee
Project products and documentation - Methodology for matching individual data, 2022 Statistical Methodology Days (Journées de méthodologie statistique 2022)
- Probabilistic or deterministic, matching methods put to the test by the RéSIL programme, 2022 Statistical Methodology Days (Journées de méthodologie statistique 2022)
- Impact of data cleaning on the quality of a match, 2022 Statistical Methodology Days (Journées de méthodologie statistique 2022)
- Matching: aims, practices and quality issues (French), Working document, Insee, July 2024
]]> matching administrative data in production https://ssphub-test.netlify.app/project/2021_Appariement/ Fri, 01 Jan 2021 00:00:00 GMT