ICOS Data Flow

ICOS produces quality-controlled observational data, as well as results from modelling related to greenhouse gases, and synthesis reports. These data come from national measurement networks, from our ICOS Thematic Centers and from modelers in the greenhouse gas research community.

We produce about 25-30 TB of sensor data per year, along with a bit under 1 GB of processed data products and 5-20 TB of elaborated data products. The ICOS processed data products are mostly time series of observed parameter values, for example carbon dioxide concentrations, given for every 30-minute interval. Every observation station delivers such time series for 50-100 parameters, representing the measurements made every day of every year.

Data flow schematic

The diagram shows the flow of data within ICOS. The explanations below relate to the starred numbers in the diagram:

ICOS data flow diagram

1. Data is collected at ICOS measurement stations. There are ca 100 sites in 9 countries, making up our three networks of atmospheric, ecosystem and ocean stations. Each is equipped with up to a hundred instruments and sensors that collect data. [Links!]

2. The “raw” data are stored in a safe repository as soon as possible. Environmental observation data are very precious, since it isn’t possible to go back and re-measure them if for example data are lost. Therefore, copies of all “raw” data are transferred to safe, long-term storage at a trustable data center. Typically, this is done within 24 hours of collecting the data.

3. The observation data are then passed on to the Thematic Centers for expert processing. Stations also send their sensor data to the relevant Thematic Centre - atmosphere, ecosystem or ocean - for processing and quality control.

4. The ICOS Thematic Centers take care of and process the observations following standardized procedures. At the Thematic Centers, the data are checked, quality controlled and if needed gap-filled. Some data types, for example fluxes, require quite extensive processing. Finally, data are aggregated into half-hourly or hourly averages. (Also mention the Central Analytical Laboratories?)

5. Quality-controlled data are delivered by the Thematic Centers. When ready to be released, the finalized observation data are transferred to the Carbon Portal. How often new data become available varies, but the aim is to reduce the time between collection and release to a few months.

6. The Carbon Portal is responsible for organizing and taking care of all ICOS data. The Carbon Portal is the “one-stop shop” for ICOS data products, and offers advanced search, visualization and downloading services. The portal is also responsible for all ICOS data management, such as assigning identifiers (DOIs) to data to make it easy to cite, the tracking of how ICOS data are used, and of course long-term archiving. Finally, it will also provide user community support.

7. Users can freely and openly access the ICOS data. (And some contribute as well!) Everyone who wants to access, view or download ICOS data products is welcome to do so. All we ask is that the data are properly referenced and acknowledged. Some ICOS data users, including atmospheric and ecosystem modelers, also use the Carbon Portal to distribute their research results.

8. All ICOS data products are safely stored in the ICOS repository. Copies of all data products that are handled by the Carbon Portal are stored in a safe, long-term manner in the ICOS repository. This also includes all metadata that is necessary to make sense of the data themselves. The Repository is based on the B2SAFE service provided by EUDAT.

9. Descriptions of the ICOS data products and their contents must be easily found. ICOS is of course operating its own data catalog, but we also share information about all the ICOS data products and how to access them with other data providers - including environmental and climate data portals.

10. ICOS data can be effectively and quickly sent on to other computing centers. Our collaboration with European e-infrastructure providers like EUDAT and EGI makes it easy to transfer ICOS data to and from high-performance computing centers.