About ICOS data

ICOS Research Infrastructure builds on three domains: ecosystem, atmosphere and ocean. Each domain is a network of measurement stations. The stations can also be called 'research sites' or 'platforms'. The stations within one domain deliver in principle the same collection of measurements and by following the defined protocols each station will deliver the quality that is required by ICOS. Below you find more details about the data flow with in ICOS from sensor to data-set, about the measurement protocols and data quality.

Data flow within ICOS

ICOS data levels, quality an data products

Data ready to be used: information on measurements methods and data processing

International standards and ICOS data

Uploading data to the CP

ICOS Data Flow: From field to computer

ICOS produces quality-controlled observational data, as well as results from modelling related to greenhouse gases, and synthesis reports. These data come from national measurement networks, from our ICOS Thematic Centers and from modelers in the greenhouse gas research community.

We produce about 25-30 TB of sensor data per year, along with a bit under 1 GB of processed data products and 5-20 TB of elaborated data products. The ICOS processed data products are mostly time series of observed variables, for example atmospheric carbon dioxide concentrations, given for every 30-minute interval. Every observation station delivers such time series for 50-100 variables, representing the measurements made every day of every year.

Data flow schematic

The diagram shows the flow of data within ICOS. The explanations below relate to the starred numbers in the diagram:

ICOS data flow diagram

1. Data is collected at ICOS measurement stations. There are ca 120 sites in 12 countries, making up our three networks of atmospheric, ecosystem and ocean stations. Some sites or stations are equipped with up to a hundred instruments and sensors that collect data.

2. The “raw” data are stored in a safe repository as soon as possible. Environmental observation data is very precious, since it isn’t possible to go back and re-measure if data is lost. Therefore, copies of all “raw” data are transferred to safe, long-term storage at a trusty data center. Typically, this is done within 24 hours after collecting the data.

3. The observation data is then passed on to the Thematic Centers for expert processing. Each station send their sensor data to the relevant Thematic Centre - atmosphere, ecosystem or ocean - for processing and quality control.

4. The ICOS Thematic Centers take care of and process the observations following standardized procedures. At the Thematic Centers, the data are checked, quality controlled and if needed gap-filled. Some data types, for example fluxes, require quite extensive processing. Finally, data is aggregated into half-hourly or hourly averages. The Central Analytical Laboratory is a fourth centre (not shown in the figure) that delivers data on calibration for processing.

5. Quality-controlled data are delivered by the Thematic Centers. When ready to be released, the finalized observation data is transferred to the Carbon Portal. How often new data become available varies, but the aim is to reduce the time between collection and release as much as possible.

6. The Carbon Portal is responsible for organizing and taking care of all ICOS datasets. The Carbon Portal is the “one-stop shop” for ICOS data products, and offers advanced search, visualization and downloading services. The portal is also responsible for all data management, such as assigning identifiers (DOIs) to make it easy for citation, for tracking of how ICOS data is used, and of course long-term archiving. Finally, it will also provide user community support.

7. Users can freely and openly access the ICOS data. And contribute as well! Everyone who wants to access, view or download ICOS data products is welcome to do so. All we ask is that the data are properly referenced and acknowledged. Some ICOS data users, including atmospheric and ecosystem modelers, also use the Carbon Portal to distribute their research results. If you have a data product that is based on ICOS data you are invited to make you data available through the Carbon Portal.

8. All ICOS data products are safely stored in the ICOS repository. Copies of all data products that are handled by the Carbon Portal are stored in a safe, long-term manner in the ICOS repository. This also includes all metadata that is needed to make sense of the data themselves. The Repository is based on the B2SAFE service provided by EUDAT.

9. Descriptions of the ICOS data products and their contents must be easily found. ICOS is of course operating its own data catalog, but we also share information about all the ICOS data products and how to access them with other data providers - including environmental and climate data portals.

10. ICOS data can be effectively and quickly sent on to other computing centers. Our collaboration with European e-infrastructure providers like EUDAT and EGI makes it easy to transfer ICOS data to and from high-performance computing centers.

 

ICOS data levels, quality and data products

In the data portal three levels of data are considered, represented by 0, 2 and 3. Each level is considered as an ICOS data product.

These are derived from the data levels considered in the data flow from measurement instrument to elaborated product. Note, that in the data flow more levels are presented than available as data product.

Raw data

Raw data is information or objects directly obtained from human measurements or automated sensors and having received no transformation since. They can provide a quantitative or qualitative information about physical variables of the environment, and may be of different forms like images, text files, human activities or physical samples.

Level 0 data

Level 0 data is data in physical units either directly provided by the instruments or converted from engineer units (e.g. mV, mA, Ω) to physical units at the Thematic Centre. They may have been filtered by a quality check (e.g. thresholds).

Level 1 data

Near Real Time Data (NRT) data is generally developed for fast distribution as timing between measurement and published data differs between the domains. Thanks to developments during the past years, sufficient Quality Check (QC) can now be applied automatically in almost real-time and the NRT data is processed in a more consolidated way. NRT data is therefore defined as a high-quality data set that can be made available for distribution as data product. Note; the data is only automatically Quality Checked and can eventually differ from the final data product. NRT data products are presented in the data portal as level 1 data and in general available for direct download.

Internal Work data (IW) is not presented in the data levels of the Carbon Portal. The data that is generated as intermediate steps in the data processing from raw data to final data product are considered as Level 1, but are not shared outside ICOS RI, nor handled as persistent data. The level 1 data is used for internal quality checks and/or communication between the thematic Centers and Principle Investigator at the sites. However, for transparency reasons, handling of L1 data generate relevant metadata that is stored following with the resulting NRT or L2 data sets.

Level 2 data: observational data

Level 2 data is the final quality checked ICOS RI data set, published by the CFs, to be distributed through the Carbon Portal. This level is the ICOS-data product and free available for users.

Level 3 data: elaborated products

All kinds of elaborated products by scientific communities that rely on ICOS data products are called Level 3 data. The CP will provide resources to integrate and disseminate L3 products that will be provided by the research community on a voluntary basis and/or, if agreed upon, from collaborative projects. If you have data products that are based on ICOS observational data, you are invited to present your product through the Carbon Portal as an elaborated product.

 

Data ready to be used: information on measurements methods and data processing

ICOS gives free access to data. But data needs to be described well to make it useful. We aim to describe the dataset with rich metadata. As a user you not only need to know what each variable means, but also the used methods behind each value must be transparent. Although not yet fully operational, the metadata for each dataset, presented on the landing page, will give access to information on how the variables were measured and processed. Today, the ICOS measurements methods are described in protocols and various datasets are supported by specific documentation. Description of instruments and measurement protocols are today available through the Thematic Centres:

Ecosystem Stations, ETC

Atmospheric Stations, ATC

Ocean Stations, OTC

Central Analytical Laboratories, CAL is presenting the list of used instruments

International standards and ICOS data

ICOS including Carbon Portal is following the international developments and standardization within data management. An example is the handling of metadata, the data that describes the datasets. To assure that ICOS data is FAIR, it is essential to apply international recognized standards. The aim of our metadata profile is in the first place to make ICOS data searchable, findable and interoperable. We develop currently our metadata profile which is based on the   We follow here the European INSPIRE standard. INSPIRE is again based on the ISO 19115 metadata standard.

 

Uploading data to the CP

The ICOS measurements that are collected in datasets are in general uploaded in an automatic procedure from station to the trusted repository. The Carbon Portal makes this data accessible. It is however possible that other data needs to be uploaded manually to the Carbon Portal and added to the repository. The Carbon Portal has developed an upload service. If you want to upload data to the Carbon Portal you should in the first place contact us. A general technical description of the upload procedure is available on the GitHub site of ICOS Carbon Portal.  A GUI or internet service for uploading of single files is under development. Uploading files is coordinated from the Carbon portal and only possible if you have given permission for upload.