New Pydap service improves access to NetCDF data

ICOS data in the NetCDF format are now even easier to use, thanks to a new Pydap service adapted and deployed by the Carbon Portal for the ICOS data users. The service makes it possible for researchers to focus on their scientific work instead of spending time downloading, storing and managing large data files. 

NetCDF, or Network Common Data Form, is a file format for storing multidimensional scientific data such as temperature, humidity, pressure, wind speed, and direction. The ICOS Carbon Portal, operated from Lund, Sweden, has launched a Pydap service that makes it easier for ICOS data users to access and use data in the NetCDF format. 

One common challenge with NetCDF data is the size of the files. Previously, researchers often had to download large data files on their computers and open them using analysis tools before even beginning their research. Downloading and storing the files took time and storage space, and required some expertise from the researchers’ part. 

“There was an intermediate step of managing data before actually using it for research, which made things slower”, describes software developer Zois Zogopoulos, who leads the development of the Pydap service at ICOS Carbon Portal. “Even advanced users who knew how to access the data through our Jupyter services still needed considerable knowledge and guidance on how to access them.” 

Through the Pydap service, the same NetCDF data can now be accessed using a small code snippet in Python or other programming languages. Users can also request only the part of a dataset that they need, rather than downloading the entire file. The purpose of the service is to minimise the steps between requesting data and using them in research.

“You can make a request to the ICOS Pydap service and receive the results in a formatted way”, Zogopoulos says. “We want researchers to be able to do the research, not spend their time managing the data.” 

Improved efficiency and data access for modellers, researchers and data scientists

The Pydap service adapted for ICOS is intended for all researchers, modellers and data scientists working with NetCDF data. It’s built on top of the OPeNDAP protocol, which is already familiar to many users of NetCDF data. The ICOS Carbon Portal developers retained its core functionality and built additional features so that it could work together with other ICOS services. 

“We introduced the authentication required to access ICOS data and added support for accessing data through persistent identifiers, or PID’s. We also connected it to the metadata service, so you also receive the metadata together with the data, which is important”, Zogopoulos says.

Although most observational data have historically been stored in the CSV format, NetCDF is now becoming the standard format due to its efficiency, high level of standardisation, and descriptive data that can be used in data interpretation. The ICOS Carbon Portal has converted a large number of CSV files to NetCDF to improve access and data processing of those data. 

“We have also recently converted the ICOS FLUXNET archive datasets from large and complex CSV files into a single NetCDF file for each station. This gives users access to all variables and time aggregations, more than 720 variables in total, through one file. Users can query one or more variables for individual stations, several stations, all stations within a particular area, or stations representing a certain ecosystem type. Instead of first downloading and storing around 700 GB of data, they can receive the requested results directly in their script within seconds”, Alex Vermeulen, Director of ICOS Carbon Portal describes. 

Learn more: 

ICOS Carbon Portal webinars and other educational resources

ICOS Jupyter notebooks for education