C3S 311c Lot 2

The C3S Historic In Situ Upper Air Database Service shapes the existing upper air data collections into a form that is suitable as input for future reanalysis as well as for all kinds of queries from the Internet via the Copernicus Data Store. The objective of the Service is to develop and maintain a quality-controlled global database containing all known digitised in-situ upper-air weather observations made prior to 1979, together with metadata and information needed for data assimilation such as bias adjustments and uncertainty estimates. The C3S 311a Lot 3 service is mostly concerned with post 1979 upper air data and therefore the database proposed in this tender completes the full time period for which there are upper air observations.

The team of experts in upper air data formed for this tender ensures that the inventory of sources, the database and the tools produced will be state-of-the-art and fit for use, building upon existing data standards. The Service will build upon the structures developed in C3S 311a and ERA-CLIM(2) (EU FP7), which will be adapted to the needs of collecting and storing existing and newly digitized upper air data where necessary. The Common Data Model (CDM) developed in the C3S 311a Lots appears general enough to fit in all aspects of upper air data needed for data assimilation and climate research. Should extensions be necessary those will be coordinated with C3S 311a. Close collaboration with the Scientific Software Engineers (SSEs) employed in this service, as well as with core developers at ECMWF is envisaged. The aim is seamless integration of the upper air data set into the other in situ databases made available via the Climate Data Store, thereby facilitating work for developers and scientists alike.

The upper air data set, in particular the temperature and humidity records, are known to need sizeable bias adjustments to fulfil the accuracy demands of future reanalyses. Upper air wind data are subject to occasional bias adjustments as well. Such adjustments have been developed within EU FP7 projects ERA-CLIM and ERA-CLIM2 and it is expected to improve those once again while developing the Service. Particular emphasis will be given to adjusting also shorter records since those would otherwise compete with adjusted records, thereby reducing the overall effect of the adjustments on reanalyses.

The uncertainties of these adjustments will be continuously assessed, by comparing them with independent surface and satellite data after 1979 but also by comparing them with the satellite data to be rescued and processed as described in Lot1 of this ITT. These will complement standard uncertainty estimates from reanalysis departure statistics. An additional method of uncertainty assessment is the comparison with historic radiosonde intercomparison (RI) experiments. There is a sizeable number of reports about those back to the early 1950s. The intercomparisons are generally better documented compared to contemporaneous standards. Yet the data are spread over different places, are often not available in digital form or have not been compared against reanalyses so far. Given the lack of independent reference data those data seem highly valuable for uncertainty estimation.

The upper air data, together with their bias adjustments, will be actively managed and backed up and are therefore resilient to software problems, data corruption, and hardware failure. Source codes, configuration and content will be kept under version control, and will be maintained for the duration of the Service at the highest standards available.  The software, including the bias adjustment software, will be made available publicly, such that it can be potentially maintained with relatively little effort after the end of the project. The SM and his deputy will register in the C3S user support system to be able to help with queries also after the end of this contract. The Service will produce annual implementation reports of the state of upper air data supplying information about progress, needs, gaps and priorities.