The Main objective is to set up  a pilot hub, to collect, handle, archive, and exploit in a synergetic way observational data, as will be provided by the future European integrated atmospheric composition ground-based network infrastructure for the validation of ESA and Copernicus missions. The backbone of this synergy is GRASP (Generalized Retrieval of Aerosol and Surface Properties) algorithm.

Phase 1: Preparation of inputs (lidar, photometer, spectrometer), GRASP prototype, and processing environment

  1. Coordination of the LIDAR, photometer and spectrometer network operations, including centralized archiving and processing
  2. Development of standard procedures (best practice and schedules) for calibration and operation of the instruments.
  3. Development of a data aggregation and processing system providing advance environment for applying cutting-edge algorithms

Phase 2: Development of advanced retrieval algorithms, testing and operation of the DIVA platform

  1. On single instruments level advance measurement techniques and consolidate calibration procedures considering also the latest instrument generation und upgrades, by developing advanced QC and processing codes and the associated new (single sensor) data products.
  2. Development of new retrieval algorithms, exploiting synergistic approaches combining multiple measurement techniques and extended records from single and multiple instruments.
  3. Development and deployment of the DIVA operative system, starting from the Phase 1 prototype. The system shall include tools for data analysis and data mining functionalities.
  4. Demonstration period for 6 months, providing access to the various DIVA services in an operational mode.

Work Breakdown Structure – Phase 1

The work is organized in 5 technical work packages, and one management and outreach work package. Work package 1 to 3 covers the instrument-specific technical work, i.e. development and documentation of calibration tools and standard operation procedures corresponding to lidar, photometer and spectrometer networks, including description of data formats and communication standards. Work package 4 focuses on the development of the data processing tool based on GRASP, while WP5 focuses on the integration of all data and tools, including the harmonization of the data formats, and implementation of the validation services.

Work Logic flow chart

Lidar input

This work focus on fostering uniform and timely lidar measurements, and to allow a traceable quality of the data products. For this, INOE will document the EARLINET instrument quality assurance procedures and operation best practices, and make them easy to be implemented at any lidar station. Optimization of the lidar data format, in order to: a) be self-consistent; b) be traceable; c) be understandable by external users; c) be potentially harmonized with the data formats from other sensors / data sources, is also considered. Several software tools will be developed, to allow the automatic use of the EARLINET’s Single Calculus Chain (SCC), fostering near-real time delivery of the data products.

Photometer input

This work focus on documenting photometer calibration tools and operation procedures for sun/sky/polar/lunar photometer calibration and quality assurance, and to document possible new observation protocols to increase aerosols retrievals frequency. During Phase 1, the activity will focus therefore on: (i) documentation on the new CE318-T photometer QA/QC procedures and best practices for site maintenance; (ii) Near Real Time delivery of the primary products (calibrated radiometry)  requested for WP 4000; (iii) design of faster observation protocols more adapted to cloudy situations.

Spectrometer input

This work focus on: a) developing a set of documents to define the legal, logistical, and technical requirements to run the Pandonia network in an operational mode; b) adapting Pandonia calibration, operation, data formats and algorithms so that the GRASP algorithm can be applied to them; c) equipping one Pandora instrument with polarizing filters to be capable of retrieving angle and degree of polarization of the sky radiance as preparation to apply the GRASP algorithm on these data.

Inversion Algorithm and Interfaces

This work focus on the development of a prototype of software for automated data control and processing, that could process observation data in automated regime with minimized interaction of user. The software shall use GRASP algorithm and shall allow a variety of retrieval options to combine observations, change the retrieval assumptions, retrieval regimes, modes and content of output generation and presentation. In principle, the software should adapt full flexibility in data processing that is already implemented in GRASP retrieval package. At this stage, it is expected that the software will be ready to demonstrate the use of data from AERONET/PHOTONS and EARLINET. All these data should be treated in similar format, appropriate for straightforward control, visualization. The software also should serve as a prototype of elaborated environment for processing of the observation by GRASP algorithm. Therefore, this WP includes the efforts on preparing and linking the data to the inputs of GRASP.

System Integration and Validation

This work focus on the integration of all developed software in one backend processing infrastructure. The software is developed distributed, but integrated in one platform. Webservices and tools are implemented to integrate, run, monitor, distribute, visualize the applications. Data are stored and transformed to a standard file format.

Work Breakdown Structure – Phase 2

The project is organized in 4 Work Packages. WP1 Evolution of instruments and measurement techniques includes activities concerning consolidation and evolution of the instruments, already involved in Phase 1 (i.e.: lidars, photometers and spectrometers), and their corresponding measurement techniques. WP2 Retrieval Algorithms aim to fully integrate in the hub the retrieval algorithms based on the GRASP prototype from phase 1, and to enhance the algorithm to fully exploit the combination of available data from photometers (Cimel/Aeronet) and spectrometers (Pandora/PGN) to simultaneously retrieve advanced aerosol (optical and microphysical) products as well as trace gas products. WP3 Development, deployment and test of the full DIVA system include deployment of the DIVA operative system, starting from the Phase 1 prototype. The platform capabilities will be extended including tools for data analysis and data mining functionalities with the aim to enhance data exploitation.

Deliverables – Phase 1

Doc ID Title Type of deliverable
D0 Demonstration Instrument Site List Documentation
D1 DIVA Algorithm Technical Baseline Document Documentation
SW4 Lidar format converters Software
PIWP Project Web Information Pages Other
D2 Standard DIVA data formats descriptions Documentation
D3 DIVA Architectural Design Document Documentation
D4 DIVA standard calibration procedures and tools documentation Documentation
D5 Standard DIVA operation procedures documentation Documentation
D6 User manuals for DIVA Data processing tool prototype, runtime environment and validation service prototype Documentation
D7 DIVA Data processing tool prototype test report Documentation
SW1 DIVA Data processing tool prototype Software
SW2 DIVA validation service prototype Software
SW3 DIVA runtime and monitoring webservice Software

Deliverables – Phase 2

Doc ID Deliverable title Deliverable type
D1 DIVA Retrieval algorithms for Lidars Technical Note / Algorithm Theoretical Basis Documents
D2 Stand-alone DIVA retrieval algorithms for Photometers Technical Note / Algorithm Theoretical Basis Documents
D3 Photometer Aerosol products with increased sampling Technical Note
D4 Pandora products optimised for DIVA Technical Note / Algorithm Theoretical Basis Documents
D5 Calibration and data file format for Pandora Technical Note
D6A Diva (Synergy) Product List and Description Technical Note
D6 Data Processing Tool Technical Note
D7 Demonstration and updates of DIVA Ph2 Processing Tool (incl. full Pandora data integration) Test Report
D8 DIVA data analysis functionalities Design Document
D9 DIVA evolution and performance Test Plan
D10 DIVA evolution and performance Test Report
D11 Final & Evolution Report Documentation
CCD Contract Closure Documentation Documentation
SW1 Calibration verification and QA tools for Lidars Software
RN1 Calibration verification and QA tools for Lidars Release Note
SW2 Data Processing Tool Software
RN2 Data Processing Tool Release Note
WS Web Story Other
PW Project Website Other

Meetings – Phase 1

KoM: 2 October 2017

Technical Meeting: 9 November 2017

Preliminary Design Review Meeting: 18-19 of December 2017 in Magurele, Romania

Critical Design Review (CDR) meeting: 28 June May 2018, remote

Acceptance Review (AR) meeting: 11 September 2018, Lille, France

Meetings – Phase 2

KoM: 17 October 2019, Magurele, Romania

Progress Meeting 1: January 2020, remote

Progress Meeting 2: to be announced, Vienna, Austria

Acceptance Review meeting: to be announced, remote

Final Presentation meeting: to be announced, Frascati, Italy