Objectives
The Main objective is to set up a pilot hub, to collect, handle, archive, and exploit in a synergetic way observational data, as will be provided by the future European integrated atmospheric composition ground-based network infrastructure for the validation of ESA and Copernicus missions. The backbone of this synergy is GRASP (Generalized Retrieval of Aerosol and Surface Properties) algorithm.
Phase 1: Preparation of inputs (lidar, photometer, spectrometer), GRASP prototype, and processing environment
- Coordination of the LIDAR, photometer and spectrometer network operations, including centralized archiving and processing
- Development of standard procedures (best practice and schedules) for calibration and operation of the instruments.
- Development of a data aggregation and processing system providing advance environment for applying cutting-edge algorithms
Phase 2: Development of advanced retrieval algorithms, testing and operation of the DIVA platform
- On single instruments level advance measurement techniques and consolidate calibration procedures considering also the latest instrument generation und upgrades, by developing advanced QC and processing codes and the associated new (single sensor) data products.
- Development of new retrieval algorithms, exploiting synergistic approaches combining multiple measurement techniques and extended records from single and multiple instruments.
- Development and deployment of the DIVA operative system, starting from the Phase 1 prototype. The system shall include tools for data analysis and data mining functionalities.
- Demonstration period for 6 months, providing access to the various DIVA services in an operational mode.
Work Breakdown Structure – Phase 1
The work is organized in 5 technical work packages, and one management and outreach work package. Work package 1 to 3 covers the instrument-specific technical work, i.e. development and documentation of calibration tools and standard operation procedures corresponding to lidar, photometer and spectrometer networks, including description of data formats and communication standards. Work package 4 focuses on the development of the data processing tool based on GRASP, while WP5 focuses on the integration of all data and tools, including the harmonization of the data formats, and implementation of the validation services.
Work Logic flow chart
Lidar input
This work focus on fostering uniform and timely lidar measurements, and to allow a traceable quality of the data products. For this, INOE will document the EARLINET instrument quality assurance procedures and operation best practices, and make them easy to be implemented at any lidar station. Optimization of the lidar data format, in order to: a) be self-consistent; b) be traceable; c) be understandable by external users; c) be potentially harmonized with the data formats from other sensors / data sources, is also considered. Several software tools will be developed, to allow the automatic use of the EARLINET’s Single Calculus Chain (SCC), fostering near-real time delivery of the data products.
Photometer input
This work focus on documenting photometer calibration tools and operation procedures for sun/sky/polar/lunar photometer calibration and quality assurance, and to document possible new observation protocols to increase aerosols retrievals frequency. During Phase 1, the activity will focus therefore on: (i) documentation on the new CE318-T photometer QA/QC procedures and best practices for site maintenance; (ii) Near Real Time delivery of the primary products (calibrated radiometry) requested for WP 4000; (iii) design of faster observation protocols more adapted to cloudy situations.
Spectrometer input
This work focus on: a) developing a set of documents to define the legal, logistical, and technical requirements to run the Pandonia network in an operational mode; b) adapting Pandonia calibration, operation, data formats and algorithms so that the GRASP algorithm can be applied to them; c) equipping one Pandora instrument with polarizing filters to be capable of retrieving angle and degree of polarization of the sky radiance as preparation to apply the GRASP algorithm on these data.
Inversion Algorithm and Interfaces
This work focus on the development of a prototype of software for automated data control and processing, that could process observation data in automated regime with minimized interaction of user. The software shall use GRASP algorithm and shall allow a variety of retrieval options to combine observations, change the retrieval assumptions, retrieval regimes, modes and content of output generation and presentation. In principle, the software should adapt full flexibility in data processing that is already implemented in GRASP retrieval package. At this stage, it is expected that the software will be ready to demonstrate the use of data from AERONET/PHOTONS and EARLINET. All these data should be treated in similar format, appropriate for straightforward control, visualization. The software also should serve as a prototype of elaborated environment for processing of the observation by GRASP algorithm. Therefore, this WP includes the efforts on preparing and linking the data to the inputs of GRASP.
System Integration and Validation
This work focus on the integration of all developed software in one backend processing infrastructure. The software is developed distributed, but integrated in one platform. Webservices and tools are implemented to integrate, run, monitor, distribute, visualize the applications. Data are stored and transformed to a standard file format.
Work Breakdown Structure – Phase 2
The project is organized in 4 Work Packages. WP1 Evolution of instruments and measurement techniques includes activities concerning consolidation and evolution of the instruments, already involved in Phase 1 (i.e.: lidars, photometers and spectrometers), and their corresponding measurement techniques. WP2 Retrieval Algorithms aim to fully integrate in the hub the retrieval algorithms based on the GRASP prototype from phase 1, and to enhance the algorithm to fully exploit the combination of available data from photometers (Cimel/Aeronet) and spectrometers (Pandora/PGN) to simultaneously retrieve advanced aerosol (optical and microphysical) products as well as trace gas products. WP3 Development, deployment and test of the full DIVA system include deployment of the DIVA operative system, starting from the Phase 1 prototype. The platform capabilities will be extended including tools for data analysis and data mining functionalities with the aim to enhance data exploitation.
Deliverables – Phase 1
| Doc ID | Title | Type of deliverable |
| D0 | Demonstration Instrument Site List | Documentation |
| D1 | DIVA Algorithm Technical Baseline Document | Documentation |
| SW4 | Lidar format converters | Software |
| PIWP | Project Web Information Pages | Other |
| D2 | Standard DIVA data formats descriptions | Documentation |
| D3 | DIVA Architectural Design Document | Documentation |
| D4 | DIVA standard calibration procedures and tools documentation | Documentation |
| D5 | Standard DIVA operation procedures documentation | Documentation |
| D6 | User manuals for DIVA Data processing tool prototype, runtime environment and validation service prototype | Documentation |
| D7 | DIVA Data processing tool prototype test report | Documentation |
| SW1 | DIVA Data processing tool prototype | Software |
| SW2 | DIVA validation service prototype | Software |
| SW3 | DIVA runtime and monitoring webservice | Software |
Deliverables – Phase 2
| Doc ID | Deliverable title | Deliverable type |
| D1 | DIVA Retrieval algorithms for Lidars | Technical Note / Algorithm Theoretical Basis Documents |
| D2 | Stand-alone DIVA retrieval algorithms for Photometers | Technical Note / Algorithm Theoretical Basis Documents |
| D3 | Photometer Aerosol products with increased sampling | Technical Note |
| D4 | Pandora products optimised for DIVA | Technical Note / Algorithm Theoretical Basis Documents |
| D5 | Calibration and data file format for Pandora | Technical Note |
| D6A | Diva (Synergy) Product List and Description | Technical Note |
| D6 | Data Processing Tool | Technical Note |
| D7 | Demonstration and updates of DIVA Ph2 Processing Tool (incl. full Pandora data integration) | Test Report |
| D8 | DIVA data analysis functionalities | Design Document |
| D9 | DIVA evolution and performance | Test Plan |
| D10 | DIVA evolution and performance | Test Report |
| D11 | Final & Evolution Report | Documentation |
| CCD | Contract Closure Documentation | Documentation |
| SW1 | Calibration verification and QA tools for Lidars | Software |
| RN1 | Calibration verification and QA tools for Lidars | Release Note |
| SW2 | Data Processing Tool | Software |
| RN2 | Data Processing Tool | Release Note |
| WS | Web Story | Other |
| PW | Project Website | Other |
Meetings – Phase 1
KoM: 2 October 2017
Technical Meeting: 9 November 2017
Preliminary Design Review Meeting: 18-19 of December 2017 in Magurele, Romania
Critical Design Review (CDR) meeting: 28 June May 2018, remote
Acceptance Review (AR) meeting: 11 September 2018, Lille, France
Meetings – Phase 2
KoM: 17 October 2019, Magurele, Romania
Progress Meeting 1: January 2020, remote
Progress Meeting 2: to be announced, Vienna, Austria
Acceptance Review meeting: to be announced, remote
Final Presentation meeting: to be announced, Frascati, Italy