Pipeline Processing

 

Pipeline processing will be available for a subset of the instrument modes described by standard Templates. Pipeline procedures will be subject to extensive testing and qualification procedures for robustness and speed. The header structure of the data will be used by the Data Organizer to classify and associate the incoming Frames, identify Reduction Recipes and create Reduction Blocks, as described in [6]. There will not be one but several pipelines located in Garching and Paranal and executing different kind of procedures. The locations indicated hereafter correspond to operations in classical and service observing modes.

• Calibration Pipeline (Garching)
  Will be used to process the data acquired during Technical Programs and prepare pre-calibrated solutions (aka Derived Data). Derived data may be prepared as well using calibration data generated during the observing runs. Derived data will be controlled by the DFIR before submission to the archive.
• Reduction Pipeline (Paranal)
  Quasi real-time calibration of scientific data obtained by supported templates. The pipeline operates mostly without supervision. Results are looked at by the Service Observer and/or the Observer Astronomer. The Reduction pipeline applies Reduction Rules which by default will select the derived data to be applied to the data. This guarantees robustness and stable performance of the pipeline. The possibility to produce on-the-fly derived data will be evaluated during the prototyping phase. The Reduction pipeline controls its own execution and reports are displayed at the telescope on a user interface and may be replicated on a central DFS message server.
• Quality Control Pipeline (Paranal)
  Quality assessment of the data involving instrument performance control (e.g calibration unit testing, exposure levels, flexures, read-out noise), and observational conditions assessment (e.g. sky background level, seeing). When calibration data are taken during an observing program, the quality control pipeline will verify the adequacy of the derived data for this configuration. Quality control involves on-line and off-line verifications, and the quality control pipeline implements the on-line part only (see Section 3.5). When calibration data are taken during an observing program, the quality control pipeline will verify the adequacy of the derived data for this configuration. If a problem appears, immediate notification will be given to the Service Observer and/or the Observer Astronomer, and the Quality Status of the observation will be marked. Quality control reports will be subsequently verified by the Data Flow Instrument Responsible. In the case scientific data appears corrupted or cannot be calibrated, the observation may be rescheduled.
• Observatory Pipeline (Garching)
  Calibration pipeline based on improved derived data and procedures (re-calibration, on-the-fly calibration, final archive preparation). Results of this pipeline are verified by the Data Flow Instrument Responsible. This pipeline can access all the data contained in the Science Archive.

  
Figure 2: Pipeline and Quality Control processes

Fig. 2 shows the most important instances of the pipeline involved in the DFS. For the sake of simplicity of the diagram, not all relationships have been shown, in particular between the reduction and quality control pipelines in Paranal.

A Data Organizer will classify the incoming Frames based on the FITS header to associate them to the relevant Derived data and generate Reduction Blocks ([6]). The relevant Derived Data are not hard-coded in the system but derived from the application of rules. This allows exported versions of the pipeline to process user data.