PIPELINES AT ESO

This page provides information about pipeline processing and data types.

Raw data are selected, associated and inserted into a reduction mechanism which produces calibration products, science products and quality control information. This mechanism is the data processing pipeline. There is one such pipeline for each VLT and VLTI instrument.

Find general information about ESO reduction pipelines here.

The main functionalities of the pipelines are:

• create master calibration data,
• reduce science frames,
• extract QC information from the data.

QC Garching creates

• master calibration data from raw calibration data, both for Visitor Mode and Service Mode runs . The raw data are stored in the ESO Archive and are public. They are assessed and selected in terms of their quality and used for QC checks and for trending;
• reduced science data (currently only for Service Mode runs) . For SM programmes, QC Garching runs the pipeline on all science raw data, using the best available, quality-checked master calibration data.

Science data are processed by the pipeline with the best available calibration data. Please note that ESO is not assuming any responsibility in respect to the usefulness of the reduced data. The adopted reduction strategy may not be suitable for the scientific purpose of the observations.

OPERATIONS

There are two instances of the data reduction pipelines:

• at the instrument workstation on Paranal, running in automatic mode,
• at HQ Garching, run by the Quality Control Group in the optimized mode.

The automatic mode is used for quick look purposes and for on-site quality control. It processes all raw data sequentially, as they arrive from the instrument. If calibration products ("master calibrations") are required for processing science data, these are taken from a database with standard, pre-manufactured calibration products. The automatic mode is not tuned to obtain the best possible results.

The optimized mode is the mode, which uses all data of a night, including the daytime calibrations. The calibration data are sorted and grouped according to their dependencies. Master calibration data are created. Their quality is checked. The certified, closest-in-time products are finally associated to the science data of a night. This is the way all calibration data in supported modes are processed by QC Garching, as are all science data from Service Mode programmes.

The GIRAFFE pipeline is able to support both the old and the new CCD, with some modified static calibration files describing the changes in alignment. The geometry of the new CCD is the same as for the old (2kx4k). It features increased efficiency in the red, and reduced fringing. It is in operations since 2008-05-26. Last science night with the old CCD is 2008-03-13. The main changes for the pipeline is a modified strategy for removing the BIAS.

The GIRAFFE data processing pipeline is operational since June 2004. A list of existing and supported setups is maintained here.

The GIRAFFE pipeline is publicly available (check out here). Under this link you also find the pipeline Users Manual.

It is based on the GIRAFFE Baseline Data Reduction System (BLDRS) of the Observatoire de Geneve.

Find the description of GIRAFFE data processing and pipeline recipes here:

GIRAFFE pipeline
calibration
science reduction

Both in Service Mode and Visitor Mode, GIRAFFE observations are supported in certain standard settings:

GIRAFFE standard settings

QC Garching currently processes the following GIRAFFE data:

• all calibration data, both in Service and Visitor Mode;
• all science data taken in Service Mode.
  

Raw data. GIRAFFE has a single CCD with 2048 photon-sensitive columns and 4096 rows. To these add 50 pre-scan and 50 post-scan columns. Until 2008-03-13, the CDD was an EEV chip optimized for the blue ("Bruce"). Since 2008-05-26, GIRAFFE has an e2v CCD ("Carreras") with much higher red efficiency and lower fringing.

Find more details about the GIRAFFE CCD format and properties here.

In total, GIRAFFE raw frames have 2148x4096 pixels. Pixel size is 15x15 microns. There is only one read-mode available, which is unbinned, low gain. Raw frames have a size of 17.4 MB each. Find example (reference) frames here.

Bruce had three columns with hot pixels, at x=416-418, starting at y=1466. Carreras has none.

Extensions. Raw data come as FITS files with three HDUs (header units). The main plane (HDU1) has the header with all primary keywords and telescope, positioner, detector, observation, instrument etc. keywords, as well as the pixel data. Two extensions (HDU2 and HDU3) host binary tables. The first binary table is the OzPoz table. It includes the association of the objects and the fibre buttons, and also contains fibre positioner keywords. This is the PAF file generated by FPOSS (the fibre positioner observation support software tool), with some OzPoz information added (such as R, Theta and errors).

The second binary table contains the fibre description and the association between fibre buttons and fibre position in the slits, and also includes lab-measured fibre transmission values. This table is static.

Find here a description of the GIRAFFE binary tables. The Users Manual has also a detailed description of the GIRAFFE raw data structure.

Products. Pipeline products have varying formats. Typical for almost all products beyond the extraction step is that the X axis records the fibre index, while the Y coordinate records pixels or wavelength bins. Hence typically there is one column per fibre. More details are described here.

Except for the BIAS and DARK calibration frames, all GIRAFFE data come in a specific data format which is due to the fibre systems being projected onto the detector. The format is described here.

The GIRAFFE spectrograph receives light from the fibre positioner called OzPoz. It consists of two plates which are functionally identical. Each plate hosts three fibre systems (Medusa, IFU, UVES), one plate accomodates another IFU system called 'Argus'.

The Medusa system has nominally 137 fibres which are projected into one slit called Medusa<n> (where n is 1 or 2). The slit is subdivided into

• 8 subslits with nine object fibres each
• 5 subslits with 12 object fibres and 1 calibration fibre each.

The calibration fibres can be used for simultaneous calibrations (SimCal mode) or as object fibre (OzPoz mode).

Find more about the MEDUSA spectral format here.

Schematic layout of the MEDUSA spectral format. Blue solid lines: object fibres. Red dotted lines: calibration fibres. The directions of the increasing fibre number in the slit (FPS) and of increasing wavelength are indicated.

The IFU fibre system has 320 fibres being projected into one slit called IFU<n> (where n is 1 or 2). The slit is subdivided into

• 10 subslits with 20 object fibres and 1 sky fibre each,
• 5 subslits with 19 object fibres and 1 sky fibre and 1 calibration fibre each.

The calibration fibre can be used for simultaneous calibrations (SimCal mode) or as object fibre (OzPoz mode).

Find more about the IFU spectral format here.

Schematic layout of the IFU spectral format. Blue solid lines: object fibres. Red dotted lines: calibration fibres. Green dots: sky fibres. The directions of the increasing fibre number in the slit (FPS) and of increasing wavelength are indicated.

The Argus fibre system is offered since P72 (October 2003). It is similar to the IFU system. It also has 15 sky fibres.

Find more about the ARGUS spectral format here.

Schematic layout of the ARGUS spectral format. Blue solid lines: object fibres. Red dotted lines: calibration fibres. Green dots: sky fibres. The directions of the increasing fibre number in the slit (FPS) and of increasing wavelength are indicated. Note that the direction of FPS is inverted with respect to IFU and Medusa.

Since June 2004, runs performed in Service Mode receive a set of DVDs containing

• all raw science data,
• reduced science data (if applicable),
• associated calibration products,
• associated raw calibration data,
• listings and logs,
• QC reports and OB reports.

  SM package

Runs finished and delivered earlier have received raw data only.