PIPELINES AT ESO

This page provides information about pipeline processing and data types.

Raw data are selected, associated and inserted into a reduction mechanism which produces calibration products, science products and quality control information. This mechanism is the data processing pipeline. There is one such pipeline for each VLT and VLTI instrument.

Find general information about ESO reduction pipelines here.

The main functionalities of the pipelines are:

• create master calibration data,
• reduce science frames,
• extract QC information from the data.

QC Garching creates

• master calibration data from raw calibration data, both for Visitor Mode and Service Mode runs . The raw data are stored in the ESO Archive and are public. They are assessed and selected in terms of their quality and used for QC checks and for trending;
• reduced science data (currently only for Service Mode runs) . For SM programmes, QC Garching runs the pipeline on all science raw data, using the best available, quality-checked master calibration data.

Science data are processed by the pipeline with the best available calibration data. Please note that ESO is not assuming any responsibility in respect to the usefulness of the reduced data. The adopted reduction strategy may not be suitable for the scientific purpose of the observations.

OPERATIONS

There are two instances of the data reduction pipelines:

• at the instrument workstation on Paranal, running in automatic mode,
• at HQ Garching, run by the Quality Control Group in the optimized mode.

The automatic mode is used for quick look purposes and for on-site quality control. It processes all raw data sequentially, as they arrive from the instrument. If calibration products ("master calibrations") are required for processing science data, these are taken from a database with standard, pre-manufactured calibration products. The automatic mode is not tuned to obtain the best possible results.

The optimized mode is the mode, which uses all data of a night, including the daytime calibrations. The calibration data are sorted and grouped according to their dependencies. Master calibration data are created. Their quality is checked. The certified, closest-in-time products are finally associated to the science data of a night. This is the way all calibration data in supported modes are processed by QC Garching, as are all science data from Service Mode programmes.

The ISAAC data processing pipeline is operational since April 1999 (ESO observing period 63).

The ISAAC pipeline is publicly available (check out here). Under this link you also find the pipeline Users Manual.

Find the description of ISAAC data processing and pipeline recipes here:

ISAAC pipeline
calibration
science reduction

Both in Service Mode and Visitor Mode, ISAAC observations are supported in the following instrument modes:

• SW-arm Imaging
• SW-arm Spectroscopy
• LW-arm Imaging (Jitter and Chopping)
• LW-arm Spectroscopy (Jitter and Chopping)

The following instrument modes are not supported by the pipeline:

• SW-arm Imaging Polarimetry
• LW-arm FastJitter and Burst mode

QC Garching currently processes the following ISAAC data:

• all calibration data, both in Service and Visitor Mode;
• all science data taken in Service Mode.
  

We maintain a log file showing the link between the pipeline versions and the date from which on it was used for.

Raw data. The short-wavelength arm of ISAAC is equipped with a Hawaii array with 1024 x 1024 pixel. The long-wavelength arm has an Aladdin array of 1024 x 1024 pixel.

The Hawaii array is operated either in double correlated or in nondestructive read mode. The file format is a single FITS frame with a single header unit. The Aladdin array is operated in uncorrelated read mode at high bias or in double correlated read mode. The latter one operates in high bias or low bias read mode. The file format of the LW-arm Aladdin array depends on the instrument mode. Table 10 of the ISAAC User Manual shows the links between detector operations mode and instrument mode.

• In Imaging and Spectroscopy using Jitter, a single FITS frame with two axes and a single header unit is written.
• All calibration raw frames use the same format (a single FITS frame with two axis and one header unit), with two exceptions: the spectroscopic internal lamp flat for the setup using the LR-grating in the M-band at 4.75 micron uses a 512 x 1024 pixel window read out for technical reasons, and the telluric standard stars in chopping mode use the chopping specific file format (see next point).
• In Imaging and Spectroscopy using Chopping, a single FITS frame cube with three axis and a single header unit is written. The cube has a size of 1024 x 1024 x 2 pixel, of which the first layer contains the data of the first chopping position and the second layer of the data cube contains the pixel data of the second chopping position.
• In FastJitter and Burst mode a single FITS data cube is written with one header unit. The number of pixel in each axes depends on the specified window size of the readout (first and second axis) and the user defined time resolution and total time (third axes).

The ISAAC Users Manual has also a detailed description of the ISAAC raw data structure.

Products. Pipeline products have varying formats.

• Imaging products (Jitter and Chopping) are FITS frames with two axis. The number if pixel can be slightly larger than in the raw frames in order to cover the whole sky field on which the telescope jittered or chopped.
• Calibration products have the same file format and number of pixel as the raw frames.
• Spectroscopy products (Jitter and Chopping) have the same file format as the raw frames.
• In addition to FITS images the pipeline generates binary FITS tables that contain numerical results like coefficients or extracted spectra. More details are described here.

Since April 1999 (ESO observing period 64) , observing runs performed in Service Mode receive a data package containing

• all raw science data,
• reduced science data (if applicable),
• associated calibration products,
• associated raw calibration data,
• listings and logs,
• QC reports OB reports.

  SM package