VLT/VLTI Data Pipelines and Calibration Plans

Last Update: 2008 January 22

	Data pipelines
	VLT instruments: ------------------------------ CRIRES FORS1/2 FLAMES/GIRAFFE FLAMES/UVES ISAAC NACO SINFONI UVES VIMOS VISIR VLTI instruments: ------------------------------ VLTI/AMBER VLTI/MIDI

Introduction

For each VLT and VLTI instrument, ESO provides a data processing pipeline. This page contains a brief summary of their status. It also lists the current calibration plans. Users are urged to refer to the appropriate user's manuals for complete details.

Information about the public availibility of ESO pipelines, including user manuals, can be found here.

For a general statement about the use of pipelines within ESO operations, see Section 8 in the ESO Call for Proposals.

Science Product Disclaimer

The accuracy of the science products produced by the ESO pipelines can be limited both by the quality of the available master calibration products and by the algorithmic implementation of the pipelines themselves. In particular, adopted reduction strategies may not be suitable for all scientific goals. Therefore, ESO assumes no responsibility for the usefulness of pipeline reduced data for any specific scientific project.

What's New

2008 January 21: Revised for Period 82 Call for Proposals

2007 September 12: Updated CRIRES pipeline

2007 June 20: Public release of the UVES pipeline

2007 June 1: Spectroscopic FORS pipeline public

2006 August 07: Revised for Period 79 Call for Proposals

• New pipeline for FORS1/2, supporting LSS, MOS, and MXU (all grisms)
• New section for CRIRES
• VISIR and GIRAFFE pipelines public

2006 February 13: Revised for Period 78 Call for Proposals

• Interference filters offered for UVES
• Section added for VLTI/AMBER
• Full pipeline support for FLAMES/Giraffe

2005 August 12: Revised for Period 77 Call for Proposals

2005 February 23: Revised for Period 76 Call for Proposals

• pipeline support for VISIR available

2004 Sept 8: Revised for Period 75 Call for Proposals

• pipeline support for VIMOS IFU available
• pipeline support for FLAMES GIRAFFE available for many standard setups
• UVES: new standard settings with 760nm central wavelengths
• preliminary information about SINFONI is available

2004 February 24: Revised for Period 74 Call for Proposals

• pipeline support for FORS2 MOS mode available now.
• wavelength calibration in the ISAAC pipeline considerably improved
• pipeline support for MIDI available
• pipeline support for VIMOS IFU image reconstruction available for all quadrants

The CRIRES calibration plan includes day-time measurements of dark exposures, flat-fields, and wavelength calibrations. They are executed with instrument settings matching those used during the night (DITs for darks and reference wavelength of the grating for flat and wavelength calibrations).

The repeatability of the spectrograph settings between night-time and day-time is about 3 pixels. For higher precision wavelength calibration, calibration templates can be executed immediately before or after the science exposures on request by the PI. Calibration exposures either use a ThAr arc lamp or a N2O absorption cell, depending on the setting. For reference wavelengths above 2400nm, the usage of sky lines is usually best. Please refer to the CRIRES user manual for a detailed description of recommended calibration sources.

The CRIRES pipeline provides the following calibration steps: dark subtraction, correction for detector non-linearity, flat-fielding, combining of nodding images, spectrum extraction, and wavelength calibration. In general, it gives useful results for single, point-like, continuum-emitting sources with S/N (in the combined image) of higher than about 5. The pipeline supports wavelength calibration using either sky lines or dedicated calibration exposures. It also provides results from a physical model of the spectrograph which can be used in addition for wavelength calibration.

For further information about CRIRES data handling and processing, as well as CRIRES specific quality control parameters, see the CRIRES section of the ESO quality control web pages. The CRIRES pipeline is publicly available.

top FLAMES

FLAMES consists of two major modes: UVES-Fiber and GIRAFFE. These major modes are discussed separately.

The FLAMES calibration plan ensures that ESO maintains and provides master bias, dark, spectroscopic slit and fiber flat field frames. Order definition frames as well as arc lamp spectra are also provided. Flux standard stars are not observed. CCD characteristics like read-out noise and gain are measured on a monthly basis. Daytime arc lamp (ThAr) spectra are used for the wavelength calibration of science frames. When highly accurate radial velocity measurements are required with the Red 580 setting, the first of the eight fibers can be used to record an arc lamp spectrum simultaneously with the science observations. All FLAMES-UVES calibration data taken in standard modes are processed and quality checked. The results are posted on the ESO Quality Control Web pages. The following settings are supported (all in 1x1 binning):

Red 520 nm (CD#3)
Red 580 nm (CD#3)
Red 860 nm (CD#4)

The following corrections of the science spectra are available for each fiber: bias subtraction, inter-order background subtraction, flat field correction, order extraction, rebinning to wavelength scale, and order merging. The two detectors are processed independently. An optimum extraction algorithm with cosmic rejection is applied. Average extraction is also available. The FLAMES/UVES pipeline is based on procedures available within the MIDAS FLAMES context, which is available here. Support for FLAMES/UVES within the CPL-based UVES pipeline is in developement. For further information about FLAMES/UVES data handling and processing, as well as FLAMES/UVES specific quality control parameters, see the FLAMES/UVES section of the ESO quality control Web pages.

FLAMES/GIRAFFE

The GIRAFFE calibration plan foresees fiber flats and arc lamp spectra to be taken on a daily basis for each science setup used in the previous night. For ARGUS, the fiber flats are taken during the night using the Nasmyth screen. For all other modes, they are taken as daytime calibrations using the robotic arm. Bias frames are taken daily, as are fiber flats in one selected setup per fiber system. Dark frames and other technical calibrations to monitor the CCD performance are taken about monthly. Flux standard stars are measured routinely with ARGUS, and on request during the night (or in twilight) with one of the IFUs. To provide flux calibration for the other IFUs, Nasmyth flats are measured. There are also technical calibrations to monitor overall efficiency of the instrument and the accuracy of radial velocity astrometry. Pipeline support is available for all three modes (MEDUSA, IFU, ARGUS). Extracted, wavelength calibrated and flat-field corrected spectra are provided. Data are currently extracted with a fixed numerical filter. Optimal extraction is not yet available.

For further information about FLAMES/GIRAFFE data handling and processing, as well as quality control procedures and parameters, see the FLAMES/GIRAFFE section of the ESO quality control Web pages.

The calibration plan for the two FORSes is almost identical. It ensures that ESO maintains and provides detector biases on a regular basis. In imaging (IMG) mode, twilight flat field frames and photometric standard stars are regularly obtained for the UBVRI filters. From the standard star observations, zero points, first-order extinction terms, and first-order color terms are determined. Spectrophotometric stars are observed when narrow-band imaging science observations are obtained. In long-slit spectroscopic (LSS) mode, screen flats and arc-lamp frames are obtained with a one (1) arcsec slit. Additional arc-lamp exposures are obtained as needed to support requested slits. Spectra of spectrophotometric standard are also obtained for the determination of response curves. In multi-object spectroscopic modes (MOS and MXU), flat-field and arc-lamp frames are obtained for all slit configurations (MOS) or masks (MXU). The FORS pipeline is used for processing data from both FORSes. The IMG pipeline generates master bias and flat-field frames as well as photometric zeropoints. When processing science frames, it applies bias and flat field corrections.The new FORS spectroscopic pipeline produces a wavelength calibration table and and normalized flat fields for all the slits. When processing science frames, it applies bias and flat-field corrections to all slits (MOS, MXU, and LSS). The wavelength calibration table is then used to produce a wavelength distortion corrected frame with linear dispersion. In addition the spatial distortion is corrected for MOS and MXU mode. If a sufficient number of spatially distributed slitlets is present (more than 12) the global distortion is also determined. The pipeline now supports all grisms. In the SCIENCE specotroscopic observations, the object spectra are detected and optimally extracted. The spectroscopic FORS pipeline is publicly available.

Support for polarimetric data is in the development plan of the spectroscopic pipeline. It is also planned support for jittered observations, and the determination of response curves for the relative flux correction. For further information about FORS1/2 data handling and processing, as well as specific quality control parameters (e.g. zeropoints and color terms), see the FORS1 (FORS2) section of the ESO quality control Web pages.

top ISAAC

The ISAAC calibration plan ensures that ESO maintains and provides dark frames for the DITs used during the observations. In SW imaging (SWI) mode, twilight flat-field frames and photometric standards are regularly obtained for the JsJHK filters. In other SWI filters, flat-fields and standard star observations are obtained as needed. The latter is also true for LW imaging (LWI) mode. For long-slit spectroscopy, screen flats and arc-lamp frames are obtained during the day as needed for the setups (slits and grating position) used at night. Telluric absorption standards are observed as needed, see the user's manual for more details. Star trace frames are obtained periodically to determine the optical distortion in the spatial (y) direction. The response function is determined approximately once per semester via the observation of spectrophotometric standards. Detector linearity is monitored regularly for the Aladdin array.

JHK broad-band imaging with the LW-arm Aladdin array is calibrated with twilight flats and zero points.

In SW imaging (SWI) mode, the data reduction pipeline handles creation of master dark frames with the appropriate DIT, twilight flats, and photometric zero points. It supports co-addition of multiple frames taken in the AutoJitter, AutoJitterOffset and FixedSkyOffset modes.

In SW spectroscopy (SWS) mode, the pipeline handles the creation of master dark frames, spectroscopic flats, and arc frames to correct optical distortion and to apply a wavelength dispersion solution. It supports co-addition, optical distortion rectification, and wavelength dispersion correction of multiple frames taken with the AutoNodOnSlit and StandardStar modes. Startrace calibration tables are created and applied to correct the optical distortion in y-direction. In these templates, the pipeline also extracts a wavelength calibrated spectrum of the brightest source.

In LW imaging mode (LWI), the pipeline handles the creation of dark frames for the DITs used during the observation, twilight flats (in non-chopping modes) and sky flats (for chopping modes) in the filters used during the observation. It also processes detector non-linearity correction and standard star observations. It supports co-adding of multiple frames taken in modes AutoJitter, AutoJitterOffset (non-chopping modes for filters NB_3.21 and NB_3.28), AutoChopNod (chopping modes with filters L and M), and StandardStar.

In LW spectroscopy (LWS) mode, the pipeline handles the creation of master dark and spectroscopic flat-field frames, as well as the processing of arc frames (in first-order for L-band and third-order of M-band) to determine optical distortion and dispersion solution in y-direction. The pipeline supports the co-addition of multiple frames taken with the AutoChopNod and StandardStar templates in chopping mode, and the AutoNodOnSlit and StandardStarNod templates in non-chopping modes. Startrace calibration tables are created and applied to correct the optical distortion in y-direction.

The current version of the ISAAC pipeline is based on procedures available within CPL, which is available for use by external users.

The current version of the NAOS-CONICA (NACO) Calibration Plan includes following observing modes: imaging, spectroscopy, coronography, polarimetry, Simultaneous Differential Imager (SDI) and Fabry Perot Imager (FPI). The plan ensures that ESO maintains and provides dark frames for the DITs and cameras used during the night, internal lamp and/or sky flat field frames, and standard star observations. Flat fields are obtained for the default readout-modes defined for every template. The twilight flats are taken without FP, the focal plane masks, the polarimetric mask and without the polarizer.

For spectroscopy, ESO obtains spectroscopic flats (from internal lamps) in all SW spectroscopic modes, slits and readout modes. The arc-lamp exposures are taken in all spectroscopic modes and slits, however, the LW arcs are not supported. For slitless spectroscopy arcs with 86mas slit are provided. Observations of telluric standard stars, with the setup used for the science target, are performed whenever the grisms are used. For polarimetry two sets of flats are taken - with and without the Wollaston prism and the wire grids.

The current version of the NACO data reduction pipeline is based on ESO's Common Pipeline Library. Currently it only supports the imaging mode and performs coaddition of multiple frames from the AutoJitter, AutoJitterOffset and FixedSkyOffset templates. Strehl ratio measurements are computed from PSF CALIBRATOR frames and from acquisition templates as well as from observations of photometric zeropoint stars.

Presently operational pipeline support is not available for any other NAOS-CONICA mode. For further information about NAOS-CONICA data handling and processing as well as NAOS-CONICA quality control parameters, see the NACO section of the ESO quality control Web pages.

top SINFONI

The SINFONI calibration plan ensures that ESO maintains and provides dark frames for the DITs used during the observations. Spectrosocpic screen flats and arc-lamp frames with the same grating and pre-optics are taken during the day. Telluric absorption standards are observed as needed, see the user's manual for more details. Fiber frames are obtained periodically for each grating to determine the optical distortion and the positions of the slitlets in order to reconstruct the data cube. Detector linearity is monitored regularly for the array.The data reduction pipeline handles creation of dark frames, spectroscopic flats, and arc-lamp frames to identify the positions of the slitlets and to apply a wavelength solution. It extracts and applies optical distortion correction and aligning of slitlets. It supports co-addition of calibration stars (telluric standard stars and PSF calibration stars) and science observations and sky subtraction. For science stacks without embedded SKY frames the sky is extracted from the OBJECT frames.Currently the science recipe generates 3D spectrum, a series of intermediate non-coadded science cube products and sky frames.

Information about SINFONI data handling and processing, as well as SINFONI specific quality control parameters, see the SINFONI section of the ESO quality control Web pages.

The UVES Calibration Plan ensures that ESO maintains and provides bias, dark, spectroscopic flat field, and order definition frames and arc lamp spectra. Flux standard stars are observed in three dedicated settings only (346+580, 390+564, and 437+860). They are used to create master response curves (valid for relative flux calibration over certain periods) and to monitor the overall instrument efficiency. CCD characteristics like read-out noise and gain are measured on a monthly basis.

Daytime calibration lamp (ThAr) spectra are used for the wavelength calibration of science frames. When highly accurate radial velocity measurements are required, additional calibration spectra have to be taken during the night. All UVES calibration data taken in standard modes are processed and quality checked. The results are posted in the UVES section of the ESO Quality Control Web pages. The UVES pipeline supports the following standard central wavelength settings:

Blue 346 nm (CD#1),
Blue 437 nm (CD#2),
Red  520 nm (CD#3),
Red  580 nm (CD#3),
Red  600 nm (CD#3, for use with iodine cell only),
Red  860 nm (CD#4),
Dic1 346+580, Dic1 390+564, Dic1 346+564, Dic1 390+580,
Dic2 346+760, Dic2 390+760, Dic2 437+760,
Dic2 346+860, Dic2 390+860, Dic2 437+860.

As of Period 79, the new CPL-based UVES pipeline has been used both online at Paranal and offline by QC in Garching. The main improvements in the CPL-based pipeline in comparison to the venerable previously used MIDAS-based one are:

• Improved Optimal Extraction: The CPL optimal extraction algorithms solve the ripple-like artifacts seen in the MIDAS-based pipeline reductions of high signal to noise spectra.
• Faster execution times in comparison to the MIDAS-based pipeline for the same operations/algorithms.

By default, the calibration database at Paranal is populated with calibrations for the standard modes in the 225kHz,1x1,low and 50kHz,2x2,high readout modes only. For visitors observing with non-standard settings, the online pipeline at Paranal can be prepared to handle (in most cases) their settings (limited to 2 non-standard settings per visitor run). The science data are calibrated with calibration exposures obtained upon arrival of the visitor. For processing in Garching by QC, and hence for the data packages delivered to Service Mode PIs, the pipeline can, in principal, handle any instrument/readout mode combination. However, since there are in principal an infinite number of such combinations (since the central wavelength is continuously adjustable) full trust in the Quality Control process and the scientific efficacy of any delivered pipeline products can only be put in those for the standard instrumental setups (as above) in the 225kHz,1x1,low and 50kHz,2x2,high readout modes. There are also some known specific limitations of the current version of the UVES CPL-based pipeline:

• Problems with order tracing for the 625kHz,1x1,low readout mode, probably due to the missing column between the two halves of each chip readout by the two amplifiers for each chip.

The following corrections of the science échelle spectra are available: bias subtraction, inter-order background subtraction, flat field correction, order extraction, sky subtraction, rebinning to wavelength scale, order merging, and (relative) flux calibration. All three detectors (1 blue, 2 in the red mosaic) are processed independently. For point sources, an optimum extraction algorithm with sky subtraction and cosmic rejection is applied, either an analytical cross-order profile or a virtual profile with a sampling of either 5 or 20 points per pixel is used according to an initial estimate of the signal to noise being low (less than 10), medium (10-200) or high (above 200) respectively; image slicer data are extracted as the sum over the slicer length, no sky subtraction is available here. For extended sources, by default a two-dimensional extraction is done and in addition a 1-D optimal-extraction is attempted, however optimal extraction occasionally fails for truely extended sources when the object completely fills the slit and the pipeline has trouble to estimate the sky background. Generally, the UVES pipeline provides useful results under the following requirements:

• point-like, continuum-emitting source well centered on the slit
• S/N ratio from 2 to 400

With Period 77, a set of eight interference filters is offered in Visitor Mode. The filters can be used in the red arm to isolate certain echelle orders so that the full slit length of 30 arcsec can be chosen. Reduction of data obtained with interference filters is currently not supported by the pipeline. Both the historical MIDAS-based UVES pipeline and the new CPL-based pipeline are available for use by external users, see here. For further information about UVES data handling and processing, as well as UVES specific quality control parameters, see the UVES section of the ESO quality control web pages.

top VIMOS

The VIMOS calibration plan ensures that ESO maintains and provides all necessary calibration files on a regular basis. Detector bias frames for all operational modes of the instrument are measured daily. The dark current is determined on a monthly basis. In imaging (IMG) mode, twilight flat field frames and observations of photometric standard stars are regularly obtained for UBVRI filters. For the Gunn z filter, OBs for photometric calibration have to be provided by the Service Mode observer. From standard star observations, zero points (nightly), first-order extinction terms (several times per year), and first-order color terms (several times per year) are determined. In multi-object spectroscopy (MOS) and in integral field unit (IFU) modes, flat-field and arc-lamp frames and spectrophotometric standard star observations are obtained for all instrument settings. For imaging science data, the VIMOS pipeline applies bias and flat field corrections, cleans cosmic rays, and applies a photometric calibration. Co-addition of multiple frames obtained in jitter mode is currently not supported. In the special case of pre-imaging, the pipeline applies bias and flat field corrections only. To accelerate the pre-imaging delivery process, archival master bias and flat-field frames are used, not the master frames for the specific night. Therefore, processed pre-images may not be suitable for detailed photometric analysis. The pipeline also updates the Sky-to-CCD transformation matrix in the header into a format readable by the VIMOS mask preparation tool (vmmps). Processed pre-imaging frames will typically be made available to users via anonymous FTP within 24 hours of data acquisition. ESO will notify the principle investigator whenever pre-imaging frames are ready for retrieval. The MOS part of the VIMOS pipeline processes bias, flat-field, arc lamp, standard star, and science frames. For science frames, bias and spectral curvature corrections as well as wavelength calibrations are applied. Fringing correction is performed for LR_red and HR_red gratings. The detected objects in each slit are extracted. Correction for instrument response is currently not applied. Co-addition of frames obtained in jitter mode is supported. The following MOS science products are provided:

• one FITS file containing the one-dimensional extracted spectra (one per line); one FITS file containing the one-dimensional sky spectra (one per line) that have been subtracted; one FITS file containing the background subtracted two-dimensional object spectra; one FITS file containing the two-dimensional background spectra that have been subtracted; two FITS tables containing information on object positions;
• one FITS file containing a map of the sky fringes (LR_red and HR_red only).

The IFU part of the pipeline processes bias, flat field, arc lamp, standard star and science frames. Science frames are corrected for bias and relative fiber transmissions. Wavlength calibration is applied and the spectrum of each fiber extracted. Flat fielding and fringing corrections and instrument response curves are currently not applied. The reduced fiber spectra are used to reconstruct the image field of view. Co-addition of frames obtained in jitter mode is not pipeline-supported.The following IFU science products are provided:

• one FITS file containing the one-dimensional extracted fiber spectra (one per line);
• one FITS file containing the reconstructed field of view.

The VIMOS pipeline is publicly available. For further information about VIMOS data handling and processing, as well as VIMOS specific quality control parameters, see the VIMOS section of the ESO quality control web pages.

top VISIR

As the mid-IR observations depend strongly on the ambient conditions the VISIR calibration plan ensures that ESO maintains and provides observations of necessary standard stars.

For imaging, observations of photometric standards are obtained within a time interval of three hours from the science data. A special template is used with pre-defined settings. Filter and pixel scale are adjusted according to science observations.

For spectroscopy, ESO provides spectro-photometric observations of telluric standard star in the low resolution mode, with an airmass difference no larger than 0.2 with respect to the science target. Such a measurement is taken at least once per night, per instrument configuration. Again, a template with pre-defined instrument setting is used and the wavelength as well as slit width keys are set to match the science observations. The observatory does not provide calibrations for VISIR medium and high resolution spectroscopy. If necessary, they have to be supplied by the observer.

Starting from period 79 new, BURST observing mode is made available, so far, only for visitors.

The current version of VISIR pipeline supports most of the offered imaging and spectroscopic modes. The only ones that are not supported are the RASTER and BURST modes. The products of pipeline recipes include reduced image or spectrum, image contribution map and parameter files in case of standard star recipes.

The public version of VISIR pipeline, as well as the user's manual, are available at the VLT Instrument Pipelines webpage http://www.eso.org/pipelines. For further information about VISIR data handling and processing, as well as VISIR specific quality control parameters, see the upcoming VISIR section of the ESO Quality Control web pages.

top VLTI/AMBER

The calibration plan ensures that adequate calibrations are obtained to reduce the science data. This include the calibration of the P2VM (Pixel To Visibility Matrix) and the observation of astronomical calibrators. Data are taken to measure the Bad Pixel Map and the Flat Field Map, as well as the position of each beam for the different instrument settings. The current version of the pipeline supports all the modes offered in Visitor and Service Mode.

top VLTI/MIDI

The MIDI calibration plan ensures that calibration data are obtained on a regular basis. These calibrations include technical calibrations to monitor the health of the instrument (detector, dispersive elements) and the observations of astronomical calibrators in order to define the characteristics of the instrument and to calibrate the visibility points of the scientific objects. They also include the specific calibrations for the different instrument modes: photometric observations of the object taken after the fringe files for the HIGH SENS mode and data for the determination of the Kappa Matrix for the SCI PHOT mode. There are pipeline recipes for the technical calibrations and to process raw data taken with the MIDI instrument mode HIGH_SENS using the PRISM or the GRISM. The pipeline produces several products depending of the category (calibrator or science). For both type of objects, calibrated raw dispersed visibility points are produced. In the case of a calibrator present in the VLTI database of calibrators, a dispersed instrumental transfer function is estimated by the pipeline and used to calculate the dispersed calibrated visibility on a science object.For the SCI PHOT mode, a Kappa Matrix is provided by the pipeline, the application of this matrix to the data is still under development.

For further information about MIDI data handling and processing as well as MIDI quality control parameters, see the MIDI section of the ESO quality control Web pages.