This page provides information about pipeline processing and data types.
Raw data are selected, associated and inserted into a reduction mechanism which produces calibration products, science products and quality control information. This mechanism is the data processing pipeline. There is one such pipeline for each VLT and VLTI instrument.
Find general information about ESO reduction pipelines here.
The main functionalities of the pipelines are:
QC Garching creates master calibration data from all raw calibration data. The raw data are stored in the ESO Archive and are public. They are quality-checked and used for data reduction and for trending.
Before October 2011 QC Garching processed science data, using the best available, quality-checked master calibration data. As of October 2011 this service is not offered anymore.
There are two instances of the data reduction pipelines:
The automatic mode is used for quick look purposes and for on-site quality control. It processes all raw data sequentially, as they arrive from the instrument. If calibration products ("master calibrations") are required for processing science data, these are taken from a database with standard, pre-manufactured calibration products. The automatic mode is not tuned to obtain the best possible results.
The optimized mode is the mode, which uses all data of a night, including the daytime calibrations. The calibration data are sorted and grouped according to their dependencies. Master calibration data are created. Their quality is checked.
The latest version of the SINFONI pipeline is publicly available. The pipeline and its "Users Manual" can be downloaded from here.
The complete reduction of SINFONI data is done at QC in Garching in a semi-automatic manner. Raw data is selected and combined into its associated types and then processed by the pipeline to produce calibration produced and reduced science data. The products and their associated pipeline recipes can are explained in the following links:
A useful reference for getting started with SINFONI data reduction is the SINFONI data reduction cookbook (NOTE: this link opens as a PDF file).
Virtually all sinfoni observational modes and techniques are supported by instrument operations and by the pipeline processing. The only exception to this is the use of the sinfoni sky spider and spectral dithering.
The sinfoni pipeline is constantly being improved on to provide the best possible data reduction. The most recent public release, as well as complete documentation can be found at: http://www.eso.org/pipelines
Current pipeline projects include extinction values and zeropoints from the nightly telluric standards, as well as a correction for the atmospheric dispersion within the sinfoni data cubes (i.e. atmospheric dispersion can result in a small shift in the position of a source as a function of wavelength with the data cube. This is most apparent in the largest wavelength coverage and the smallest pixel scale: H+K and 25 mas.
To process the data of Period 87 beginning with dates after 2011-06-08 the sinfoni pipeline version 2.2.7 is being used.
The following is a summary of previous releases of the sinfoni pipeline. Pipeline versions given with a date are those versions used to process data within Garching Quality Control.
2011-06-08 (P87) version 2.2.7: This pipeline version now corrects the bad rows in SINFONI raw data that is introduced during the instrument read-out (see the SINFONI data reduction manual pg. 35).
2009-12-09 (P84) version 2.1.0: This pipeline version incorporates a major improvement in the sinfo_rec_jitter recipe. For all grism (J, H, K, and H+K) and for the pixel scales 25 and 100 mas, the data cubes are now corrected for atmospheric dispersion. Depending on the airmass and ADA rotation angle the source centroid can shift from one wavelength end of the cube to the other by several pixels. For the largest wavelength range cube (i.e. for the H+K band) and at the smallest pixel scale, this shift could be as large as 6-8 pixels. This shift is now corrected within sinfo_rec_jitter and is applied to the telluric STD star cubes, the PSF standard star cubes, and to all science cubes. For the 250 mas pixel scale, no correction is applied. This is due to the fact that the shift from atmospheric dispersion is less than 1 pixel at this pixel scale. The atmospheric dispersion correction relies on additional fits tables that provide the size of the correction based on airmass, waveband, and pixel scale. These files are available in the pipeline distribution (2.1.0) at the bottom of this page.
2009-09-15 (P83) version 2.0.6/2.08: Bug fixes including repair of missing sinfo_irplib_cpl_wrp.h
version 2.0.5: Bug fixes and repair of some compiler warnings. Fix of some unit test failures.
version 2.0.4: Increased mosaic_max_size default to 14196. This is relevant for science OB's made up of large numbers of individual exposures. This parameter is set in the pipeline configuration file sinfo_rec_jitter.rc and is called sinfoni.objnod.mosaic_max_size. The maximum allowed size for cube mosaics is set as 100 * sinfoni.objnod.mosaic_max_size. The maximum allowable size will strongly depend on the amount of RAM available in your computer.
version 2.0.3: Preliminary implementation of the routine sinfo_utl_efficiency (still to be fully verified) to compute instrument efficiency using telluric standard stars.
version 2.0.2: Improvement in the speed of the kappa-sigma clipping (in the sinfo_rec_jitter recipe).
2009-04-19 (P83) version 2.0.1: Fixed problems in the data reduction of fibre-fed PSF frames. Now returns cpl_error_code sinfo_baryvel.h and sinfo_baryvel.c. Will now properly exit if baryvel fails: sinfo_new_cubes_coadd.c
2009-04-06 (P83) version 2.0.0: Fixed segmentation fault (apparently only appears on ubuntu installations) due to improper array sizing. Now set to -100 QC.STREHL if by chance the value is NAN.
2008-10-17 (P82) version 1.9.8: From this version of the sinfoni pipeline onward, it is now possible to correct the horizontal lines in sinfoni raw frames. These occur during read-out and arise because the hardware only removes the bias offset using an "average" in the overscan area. The pipeline implements the IDL routine described on the final page of the sinfoni cookbook (WARNING: This link opens as a PDF file).
2008-08-01 (P81) version 1.9.5
2008-02-14 (P80) version 1.9.3
2008-01-10 (P80) version 1.8.8
2008-10-03 (P80) version 1.8.4
2007-09-26 (P78) version 1.8.3: Primarily an improvement to the distortion and linearity recipes (also pipeline products did not have the MJD-OBS of the first fits file in the AB)
2007-04-29 (P78) version 1.7.1/1.7.2: Minor bug fixes (repair of OBSJIT pipeline products that had no PRO.CATG headers. . . these now have PRO.CATG=SKY_DUMMY).
2007-02-03 (P77) version 1.7.0: Primarily installed for a more robust OH sky subtraction and an improved strehl ratio calculator.
2007-01-01 (P77) version 1.6.0: Implements a more robust distortion recipe.
2006-11-24 (P75) version 1.5.1: Improvements in the distortion recipe.
2006-08-27 version 1.4.0
2006-07-01 version 1.3.8: An improvement of the "staring mode" data reduction has been implemented
2006-06-06 version 1.3.7: The distortion recipe (sinfo_rec_distortion) and the determination of the slit position have been improved
2006-04-01 version 1.3.1
2006-02-26 version 1.3.0
2006-02-20 version 1.2.8
2005-07-20 version 1.0.8: (for science from 2005-06-15 onward, for calibrations of 2005-06-24 onward). The size of the science COADD products has been minimized. The SKY_MED product is corrected for optical distortion and flat-field. The median collapsed cube products have WCS included.
2005-07-11 version 1.0.7: (for science of 2005-06-14 onward, for calibrations of 2005-06-19 onward). The first frame of the stack is no longer necessarily in the center of the COADD product. Several new QC parameters introduced (linearity). Standard star and PSF products have been modified.
2005-06-17 version 1.0.6: (for science of 2004-04-28 onward, for calibrations of 2005-05-23 onward)
2005-05-24 version 1.0.5: (for science data of ~2005-04-05 onward)
2005-05-13 version 1.0.4: (for data of ~ 2005-03-30 onward)
2005-05-01 version 1.0.3 (for data of 2005-03-13 onward)
2005-02-16 version 0.5.9 (for data of 2004-11-24 onward)
2005-01-01 version 0.5.2
2004-10-01 version 0.4.8
After February 2005, SINFONI raw frames come in the following format:
Before February 2005, SINFONI raw frames come in the following format: