mirror sites: PL (internal link) HQ     [?]

Data Processing and Quality Control

CAL | HC | refs | QC
Trending & QC1
science recipe
Data Management
QC links:

GIRAFFE pipeline: science recipe

SCIENCE frames: giscience

Since 2014, more than 1 mio. science-grade 1D spectral data products are available for the entire data history of GIRAFFE Medusa1/2 mode. New data are added on a monthly basis; data products older than one year are freely accessable, younger data products are available to PIs. Find the phase3 spectral archive interface here. More ...

Historical note: Until the end of P87 (September 2011) science data were processed by QC Garching with the best possible (certified) calibrations solutions. The products were ingested into the Science Archive and delivered to the PIs. This service was terminated in October 2011.

The documentation of the GIRAFFE science recipe is kept here for its heritage value. All statements about science processing by QC refer to the status before October 2011.


[science.gif 65K]
Raw science frame (Medusa mode); old CCD "Bruce" new CCD "Carreras"


  • All Service Mode data in P80 (2007-10 ... 2008-03) have the CCD glow background correctly removed (the bright top glow in the left image). Before that period, background removal was not enabled; after that period, there is no need since the new CCD has no such glow.
  • Processed science data from 2008-01-18 and later, taken with the SimCal lamp on, have the wavelength scale differentially corrected with the SimCal fibre information.
  • As of 2008-04-01, all SimCal data also have information provided (in their product binary table) about barycentric, heliocentric and geocentric RV corrections (in km/s) per fibre. Note that these values have not been applied.

Recipe. The pipeline recipe giscience performs the full reduction of GIRAFFE science frames.

Scheme. The following steps are performed:

  • old CCD (up to 2008-03-13): de-bias with a scaled master bias (--bsremove-method=ZMASTER)
    new CCD (since 2008-05-26): de-bias with the fitted bias function taken from the OVERSCAN region (pixels 2-49, fit is a 5th order polynomial) (--bsremove-method=PROFILE+CURVE --bsremove-yorder=5)
  • remove background as defined by a master dark scaled by exposure time (old CCD only)
  • find and extract the fibres, using the localization and width solutions from the flat field
  • divide extracted science spectrum by extracted and normalized flat field
  • correct science spectra for differences in the fibre transmission (as recorded in the flat field)
  • wavelength-calibrate and resample science spectrum, using the dispersion solution
  • if SimCal has been used, the wavelength scale is differentially corrected (since 2008-01-18)

The GIRAFFE pipeline run by QC Garching on science data until October 2011 used certified and closest-in-time calibration solutions, mostly taken during daytime following the science night. As of 2008-01-18, the recipe took into account the signal in the simultaneous calibration fibres (OBJECT,SimCal data). Data taken earlier do not have this correction enabled.

Extraction. The fibre signal is extracted by summing up (average extraction) the signal based on the localization solution of the flat field (PLOC file, pro.catg = FF_LOCCENTROID). Note: the signal from the last one or two fibres does usually not fall completely onto the chip. The pipeline suppresses these fibres completely.

The pipeline version 2.5 and later can provide optimum extraction and average extraction. For operational reasons, science data delivered by QC always used the average extraction. That extraction mode is operationally more stable but cannot suppress cosmic rays. The optimum extraction mode requires a modified calibration product scheme.

[last_fibre.gif] Incomplete last fibre (right); the extracted fibre signal is marked red. This is an extracted view of a raw file, highly compressed in vertical direction in order to emphasize the fibre signal curvature

Flat-fielding. The extracted spectra are divided by the corresponding spectra of the flat field (PFEX file, pro.catg = FF_EXTSPECTRA). The flat-field signal is normalized such that the collapsed spectrum of the object fibre with the highest transmission is set to 1, and all other object fibres are scaled relative to that fibre signal. SIMCAL fibres in the flat-field are excluded from the statistics and always set to 1, while the SKY fibres get scaled like the object fibres. The extracted spectrum has the instrument response curve removed (at the price of now being affected by the lamp continuum which is however very smooth), and to a large extent also the fringing. The figure below demonstrates the effect of flattening: the 20nm scale ondulations are caused by the instrument response, while the ripples on the 1-2 nm scale are due to fringing. Both effects have disappeared after flattening.

[flat_fielding.gif] Extracted spectrum from a selected fibre (Argus LR773.4, @164). Top: without flat-field correction; bottom: with flat-field correction

Response. There is no response correction done by the pipeline, nor is it currently foreseen. In Medusa mode, no standard stars are taken, in IFU mode only on request of the user. In Argus mode, spectrophotometric standard stars are measured routinely. These are presently processed like science data. Their products are included in the data packages and can be used to derive an approximate flux calibration, using the fibres with the STD signal to derive the response curve, while the transmission scaling of the other fibres is provided by the flat field data.

Transmission. The fibre-to-fibre response differences are corrected using information from the fibre flats (the normalized factors from the above section "Flat-fielding"). Their values are stored as column TRANSMISSION in the fibre_setup table (extension 2 of the product files) and can be used to undo the correction.

Sky correction. The sky signal is recorded in the dedicated SKY fibres in the IFU and Argus modes. In Medusa mode, the user is completely free to record the sky signal in any object fibre. Since the pipeline recipe in general does not know exactly which fibres correctly record the sky signal, the sky signal is extracted by the recipe just as for any other fibre, and sky subtraction is left to the user.

Background subtraction. For data taken between October 2007 and March 2008, the recipe has subtracted the background as recorded in a master dark frame. For data later than that, there is no need for subtraction (no glow in the new CCD).

The master dark is constructed from at least 3 input raw darks (one hour exposure time each) and scaled properly for exposure time.

SimCal fibre correction. If the simultaneous calibration fibres have been used, their signal is used to incrementally correct the wavelength scale. The amount of the correction is available as column WLRES in the fibre_setup table (extension 2 of the product files) and can be used to undo the correction.

Since 2008-04-01, all SimCal science data also have information provided (in their product binary table) about barycentric, heliocentric and geocentric RV corrections (in km/s) per fibre. These corrections have not been applied by the pipeline.

Products. The following science products are created by the pipeline:

product category (PRO CATG)* product index** product code*** delivered? format comments
SCIENCE_REDUCED 0006 SRED no 2D (pixel space) debiased raw frame
SCIENCE_EXTSPECTRA 0002 SEXS yes 2D, pixels vs fibre index extracted spectra, one column per fibre
SCIENCE_EXTERRORS 0000 SEXE yes same as SEXS corresponding extraction error
Main product: SCIENCE_RBNSPECTRA 0005 SRBS yes 2D, wavelength bins vs. fibre index rebinned extracted spectra, one column per fibre
Main product: SCIENCE_RBNERRORS 0004 SRBE yes same as SRBS corresponding extraction error
SCIENCE_RCSPECTRA 0008 SIMG yes 2D, spatial coordinates reconstructed image (IFU and Argus)
SCIENCE_RCERRORS 0007 SIME yes same as SIMG corresponding error (IFU and Argus)
SCIENCE_CUBE_SPECTRA 0009 SCUB yes 3D, spatial and spectral coordinates data cube with full spatial and spectral information (Argus only)
SCIENCE_CUBE_ERRORS 0010 SCUE yes same as SCUB corresponding error (Argus only)

* coded as HIERARCH.ESO.PRO.CATG in the fits header
** index of the PIPEFILE name, coded as PIPEFILE in the fits header
*** used in the delivered name

The error files (PRO.CATG = SCIENCE_EXTERRORS etc.) contain the calculated standard deviation per pixel of the SCIENCE_EXTSPECTRA file (containing: photon noise, read noise; the flat-field extraction errors is not included here). This error is propagated to the error files of the rebinned spectrum (SCIENCE_RBNERRORS) and, for IFU and Argus, of the recombined spectro-image, SCIENCE_RCERRORS, and the datacube, SCIENCE_CUBE_ERRORS (Argus only).

Data format. All products come as FITS files with the product pixels in the first extension, and a binary table in the second extension. That fibre_setup table is described here.

Quality plots. Since April 2005 until October 2011, we delivered in the data packages, and to the archive, a set of quality control plots.

Medusa. The first QC plot for the Medusa setups has the following elements:

  1. a crosscut through the raw frame (row @2048)
  2. a closeup of box 1, displaying the central 200 pixels with the fibre PSF and the BIAS level indicated by the red broken line
  3. a S/N comparison: the mean S/N of the brightest fibre (see plot #4) is plotted over the corresponding signal, the value is marked red; for comparison, data points from other science observations in the same setup are overplotted; they are read from the giraffe_science table in the QC1 database, keys brightest_s2n vs. brightest_flux. This plot is intended to monitor any irregularity with the S/N of the reduced spectra: all similar data points should form a square-root law, and the red data point should fall somewhere onto that curve
  4. a selection of spectra (rebinned for display):

    a) the fibre with the lowest nominal magnitude (read from the ozpoz table), assumed to represent the sky signal

    b) the fibre with the brightest nominal magnitude (again read from the ozpoz table; if none found, fibre 10 is displayed)

    c) S/N for the fibre selected in b), derived by division of the spectrum by the corresponding error in the RBN_ERRORS file (SRBE)

    Both the signal and S/N are plotted under 'slope' with a compressed horizontal scale. The strategy to find the faintest and the brightest fibre assumes the usual magnitudes. If the user has chosen a different convention, the guess may fail.

  5. histograms:

    a) the lower 0-200 counts in the product file ("prod."). 0 marks the background, there should be no counts (other than noise) around that value, otherwise there is a problem with background subtraction

    b) the upper 60,000+ region in the raw file, indicative of a saturation problem.

The lower panel displays, among other things, the delta in temperature ("deg") and time ("day"), resp., between the science exposure and the wavelength calibration; these values should be small in order to be sure to have no thermal or non-thermal drifts of the temperature-sensitive grating between science and calibration exposures.

For spectra above 585 nm, sky emission lines are marked in #4a (red vertical lines), and telluric absorption lines are marked in #4b (blue vertical lines). The emission lines (at UVES resolution) are taken from Hanuschik (2003). The list of telluric lines at the UVES resolution is not yet published.


First QC plot
for Medusa data.

Three more QC plots for the Medusa modes display all extracted spectra, along with their user-provided target name and magnitude:


Spectral plots.
For the Medusa modes all extracted spectra are displayed in 3 plots, along with the user-provided target name and magnitude. This figure shows an example for fibres 1-45. n/a means 'not allocated'.

IFU, Argus. The Argus and IFU settings also have a set of QC plots. The first one is very similar to the Medusa case, with the faintest object fibre showed under #4a, the integrated signal from all fibres of subslit #6 under 4b, and the S/N for the first fibre from #6. This selection has been made since Argus and IFU spectra are not independent of each other. The guess of the faintest object may fail under the same conditions as described in the Medusa case. The S/N comparison (plot #3) is not active for Argus/IFU since there is no brightest fibre. The histogram checks in plot #5 are the same as for Medusa.


QC plot
for Argus/IFU modes.

Instead of a preview of all fibres, for Argus and IFU the reconstructed image (RCSPECTRA) is available as preview. It has been obtained by collapsing the spectra into a single number per fibre and display that value on a relative spatial grid. The mapping between fibre index and spatial coordinates has been provided by the pipeline (Fig. 2.7 of the FLAMES manual). Note this is a collapse of the whole spectrum. A narrow-band image in an emission-line can look rather different. Also note that this image is not sky subtracted (but the sky signal is available).


Send comments to <qc_flames@eso.org> 
Last update: August 8, 2017