VIMOS:
Association Rules

1. Introduction

Data classification and association are important concepts for managing the VIMOS dataflow process.

Data classification is the process of identifying the type of a file and what recipe is needed to process it. Identification is done by comparing header keywords to a classification table. In the case of VIMOS this task is performed by the Data Organizer (DO) on Paranal where the pipeline is used as a quick-look tool. In Garching scripts are used to classify raw data. The most important keywords are HIERARCH ESO TPL ID and the HIERARCH ESO DPR keywords. The end of template event occurs when HIERARCH ESO TPL START has changed with respect to the previous file. Note that all raw frames of a night can be sorted according to the time stamp in ISO format (the MJD-OBS key). Any classification system must have knowledge of the classification of the previous frames. After all files of a certain period, e.g. a complete night or many nights, have been classified, the following is known:

One or more frame stacks (lists) containing one or more frames. The last frame of a stack is signaled by an end-of-template event.
Each frame stack is associated with a reduction recipe.

Data association is defined as the process of associating one or more frames by common properties. A typical association task is to find all the calibration frames necessary to process a science frame. Data association occurs twice - when calling a reduction recipe and when packing a Service Mode data package. The tasks are very similar but not identical. Data association requires rules. The VIMOS association rules are described below.

Classification and association of VIMOS raw frames occurs at two places in the VLT data-flow system:

Paranal: raw data arrive one-by-one in a sequence. Each frame is classified and associated with available calibration frames in a local database. These are not the most recent calibration (i.e. from that night) but are certified as having high quality. Association is also used in the morning to determine what daytime calibrations are needed for science data acquired during the previous night.
Garching: all raw frames of a given night as well as a local archive of verified calibrations are available. All master calibration files are produced before science data are classified. Science files are then classified and associated with the best possible master calibration files. Since master calibration products can be available from many nights, association always uses the master calibration product which is closest-in-time as well as has matching keywords.

2. VIMOS Association Rules

2.1 VIMOS Association Rules for Imaging

Imaging (Detector) Keywords

The following keywords are used for imaging (detector) association:

HIERARCH.ESO.OCS.CON.QUAD
HIERARCH.ESO.DET.READ.CLOCK
HIERARCH.ESO.WIN1.BINX
HIERARCH.ESO.WIN1.BINY
HIERARCH.ESO.WIN1.NX (not used for zeropoint tables PZPI)
HIERARCH.ESO.WIN1.NY (not used for zeropoint tables PZPI)

Below these keywords are referred to as detector keywords.

When necessary, filter information is obtained from:

HIERARCH.ESO.INS.FILTERi.NAME
HIERARCH.ESO.INS.FILTERi.ID

where "i" indicates the quadrant number from keyword HIERARCH.ESO.OCS.CON.QUAD.

Calibration Data

Bias: bias association is done only on the raw data because biases are reduced without further calibration products. Each bias frame stack contains frames with matching detector keywords. The product tag is MBII (Master Bias Imaging).

Dark: to reduce a stack of raw dark frames, a bias calibration product (MBII) is needed. Each dark frame stack contains frames with matching detector keywords. The association rule for the required master bias is: search for the closest-in-time master bias product with matching detector keywords. The product tag is MDKI (Master Dark Image).

Sky Flats: to reduce a stack of raw sky flat frames, a bias calibration product (Master Bias: MBII) is required. Each sky flat stack contains frames with matching detector and filter keywords. The association rule for the required master bias is: search for the closest-in-time MBII product with matching detector keywords. The product tag is MKFI (Master Sky Flat Image).

Screen Flats: to reduce a stack of raw screen flats, a master bias calibration product (MBII) is required. Association is done using the same keywords as Sky Flats (see above). The product tag is MSFI (Master Screen Flat Image).

Photometric Standard Stars: to reduce a raw standard star frame, a bias (MBII) and a sky flat calibration product (MKFI) are required. The matching parameter for master bias (MBII) are the the detector keywords. The matching parameters for the master sky-flat are the detector keywords plus a filter keyword. Two products are obtained: the reduced image and a table containing all the magnitude information of the detected standard stars. The tags are PSTI (Photometric Standard Image) and PSMI (Photometric Standard Magnitudes Imaging).

Photometric Zeropoint table (PZPI): the nightly zeropoint is computed from all the Photometric Standard Magnitude (PSMI) tables with matching detector and filter keywords. The product is a table containing the zeropoint of the night. The tag is PZPI (Photometric Zeropoint Product Imaging).

Science Data

Imaging-Jitter: Each stack of science frames contains one or more frames with matching detector and filter keywords. The following calibration products can be associated with the jitter recipe:

bias : MBII : matching detector keywords
dark : MDKI : matching detector keywords
bad-pixel table: produced with Screen Flats acquired with a dedicated template called img_tec_DetLinearity : matching detector keywords
sky flat : MKFI : matching detector and filter
screen flats: MSFI : matching detector and filter
photometric standard magnitude table: PZPI : matching filter keywords and detector keywords. The detector keywords indicating the read-out size of the CCD ("DET WIN1 NX" and "DET WIN1 NY") are here excluded from the matching rule.

Note that the only calibration products that are required by the reduction recipe are: bias, sky flat and photometric standard magnitude table.

2.2 VIMOS Association Rules for MOS

MOS Keywords

In addition to the imaging (detector) keywords listed above, the following keywords are needed for MOS associations:

HIERARCH.ESO.INS.FILTi.NAME : order sorting filter
HIERARCH.ESO.INS.GRISi.NAME : grism
HIERARCH.ESO.INS.MASKi.ID : mask identification number

where 'i' is the quadrant number taken from HIERARCH.ESO.OCS.CON.QUAD. These are the so-called MOS keywords.

Calibration Data

Bias: same as for imaging mode. The product tag is MBIS (Master Bias Spectroscopy). Imaging and spectroscopy detector keywords differ for the read-out window (2148x4096 for spectroscopy and 2148x2440 for imaging) and gain (DET.READ.CLOCK keyword).

Dark: same as for imaging mode. The product tag is MDKS.

Flats: to reduce a stack of raw MOS spectroscopic flats, a bias product (MBIS) is required. To associate the raw flat frames to stack we use the detector keywords. The closest-in-time master bias (MBIS) to associate must have matching detector keywords. The product tag is MSFM (Master Screen Flat MOS).

Arc Lamps: to reduce a stack of raw MOS arc lamp frames, a bias calibration product (MBIS) is required. A flat (MSFM) is optional. A stack of raw lamp frames must have consistent detector and MOS keywords. The bias product (MBIS) must have matching detector keywords only while the flat product (MSFM) must have matching detector and MOS keywords. The product is a table containing the wavelength inverse dispersion solution, the tag is PWDM (Wave Dispersion MOS).

Spectrophotometric Standard Stars: to reduce a raw frame a bias (MBIS), a flat (MSFM) and an arc lamp product (PWDM) are required. The bias must match only in the detector keywords. The flat (MBIS) frame and arc-lamp (PWDM) product must have the same detector and MOS keywords as the input raw frame. The product is a spectro-photometric table.

Science Data

MOS-Jitter: The stack of science frames must contain one or more frames with the same detector and MOS keywords. The following calibration products can be associated with the jitter recipe:

bias : MBIS : matching detector keywords
dark : MDKS : matching detector keywords
bad-pixel table: produced with Screen Flats acquired with a dedicated template called mos_tec_DetLinearity: matching detector keywords.
flat : MSFM : matching the detector and MOS keywords
arc lamp: PWDM : matching detector and MOS keywords
spectrophotometric standard: matching detector keywords and HIERARCH ESO INS GRISi NAME

Note that the only calibration products required by the reduction recipe are: bias, and arc-lamp. Flat-fielding is skipped for the moment because due to flexure problems which result artifacts in the flat-fielded data. The spectrophotometric calibration is not yet pipeline-supported.

2.2 VIMOS Association Rules for IFU

IFU Keywords for Association

In addition to the imaging (detector) keywords listed above, the following keywords are needed for IFU associations:

HIERARCH.ESO.INS.FILTi.NAME : order sorting filter
HIERARCH.ESO.INS.GRISi.NAME : order sorting grism
HIERARCH.ESO.INS.IFUE.MAG : IFU magnification
HIERARCH.ESO.INS.IFUS.MODE : IFU shutter mode

where 'i' is the quadrant specified by the HIERARCH.ESO.OCS.CON.QUAD keyword. These are the so-called IFU keywords.

Calibration Data

Bias: in common with MOS mode. The product tag is MBIS (Master Bias Spectroscopy).

Dark: in common with MOS mode. The product tag is MDKS.

Flats and Arc Lamps: Flats and Arc lamps are both required by a single calibration recipe. A bias calibration product is required (MBIS matching detector keywords). The stack of raw flats and arc lamps must have matching detector and IFU keywords. They also have to be part of the same tamplate (matching keyword TPL START). The product are three multidimensional tables containing the wavelength inverse dispersion relation (product tag PWDF), the fiber relative transmission (product tag PTRF) and the fiber tracing (product tag PTCF).

Spectrophotometric Standard Stars: to reduce a raw frame, a bias (MBIS), and the calibration recipe products (PWDF,PTCF and PTRF) are required. The bias must match only in the detector keywords. The other products must match in both detector and IFU keywords. The products are the total standard spectrum (STXF), the extracted fiber spectra (STRF), the sky spectrum that have been subtracted (STSF) and the reconstructed image of the field of view (STVF). The computation of the spectral response curve is not yet pipeline-supported.

Science Data

IFU-science: The science frames are reduced one by one. To obtain a good reduction the calibration product associated (PWDF,PTCF,PRTF) must have in addition to the same detector and IFU keywords the same OBS ID keyword as the science (i.e. they must be result from raw files taken with the appropriate NightCalib template). The following calibration products can be associated with the science reduction recipe:

bias : MBIS : matching detector keywords
bad-pixel table: produced with Screen Flats acquired with a dedicated template called mos_tec_DetLinearity: matching detector keywords
dispersion solution file: PWDF : matching detector, IFU keywords and OBS ID keyword
tracing/extraction fiber file: PTCF : matching detector, IFU keywords and OBS ID keyword.
fiber transmission file PTRF : matching detector, IFU keywords and OBS ID keyword.

Send comments to <qc_vimos@eso.org>
Last update: 5 Jul 2004