PIPELINES AT ESO

This page provides information about pipeline processing and data types.

Raw data are selected, associated and inserted into a reduction mechanism which produces calibration products, science products and quality control information. This mechanism is the data processing pipeline. There is one such pipeline for each VLT and VLTI instrument.

Find general information about ESO reduction pipelines here.

The main functionalities of the pipelines are:

• create master calibration data,
• reduce science frames,
• extract QC information from the data.

QC Garching creates

• master calibration data from raw calibration data, both for Visitor Mode and Service Mode runs . The raw data are stored in the ESO Archive and are public. They are assessed and selected in terms of their quality and used for QC checks and for trending;
• reduced science data (currently only for Service Mode runs) . For SM programmes, QC Garching runs the pipeline on all science raw data, using the best available, quality-checked master calibration data.

Science data are processed by the pipeline with the best available calibration data. Please note that ESO is not assuming any responsibility in respect to the usefulness of the reduced data. The adopted reduction strategy may not be suitable for the scientific purpose of the observations.

OPERATIONS

There are two instances of the data reduction pipelines:

• at the instrument workstation on Paranal, running in automatic mode,
• at HQ Garching, run by the Quality Control Group in the optimized mode.

The automatic mode is used for quick look purposes and for on-site quality control. It processes all raw data sequentially, as they arrive from the instrument. If calibration products ("master calibrations") are required for processing science data, these are taken from a database with standard, pre-manufactured calibration products. The automatic mode is not tuned to obtain the best possible results.

The optimized mode is the mode, which uses all data of a night, including the daytime calibrations. The calibration data are sorted and grouped according to their dependencies. Master calibration data are created. Their quality is checked. The certified, closest-in-time products are finally associated to the science data of a night. This is the way all calibration data in supported modes are processed by QC Garching, as are all science data from Service Mode programmes.

The HAWK-I data processing pipeline is operational since April 2008. All operational HAWK-I observing modes are supported although with some caveats:

• JITTER offsets greater than 1500 can not be handled
• Limitations of the OS may impose limits on the number of RAW frames that can be handled in one recipe exection.

The HAWK-I pipeline is publicly available (check out here). Under this link you also find the pipeline Users Manual.

The following is s summary of recent changes in the pipeline:

• 1.4.2-Public Release:
  • Monolithic Science recipe hawki_img_jitter split into several component recipes to allow use with Reflex.
  • MASTER DARK product is now divided by DET.DIT value, i.e. pixel values are now [ADU/sec] (previously were [ADU])
  • A number of bugs and improvements in hawki_img_zpoint (see below).
• 1.3.5: Better memmory allowing the hawki_img_jitter recipe to handle up to ~135 frame stacks
• 1.3.5: Median level and RMS level filtering for auto selection of Flatfield sample

top CALIBRATION, REDUCTION

Find the description of HAWK-I data processing and pipeline recipes here:

HAWK-I pipeline
calibration
science reduction

o

top FILE FORMAT

Raw data. HAWK-I has a 2x2 mosaic of 4 2048x2048 photon-sensitive pixels. There are NO pre- or post- overscan columns or rows.

Further details regarding the chips can be found in Appendix B of the User Manual.

In total, HAWK-I raw frames have 4096x4096 pixels. There is only one read-mode available, which is unbinned, un-correlated. Raw frames have a size of 65MB each. Find example (reference) frames here.

Extensions. Raw data come as FITS files with FIVE HDUs (header units), so-called Multi-Extension FITS files, or MEFs. The primary HDU has the header with all primary keywords and telescope, positioner, detector, observation, instrument etc. keywords. The pixel data, plus chip-specific header info for each of the FOUR chips are then stored in four further HDUs.

Products. Pipeline products are either FITS images or FITS tables, and either have a single HDU (i.e. traditional FITS files) or 5 HDUs as for the RAW data, i.e. a primary HDU containing the header with all primary keywords and telescope, positioner, detector, observation, instrument etc. keywords and then pixel or table data, plus chip-specific header info for each of the FOUR chips in four further HDUs. Further info can be found here.

Since The beginning of HAWK-I operations (April 2008), runs performed in Service Mode receive a set of DVDs containing:

• all raw science data,
• reduced science data (if applicable),
• associated calibration products,
• associated raw calibration data,
• listings and logs,
• QC reports and OB reports.

3Gb Limit: 32bit Linux systems, as used by the ESO Quality Control Group, have a fundamental limit to the amount of memory that any single process can address at one time. This has the impact on HAWK-I data procecssing that for data processed with pipeline version 1.3.4 and earlier stacks of more than about 70 frames can not be handled. Therefore, for the purpose of QC, only the first 70 frames of stacks with more than 70 frames are used to create pipeline products.
Once identified the pipeline memory handling was improved to enable handling of stacks of up to approximately 135 frames, resulting in version 1.3.5.
This limit should disappear if a 64bit architecture is used, but this has NOT be verified in this case and thus for the moment remains pure speculation.
See, for example:

• http://kerneltrap.org/node/2450
• http://kbase.redhat.com/faq/FAQ_43_2412.shtm
• http://lwn.net/Articles/75174/
• http://linux.derkeiler.com/Mailing-Lists/Fedora/2006-04/msg02968.html

Imperfect telescope offsets: For the purposesof QC in P81, SCIENCE frames reduced by the pipeline without using the --refine command line option of the hawki_img_jitter recipe. Without the --refine command line option, the pipeline uses the offsets recorded in the headers of each RAW frame in the stack for shifting each frame onto a common reference. With the --refine command line option, the pipeline uses the offsets recorded in the headers of each RAW frame in the stack as a first guess, but refines these guesses by making a fit to a single object in the central region of the chip for shifting each frame onto a common reference.
As a result two effects are seen:

1. Non-circular Image Quality: The image quality measured in the PRO.CATG=COMBINED products is generally slightly non-circular. This is currently believed to be due to imperfect offseting of the telescope, i.e. the telescope does not perfectly achieve the requested offsets.
2. Double images: Occassionally the pipeline produces PRODUCTs with double (or even multiple) images of each object. This is believed to be due to glitches in the telescope offseting, meaning that the telescope does makes an offset significantly different from that requested (and thus recorded in the frame headers) of it.
   

Bugs in the ZPOINT recipe prior to version 1.4.2: HAWK-I ZPOINTs are measured by observing single standard stars. The star is observed four times in each filter, placing the star at the center of each chip in turn. The OB should be created as follows:

1. Acquistion Template: Preset the telescope
2. Observing Template: offset the telescope to the first chip and then make an exposure
3. Observing Template: offset the telescope to the second chip and then make an exposure
4. Observing Template: offset the telescope to the third chip and then make an exposure
5. Observing Template: offset the telescope to the fourth chip and then make an exposure
6. done