PIPELINES AT ESO
Pipelines are used at ESO to process both calibration data and science data
and to retrieve quality information.
There is a dedicated
pipeline for each VLT and VLTI instrument.
Find general information about ESO reduction pipelines here.
The main functionalities of the pipelines are:
QC Garching creates master calibration data from all raw calibration data. The
raw data are stored in the ESO Archive and are public. They are
quality-checked and used for data reduction and for trending.
- create master calibration data,
- reduce science frames,
- extract QC information from the data.
For selected instrument modes, we offer science-grade data products processed
with the pipelines.
There are two instances of the data reduction pipelines:
The automatic mode is used for quick look and for on-site quality
control. It processes all raw data sequentially, as they arrive from the
instrument. If calibration products ("master calibrations") are required for
processing science data, these are taken from a database with standard,
pre-manufactured calibration products. The automatic mode is not tuned to
obtain the best possible results.
- at the instrument workstation on Paranal, running in automatic mode,
- at HQ Garching, run by the Quality Control Group in the optimized mode.
The optimized mode is the mode, which uses all data of a night, including the
daytime calibrations. The calibration data are sorted and grouped according to
their dependencies. Master calibration data are created. Their quality is
MUSE observations in wide-field-mode (WFM) are offered with no AO
(NOAO) and AO (since early 2018) correction in two different settings:
E - extended range, to include the blue part of the spectrum and N -
nominal range of the spectrum.
In 2018, new, narrow-field-mode (NFM) was commissioned. The data are
taken only using AO system.
The MUSE pipeline is publicly available here. Under this link
you can also find the pipeline Users Manual.
CALIBRATION and SCIENCE data
Find the description of MUSE data processing and pipeline recipes for
calibrations here and for
science data here.
The pipeline supports all MUSE settings.
The MUSE raw data file consists of 24 FITS extensions, one for each
IFU. It contains the detector data as well as the IFU/detector
specific keywords, and a primary FITS HDU containing only
keywords. The 24 extensions with detector data of the IFUs can be
identified by EXTNAME FITS keyword. The extensions are called CHAN01,
CHAN02, etc. and, for technical reasons, the numbering of extensions
does not correspond to the numbers of the MUSE channels. However, the
sequence of the channels as they are stored in the FITS file is
Find more details about the MUSE CCD detectors format and properties here.
Manual has also a detailed description of the MUSE raw data
| file type || size of raw file|
| calibration or science frame || 0.9 GB|
The MUSE data reduction system (DRS) creates pipeline products in three different formats:
- FITS tables
- Pixel Tables
- Data Cubes
Pipeline Products - Images: A MUSE image, i.e. the reduced
image of a single CCD (channel) has three 2D FITS image extensions:
- DATA - the reduced image
- DQ - the data quality flags encoded in an integer value according to the Euro3D standard
- STAT - the variance of the reduced data
The calibration recipes write product files which contain data
belonging to a single IFU, thus produce 24 files for each kind of
product. In addition, the combined format is available where all 24
products are stored in a single FITS file in extensions. For example,
a combined master bias consists of 72 extensions (24 x 3 (data, error
and data quality)) with extensions called CHAN01.DATA, CHAN01.DQ,
CHAN01.STAT, CHAN02.DATA, CHAN02.DQ, CHAN02.STAT, etc.
Starting from 2014-12-08, when pipeline version v.1.0 was
installed, all the products come in combined format.
Pipeline Products - FITS tables: Some of the products of the
calibration recipes come in a format of standard FITS tables.
Pipeline Products - Pixel Tables: The pixel table format is
used to store intermediate products of the pre-processing recipe
muse_scibasic and a few post-processing recipes. It is a tabular
format where 1 raw corresponds to each CCD pixel of a single IFU and
the value of each pixel is stored together with it's "coordinates"
(position and wavelength).
Pipeline Products - Data Cubes: The final output of the MUSE science
recipes is in data cube format. Similar to the image format the data
cubes consist of up to three 3D FITS cube (NAXIS=3) extensions. They
are also named DATA, DQ and STAT. By design, the data quality
extension is not included. Instead, pixels which do not have clean
data quality status are directly encoded as NaN in the DATA
| file type || size of product file|
| Images combined || 4.9 GB|
| Fits tables || 0.1-0.8 MB|
| Pixel tables || up to 9 GB|
| Data Cubes || up to several GB|
More details are described here.