Data packages have been delivered for period P87 (April-September 2011) and before.
For new data, acquired after the begin of October 2011, data packages are no longer
created. Users can access their raw data in electronic form through the ESO User Portal.
Data for VLT pre-imaging runs are processed and delivered as
For completeness, the structure of the historical packagesis described
Science data have been processed by the pipelines with the best available calibration data. Please note that ESO is not assuming any responsibility in respect to the usefulness of the reduced data. The adopted reduction strategy may not be suitable for the scientific purpose of the observations.
|top PACKAGE DIRECTORY STRUCTURE
The top-level structure of the data package is as follows:
For each observation block (OB) that has been executed on Paranal, you find all measured raw data (FITS files) in a directory named by the OB number (FITS key HIERARCH.ESO.OBS.ID). If pipeline products exist, these are also added in the OB directory.
The GEN_CALIB directory collects all those calibration files (raw and products) that have been measured as part of the regular calibration plan, and calibration frames of a general nature (like static line tables). The GEN_INFO directory has general information, like data reports and night logs.
The tree shown above is the logical structure, which means that this is the way the data have been organized before they have been put onto media. Depending on the size of your package, the directories may be distributed across several media. It is a good idea to create the original tree on your local disk and then copy all files from the media into this tree.
<OBS_ID> (e.g. 179211)
For each executed observation block of your run, the package contains a directory with all measured data from that OB. All data under <OBS_ID> carry your run ID.
Note that some of your OBs may have been executed more than once. In particular, if time permitted, observatory staff try to re-execute OBs which produced data clearly out of the specified constraints. Check out the NIGHTLOG.html file for details (go to "OB information"). All data from OBs that have been executed multiple times are found in the same directory.
Each OB directory is further subdivided into subdirectories for science frames, calibration frames, and log files. In many cases, there will be science data only, but there may also be OBs with attached calibration data:
All acquisition frames (DPR.CATG=ACQUISITION) from the OB are contained in this directory. This directory only exists if such data exist.
All raw science frames (DPR.CATG=SCIENCE) from the OB are contained in this directory.
Here you find the pipeline-processed science data. The naming scheme can be found here.
If measured, raw calibration frames (DPR.CATG=CALIB) produced by the OB are contained in this directory ("attached calibrations"). These are the ones which have been taken upon user's request in addition to the ones from the calibration plan.
Calibrations measured as part of the regular calibration plan are stored under the GEN_CALIB directory.
The pipeline products of the raw attached calibrations are delivered here.
The CALIB products are renamed. The naming scheme can be found here.
This directory holds logging information about processing and packing
of your data:
- Association Blocks (.ab)
- association logs (.alog)
- logs of the pipeline processing (extension .rblog)
- scoring results (.html) (optional)
- extraction from the nightlog, OB grade, QC0 report (.qcm)
- a reduction comment (.cmt) (optional)
- Association Blocks (ABs) are text files which contain all the
information required to pipeline-process and pack data. This information includes
the reduction recipe, the input raw file(s), the calibration products needed
for processing, and the names of the final products. More ...
- Association logs are delivered since P80. They are a simplified version of ABs, designed to provide the association information essential for the user. More ...
- The pipeline processing log is a record of the science reduction process, with a detailed log of reduction steps, results etc.
- The scoring report is intended to give some feedback
about the data quality. It is still experimental. More ...
- The nightlog file is an extracted version (per OB) of the summary
qc0 report and the NIGHTLOG.html file (see below). More ...
That directory in addition holds QC plots, if available.
This directory collects all calibration frames from the regular calibration plan that are associated to your science data. It also contains their pipeline products, and calibration frames of a general nature (like static line tables). Calibrations that have been measured by user-defined OBs and that have been used for pipeline processing of science data may be included here in addition.
The directory has four subdirectories (gen, logs, proc, raw), two of which have further fine-structure:
Raw calibration files. These divide into raw file types (e.g. BIAS, FLAT etc.; see instrument specific section below).
Calibration products derived from the raw calibrations. These divide into file types like the raw calibration files, see instrument specific section below.
The CALIB products are renamed. The naming scheme can be found here.
Association Blocks, association logs and processing logs for the
calibration files under GEN_CALIB. There might also be scoring logs (.html files).
General calibration data of static nature.
Additional or missing raw calibration files may be retrieved anytime from the generic
ESO Archive form, or from the instrument specific forms.
Calibration data are public immediately while SCIENCE data normally have a proprietary period of one year.
This directory hosts some general information. It has the following subdirectories:
||nightlogs, OB report (HTML files), association report
||executable scripts (presently one: print_all_reports)
The data package contains the following report files:
||the package portal page: point your browser here to find all information
||a table describing the naming scheme for product files
||list of all proprietary files (SCIENCE, attached CALIBs) as read from the archive
||list of all SCIENCE files, containing the comparison between the user constraint set and the actual values
||set of html files with nightlog, OB and association information
|summary report of the fits files in each directory (these files are provided in text [*.txt] and PostScript format)
||all data directories
The executable script print_all_reports under GEN_INFO/scripts can be used to print all postscript files in your package.
Archive report: archive_<RUN_ID>
While the above listings are about files in the package, the archive report is the result of a query to the ESO Archive. It is useful as a check on the completeness of the data package. All files created by OBs which have been generated by the PI are listed here. The list includes all SCIENCE files, and the attached calibrations, and acquisitions, if applicable.
QC0 report: qc0_<RUN_ID>
This report is sent only for Service Mode runs.
This file contains a report of quality control parameters ('QC level 0' where level 0 stands for Quality Control without pipeline processing) for your raw SCIENCE files. These parameters are airmass, seeing, moon distance, and fractional lunar illumination. They have been measured on site (column 'msrd'). They are compared to the required values as defined in your OBs ('targt') and flagged (OK/NOK).
The list is intended to give a rough indication of whether or not the required constraints have been fulfilled. They should not be interpreted in a too formal way, however. E.g., there may be cases where the seeing was worse than required, but this was compensated by a longer exposure time. Check the night reports for details.
Note that the seeing values reported here are DIMM seeing values, they are not measured on the frame. If the alarm flag ("NOK") is set in the SEEING column, the DIMM seeing value was larger than your seeing constraint during the indicated obseration. However, in many cases, the delivered seeing in the instrument focal plane is better than the DIMM seeing. Whenever possible, the on-site astronomer has measured the focal plane immediately after or during execution to determine the success or failure of your observation. Thus, your observation may have been completed within your specifications, even if the SEEING alarm flag is set. Please review the affected observation carefully and check the night reports for details.
sample file (.txt)
Night logs, OB logs and Association report
This is a set of HTML files with night log information, OB grading information and data association information. All relevant information about the nights contained in your package is included here, as well as information about each OB in your delivery.
Point your browser to GEN_INFO/ObservingReports/NIGHTLOG.html (or start from the package portal page, README.html) and navigate per night (labeled as 1), per OB (2) or per set of files (3).
The HTML files also come as stripped-down, printer-friendly versions. The files are organized to have a summary on top, and details below.
You can use either the navigation bar to jump to a specific night/OB/set of files, or use the up/down arrows (night logs only) to browse sequentially. The OB navigation bar (2) uses colour coding to give you a quick impression about OB grading. There are additional links to ambient condition information.
The association report (3) organizes your data and their association. It has two main levels: the OB (observing block), and the AB (association block) which collects raw file(s) and associated information like product files, calibration files, log files etc. This report gives you an impression how the data in your package are logically linked, while the listings in each directory give you a table of contents. File names in the association report may show up several times, e.g when a calibration file has been used for processing more than one science file.
- The external links (like the ASM links: seeing, sky transparency etc.) will only work with network connection.
- The ASM links require java-enabled browsers.
- The navigation bars read best with style-sheets and java-enabled browsers.
|Sample nightlog files
Known IRAF problems
- Filename Length. To display or manipulate the FITS files
with older versions of IRAF (before 2.11), you can:
- copy these FITS files to your hard-disk and rename them with filenames <= 32
characters in length;
- create symbolic links with filenames <= 32 characters in length to your DVD
- Header Interpretation. ESO FITS files use the ESO HIERARCH FITS keyword extensions standard to all ESO telescopes. Note that IRAF treats all ESO HIERARCH header lines as COMMENT lines, i.e. IRAF and IDL cannot automatically interpret the information provided in ESO HIERARCH header lines. The problem may be solved using the tool hierarch28. Find information about this tool here.
- RA, DEC. Please note that the RA and DEC keywords are recorded in degrees. To translate these keywords so that they can be used by IRAF you have to use the asthedit task in the noao.astutil package. The help file for this task gives an example of how to translate the ESO format to the IRAF format.
Stand-alone FITS handling tools
Find information about FITS header handling tools (e.g. dfits, fitsort, hierarch28) here.
For further information about CRIRES please look at:
The GEN_CALIB directory for CRIRES has the following
All calibration frames are divided into a few general
types. These types can be found as subdirectories of the proc
and raw directories:
|Detector dark frames
||Lamp flat field exposures
||Wavelength calibration exposures
||Telluric and spectroscopic standard
General calibration data of static nature. You may
expect the following files:
||catalog of ThAr lines
||catalog of N2O lines
||catalog of sky lines
(for > 1800nm)
||catalog of OH sky lines
(for < 1800nm)
||table of photometric standard star
||configuration for physical model
CRIRES offers several options for obtaining a wavelength
calibration (see User
Manual). For pipeline-reduced science products in this package, the
following strategy has been chosen:
- If an attached wavelength calibration (ThAr lamp or
N2O gas cell) has been measured together with the science data then
the wavelength solution from this calibration has been used for calibrating
- If an attached wavelength calibration does not exist
but a daytime wavelength calibration exists then the wavelength solution
is obtained from the daytime measurement.
- If neither an attached nor a daytime wavelength calibration
has been measured then the wavelength solution is obtained from sky
lines present in the science exposures. This is usually the case for
observations redward of 2400 nm where the density of sky emission lines
is sufficiently high.
Pipeline wavelength calibration operates on each detector
separately. It performs a cross correlation between a line catalog and the actual measurement. Depending on the calibration source and the availability of
appropriate spectral lines, it is possible that a wavelength solution
is found for some but not for all 4 detectors. The success of the wavelength
calibration is given by the fits header keyword HIERARCH ESO QC XCORR
which is present in each of the four extensions of any science or wave
pipeline product. In case of an arc lamp or gas cell exposure, a value
of -1 indicates a failure of the wavelength calibration for this extension
(i.e. detector). Then, the default linear solution for this spectral setting
has been written to the output product. For science and standard star
exposures, the QC XCORR keyword is meaningful only if sky lines have been
used for obtaining a wavelength solution.
In case the line spectrum is dominated by one or two very bright lines, weaker lines may have only a negligible influence on the cross-correlation result. This can be optimized by using the logarithms of the line intensities (setting --wl_log=true for the pipeline recipe crires_spec_wavecal).
An alternative method is provided by the physical model
of the CRIRES spectrograph. The model uses the actual state of the spectrograph
and predicts a wavelength map (i.e. a wavelength for each pixel). This
map is also included in the package and can be compared to the solution
obtained from arc lamp or gas cell calibration sources. The predicted
wavelengths usually have a small offset which can be determined from this
comparison and then be used to calibrate detectors without a sufficient
number of ThAr or N2O lines. The physical model is
not available for data taken in February 2009 and later.
It cannot be guaranteed that the adopted strategy for
wavelength calibration, that has been chosen for the pipeline reduction
in this package, is appropriate for the scientific goal. A first check
can be obtained from the XCORR header keyword. It is also highly recommended
to cross-check the solutions obtained with different methods.
Vignetting on detectors 1 and
Adjacent echelle orders are isolated in
the spectrograph by usage of an additional slit after the pre-dispersing
prism (so-called intermediate slit). For settings below 1450 nm, the slit
had to be set up in a way that the outer parts of detectors 1 and 4 are
not fully exposed. See Sect. 15 of the User Manual for a detailed description of the usable wavelength ranges
for each setting. Since October 2007, updated values for the wavelength
range that is not affected by adjacent orders are written into the header
keywords INS WLEN MIN and INS WLEN MAX.
Fixed-pattern noise and detector
The CRIRES detectors have significant
gain variations from pixel to pixel which are strongest in dispersion
direction on detectors 1 and 4 (and in cross-dispersion direction on detectors
2 and 3). This results in an odd-even column effect in the extracted spectra
for these detectors. The gain variations can be corrected with flat fields
but in several cases a small pattern remains present after flat-fielding.
Since 7 June 2007, daily flat fields are measured with DIT = 2 sec (instead
of 1 sec). This significantly improved correction for fixed-pattern noise.
The detectors also become non-linear for
exposure levels above about 4000 ADUs. This effect is measured by a series
of dedicated flats with increasing exposure level (so-called
detlin flats). A quadratic function is fitted for each pixel and the coefficients
are used later on in the reduction cascade for correcting raw frames.
Pipeline products in Service Mode packages have been reduced
- Flat fields are corrected for detector non-linearity
- Raw science and standard star frames are corrected
for non-linearity and then flat-fielded.
Static calibration files (including coefficient files for correction of non-linearity) can be dowloaded here.
Reduction of science data with user-defined offsets
Science observations using the GenericOffset template can, in general, not be supported by pipeline reduction because the observation and reduction strategy is completely user-defined.
Windowed detector read-out is pipeline-supported since July 2010. A correction for non-linearity is not yet available.
Events important for data reduction
- Tilt between intermediate and entrance slit: CRIRES underwent a technical intervention between 8 and 17 July 2010 to solve the problems with the entrance slit. After this intervention it has become apparent that the problem could not be solved entirely. Whereas the slit width is adjustable to different sizes again, a tilt between the intermediate and entrance slit was detected. This tilt causes a strong vignetting of the 1st and 4th detector and flux losses along the slit. Nevertheless, useful observations can be obtained for the 2nd and 3rd detector using a small nod throw of about 5 arcsecs.
- Slit width varies along length of slit: after the May 2010 intervention, the width of the entrance slit varies along the slit length; the width is smaller on the bottom of the science detector and larger on the top. This results in a change of the spectral resolution of about 20% from bottom to top. There is also a steep gradient in flux, visible especially in sky background and in flat fields. All observations taken after 28 May 2010 until 7 July are affected.
- Detector position shifted: after the January 2009 intervention,
the position of the detectors has shifted in Y direction. This affects all
measurements taken in February 2009 and later. Calibrations taken before
the intervention cannot be used for observations after the intervention.
This is especially important for the correction of detector non-linearity.
New non-linearity coefficients have been determined in March 2009; the files can be downloaded here. Data taken in February 2009 and until March 08, 2009 have not been corrected for non-linearity within the standard pipeline reduction.
- Physical model: the physical model of the pipeline is
not available for data taken after January 2009.
- Fibre bundle installed with P81: In April 2008, a fibre bundle has been mounted
to the ThAr lamp carriage. The fibres are projected on the slit. ThAr
wavelength calibration exposures now have four to five separate light traces on
the detector. Before, the whole slit was illuminated.
- Reduced slit length: after the instrument upgrade early April 2008, the field-of-view
along the slit (and therefore the effective slit length) was reduced to
12 arcsec. On the detector, only about 200 pixels in Y direction can be
illuminated. The full slit length was restored for observations after
11 June 2008.
- Additional vignetting during early Service Mode observations: Accurate positioning of the pre-disperser
and the intermediate slit turned out to be difficult during the first
weeks of CRIRES Service Mode operations in P79 (March to May 2007). Although measures have been taken
to stabilise these elements, some observations were affected by additional
vignetting and/or contamination from adjacent orders on detectors 1 and
4. All observations have been checked for these effects and have been
repeated where appropriate.