DATA MANAGEMENT AT ESO

Once a VLT/VLTI observing run is finished, the QC Group creates a data package which is then delivered by the Science Archive Group to the PI. Currently, data packages are created for the following cases:

• VLT/VLTI: all instruments, all Service Mode runs
• La Silla: most La Silla instruments, Service Mode runs

Data packages for VLT pre-imaging runs are created and delivered just as ordinary Service Mode run packages.

The data packages contain raw and processed data (calibration and science). To distribute the data correctly, the FITS keyword 'HIERARCH ESO OBS PROG ID' is evaluated (where PROG.ID actually is a RUN.ID but it's too late to change this ...).

The following rules apply when data are distributed in a data package:

• SCIENCE and ACQUISITION files: a strict check on PROG_ID is made. PIs will receive all science files which carry that PROG_ID, and only those files.
• CALIB files: PIs will receive by default shared calibration files, i.e. those ones which were produced by the Observatory as part of the daily calibration plan. If in addition the user has defined specific calibrations all these files will carry their PROG_ID and will also be included in the data package.

• archive file name (FITS keyword: ARCFILE): this is the unique name of a raw file which consists of the parent instrument name and the timestamp of its generation. Raw FITS files produced at ESO always have extension '.fits'.
  
• original file name (FITS keyword: ORIGFILE): this is the name of a raw file generated on the mountain on the instrument workstation. It consist of:
  - the name of the parent instrument,
  - the category (e.g. BIAS, FLAT, OBS, ... ),
  - the day sequence number within the year,
  - and a sequence number counted within the night.
  Original file names are not unique (they will usually repeat after a year, sometimes even after a day).
  
• data product file name (FITS keyword: PIPEFILE): this is the name of any pipeline product. It is composed of:
  • the prefix 'r.',
  • the ARCFILE of the parent raw file stripped off by the extension,
  • and a sequence number '_0000' etc. (since there may be multiple pipeline products per raw file).

  Product files have the extension '.fits'. In some cases, ESO product files also may have the extension 'tfits' where 't' stands for 'table'. These are ordinary FITS files just like the ones with extension 'fits'.
  In case that more than one raw file has been used to generate the product, the ARCFILE of the first one is used.

• calibration product name: this is a name of calibration data products which has been chosen to have explicit information about the type of file, the date of creation, its version and the instrument parameters in the file name. It is generated after the pipeline product has been created. The FITS keyword PIPEFILE is preserved and can be used to map the two naming conventions.
  
• science product names: with period 75 (April 2005), science products have also been renamed to a scheme which allows the user to recognize from the file name the parent OB, the product type and setting parameters.

What is the general policy of data distribution?

Check out here.

Do I have to submit a request to receive my Service Mode data package?

No. Service Mode programmes are declared completed by the Users Support Department (USD) (info | mail). This signal is sent to the Quality Control Group which then prepares the data package. The data are copied onto media and sent to the PI as soon as possible.

When can I expect to receive my data package?

Once a SM run has been finished or closed by USD, the following series of steps can be expected:

• Raw data are shipped from Paranal to Garching headquartes twice a week.
• They are processed, checked and packed by QC Garching as quickly as possible.
• Once all data for a specific run have been processed, a signal is sent to the archive group to copy these data on media (presently DVDs).
• The complete set of data is mailed to the PI.

In total this chain of events is finished on average a few weeks after the last frames have been measured. Usually this time span is dominated by the shipping time from Chile to Germany.

Information about the status of your ServiceMode run can be found here.

General information about ServiceMode Observing can be found here.

GIRAFFE ARCFILE names always start with 'GIRAF'. They come as FITS files with a main plane with the CCD data, and with two extensions with binary tables containing OzPoz and fibre information.

GIRAFFE ORIGFILE names start with FLAMES.

Names of GIRAFFE product science and calibration files are explained here.

The association of raw files of different types (CALIB and SCIENCE) is a non-trivial task given the many different possible modes of GIRAFFE data. To support in this task, we have created the list files which you find in each package data directory. Since Period 75 (April 2005) there is also an association report, as part of the night and OB logs.

The GIRAFFE association scheme is described here.

FLAMES can feed two spectrographs simultaneously, GIRAFFE and UVES. If a FLAMES run has produced both UVES and GIRAFFE data, they will be packed into two separate packages, but shipped together.

Since July 2004, GIRAFFE data are pipeline-processed, and usually SM packages receive both processed calibration data and reduced science data. If you do not receive pipeline-reduced data, this may be due to the following reasons:

• Your package has been released prior to July 2004.
• Some or all of the setups you have used are not yet pipeline-supported. Find more information about the support status here.
• The pipeline may have failed for other reasons. These are reported to you in a separate file.