VERSION:

P81 (April 2008)

 

GENERAL

Programmes performed in Service Mode receive a set of DVDs.

This page contains an overview of the structure and content of the AMBER data package. For further information look at:

• AMBER Quality Control and Data Flow Operations (still under construction)
• Description of AMBER data management (still under construction) with a list of FAQs.
• Details of the AMBER Quality Control process
• General information of AMBER.

NOTE: For the time being, the pipeline available for AMBER data is still under developpment, it is based on amdLib v0.9. The reduced data should be used only to estimate the quality of the data.

DIRECTORY STRUCTURE

The top-level structure of the package for your AMBER run is as follows:

For each observation block (OB) that has been executed on Paranal, you find all measured raw data (FITS files) in a directory named by the OB number (FITS key OBS.ID). If pipeline products exist, these are also added within the OB directory. The OB package structure is hoped to be more user-oriented than the previously used DATE structure.

The GEN_CALIB directory collects all calibration files (raw and products) that have been measured as part of the regular calibration plan, and calibration frames of a general nature (like static line tables). The GEN_INFO directory has general information.

The tree shown above is the logical structure, which means that this is the way the data have been organized before they have been put onto media. Depending on the size of your package, the directories may be distributed across several media. It is a good idea to create the original tree on your local disk and then copy all files from the media into this tree.

<OBS_ID> (e.g. 179211)

For each executed observation block of your run, a directory containing all measured data is created. All data under <OBS_ID> carry your run ID.

Note that some of your OBs may have been executed more than once. In particular, if time permitted, we tried to re-execute OBs which produced data clearly out of the specified constraints. Check out the NIGHTLOG.html file for details (go to "OB information"). All data from OBs that have been executed multiple times are found in the same directory.

Each OB directory is divided into subdirectories for science frames, calibration frames, and log files. In many cases, there will be science data only, but there may also be OBs with attached calibration data:

<OBS_ID>

sci_raw sci_proc cal_raw cal_proc logs

<OBS_ID>/sci_raw

All raw science frames (DPR.CATG=SCIENCE) from the OB are contained in this directory. Find a description of raw AMBER science data here.

<OBS_ID>/sci_proc

Here you find the processed science data. Find a description of processed AMBER science data here

The delivered products of the data reduction are (sorted by index)::
- *0005.fits

With P75, all reduced SCIENCE data are renamed. The renaming scheme can be found here.

<OBS_ID>/cal_raw

If measured, raw calibration frames (DPR.CATG=CALIB) are present in this directory.

Calibrations measured as part of the regular calibration plan are stored under the GEN_CALIB directory. Find a description of AMBER raw calibration data here.

<OBS_ID>/cal_proc

The pipeline products of the raw attached calibrations are delivered here. Find a description of AMBER product calibration data here. The CALIB products are renamed. The renaming scheme can be found here.

<OBS_ID>/logs

This directory has logging information about processing and packing of your data:

• Association Blocks (extension .ab)
• logs of the pipeline processing (extension .rblog)
• association logs (extension .ablog)

Association Blocks (ABs) are text files which contain all the information required to pipeline-process and pack data. This information includes the reduction recipe, the input raw file(s), the calibration products needed for processing, and the names of the final products. ABs provide the logical link between raw files, product files, and associated calibration files. They are described here.

The pipeline processing log is a record of the science reduction process, with a detailed log of reduction steps, results etc.

Association logs are delivered since P80. They are a simplified version of ABs, designed to provide the association information essential for the user. More.

GEN_CALIB

This directory collects all calibration frames that are associated to your science data, and that have been measured as part of the regular calibration plan. It also contains their pipeline products and calibration frames of a general nature . Calibrations that have been measured by user-defined OBs and that have been used for pipeline processing of science data may be included here in addition. The directory has three subdirectories, two of which have further fine-structure:

GEN_CALIB
raw	proc	gen	logs
TBC TBC TBC TBC	TBC TBC TBC TBC

GEN_CALIB/raw

Raw calibration files. Find a description of AMBER raw calibration data here.

GEN_CALIB/proc

Calibration products derived from the raw calibrations. Find a description of AMBER product calibration data here. The CALIB products are renamed. The renaming scheme can be found here.

GEN_CALIB/logs

Association Blocks and processing logs for the calibration files under GEN_CALIB.

GEN_CALIB/gen

General calibration data of static nature.

GEN_INFO

This directory hosts some general information which is described under list files. It has the following subdirectories:

DIRECTORY	CONTENT
ObservingReports	nightlogs, OB report (HTML files), association report
scripts	executable scripts (presently one: print_all_reports)

 

You will find the following list files:
 

FILE	CONTENT	WHERE
ServiceMode.html	this file	top
product_codes.html	a table describing the naming scheme for product files	top
README.txt	short additional information	top
archive_<RUN_ID.txt	list of all proprietary files (SCIENCE, attached CALIBs) as read from the archive	GEN_INFO
qc0_<RUN_ID>.txt	log containing the comparison between the constraint set (airmass, moon distance, etc) and the actual values	GEN_INFO
NIGHTLOG.html	set of html files with nightlog, OB and association information	GEN_INFO/ObservingReports
list_sciRaw_<OBS_ID>.txt etc.	summary of the fits files in each directory (these files are provided in text [*.txt] and PostScript format)

The executable script print_all_reports under GEN_INFO/scripts will print all ps files in your package.

Archive summary file: archive_<RUN_ID>

While the above listings are about files on the DVDs, the archive summary listing is the result of a query to the ESO Archive. It is useful as a check on the completeness of the ServiceMode package. All files created by OBs which have been generated by the PI are listed here. The list includes all SCIENCE files, and the attached calibrations, and acquisitions, if applicable. This list is the result of a query to the archive.

QC0 summary file: qc0_<RUN_ID>

This file contains a report of quality control parameters ('QC level 0' where level 0 stands for Quality Control without pipeline processing) for your raw SCIENCE files. These parameters are airmass, seeing, moon distance, and fractional lunar illumination. They have been measured on site (column 'msrd'). They are compared to the required values as defined in your OBs ('targt') and flagged (OK/NOK).

The list is intended to give a rough indication of whether or not the required constraints have been obeyed. They should not be interpreted in a too formal way, however. E.g., there may be cases where the seeing was worse than required, but this was compensated by a longer exposure time. Check the night reports for details.

Note that the seeing values reported here are DIMM seeing values, they are not measured on the frame. If the alarm flag ("NOK") is set in the SEEING column, the DIMM seeing value was larger than your seeing constraint during the indicated obseration. However, in many cases, the delivered seeing in the instrument focal plane is better than the DIMM seeing. Whenever possible, the on-site observer has measured the focal plane immediately after execution to determine the success or failure of your observation. Thus, your observation may have been completed within your specifications, even if the SEEING alarm flag is set. Please review the affected observation carefully.

Night logs, OB logs and Association report

This is a set of html files with an extraction of night log information, OB grading information and data association information. All relevant information about the nights contained in your package is included, as well as information about each OB in your delivery.

Point your browser to GEN_INFO/ObservingReports/NIGHTLOG.html and navigate either per night (labeled as 1), per OB (2) or per set of files (3).

The html files also come as stripped-down, printer-friendly versions. The files are organized to have a summary on top, and details below.

You can use either the navigation bar to jump to a specific night/OB/set of files, or use the up/down arrows (night logs only) to browse sequentially. The OB navigation bar uses colour coding to give you a quick impression about OB grading. There are additional links to ambient condition information.

The association report organizes your data and their association. It has two main levels: the OB, and the AB (association block) which collects raw file(s) and associated information like product files, calibration files, log files etc. This report gives you an impression how the data in your package are logically linked, while the listings in each directory give you a table of contents. File names in the association report may show up several times, e.g when a calibration file has been used for processing more than one science file.

NOTE:

• The external links (asm: seeing, sky transparency etc.) will only work with network connection.
• The asm links require java-enabled browsers.
• The navigation bars read best with style-sheets and java-enabled browsers.

Raw data summary file: list_of_all_obs

This file gives a short description of each FITS file in the raw directory : SCIENCE files on the science target and acquisition files (ACQ and FRINGE_SEARCH) This information has been extracted directly from the FITS header. Raw files are organized per date.

Reduced data summary file: list_of_all_red

This file gives a short description of each FITS file in the reduced directory. This information has been extracted directly from the FITS header.

Calibration data summary file: list_of_all_calibs

A similar file is provided for the calibration data. It contains a listing of raw calibration files similar to the raw data summary file (list_of_all_obs).

General calibration data summary file: list_of_all_gen_calib

A similar file is provided for the general calibration data which are taken for this VLT Run ID. It is similar to the raw data summary file (list_of_all_obs). This file is in the GEN_CALIB directory.

Instrumental Transfer function data summary file: Transfer_function_<DATE>

This files shows the instrumental Transfer function obtained on the VLTI calibrators. The TF is listed here only if the raw data were successfully reduced by the pipeline and if an estimated diameter and its associated error is available for this object in the VLTI calibrator database. More calibrators may also have been observed. All the raw data on astronomical calibrators could be retrieved from the ESO archive.

Archive summary file: archive_<RUN_ID>

While the above listings are about files on the DVDs, the archive summary listing is the result of a query to the ESO Archive. It is useful as a check on the completeness of the ServiceMode package. All files created by OBs which have been generated by the PI are listed here. The list includes all SCIENCE files, all CALIB files, and acquisitions, if applicable. Please note that only the SCIENCE files are proprietary.

Observing Reports: nightlog and OBlogs

This is a set of html files with an extraction of night log information and OB information. All relevant information about the nights contained in your package is included, as well as information about each OB in your delivery.

Point your browser to GEN_INFO/ObservingReports/NIGHTLOG.html and navigate either per night (labeled as 1) or per OB (2).

The html files also come as stripped-down, printer-friendly versions. The files are organized to have a summary on top, and details below. The OB summary replace the previously delivered OB summary files.

You can use either the navigation bar to jump to a specific night/OB, or use the up/down arrows (night logs only) to browse sequentially. The OB navigation bar uses colour coding to give you a quick impression about OB grading.

There are additional links to ambient condition information.

NOTE:

• The external links (asm: seeing, sky transparency etc.) will only work with network connection.
• The asm links require java-enabled browsers.
• The navigation bars read best with style-sheets and java-enabled browsers.

Raw Data

We are sending you all the FITS files produced by Observation Blocks you created and that we were able to execute. Note that some of your OBs may have been executed more than once. In particular, if time permitted, we tried to re-execute OBs which produced data clearly out of the specified constraints. Check out the OB status report for details.

We are sending you raw and processed technical calibration data (Bad Pixel Map and Flat Field Map), amber specific calibration data (P2VM) as well as raw and pre-processed data from the astronomical calibrator you requested. The raw data of the other calibrator objects are available from the  ESO Archive. These calibration data are public while SCIENCE data have a proprietary period of one year. The technical calibration products BPM and FFM are available here.

PROBLEMS, ISSUES, HINTS

Known issues

Stand-alone FITS handling tools