FORS1:  Service Mode data package
	HOME INDEX SEARCH HELP NEWS

Programmes performed in Service Mode (SM) are distributed through a set of DVD ROMs which contain:

• all raw SCIENCE data,
• reduced SCIENCE data for the supported modes,
• suitable raw CALIBRATION data for all modes,
• master CALIBRATION data for the supported modes,
• ACQUISITION observations,
• data listings and reduction logs,
• observations reports

DIRECTORY STRUCTURE

Directory structure has changed. Naming convention for processed science frames has changed.

Association report available

The top-level structure of the package for your FORS1 run is as follows:

For each observation block (OB) that has been executed on Paranal, you find all measured raw data (FITS files) in a directory named by the OB number (FITS key OBS.ID). If pipeline products exist, these are also added within the OB directory. The OB package structure is hoped to be more user-oriented than the previously used DATE structure.

The GEN_CALIB directory collects all calibration files (raw and products) that have been measured as part of the regular calibration plan, and calibration frames of a general nature (like static line tables). The GEN_INFO directory has general information.

The tree shown above is the logical structure, which means that this is the way the data have been organized before they have been put onto media. Depending on the size of your package, the directories may be distributed across several media. It is a good idea to create the original tree on your local disk and then copy all files from the media into this tree.

<OBS_ID> (e.g. 179211)

For each executed observation block of your run, a directory containing all measured data is created. All data under <OBS_ID> carry your run ID.

Note that some of your OBs may have been executed more than once. In particular, if time permitted, we tried to re-execute OBs which produced data clearly out of the specified constraints. Check out the NIGHTLOG.html file for details (go to "OB information"). All data from OBs that have been executed multiple times are found in the same directory.

Each OB directory is divided into subdirectories for science frames, calibration frames, and log files. In many cases, there will be science data only, but there may also be OBs with attached calibration data:

<OBS_ID>

sci_raw sci_proc cal_raw cal_proc logs

<OBS_ID>/sci_raw

All raw science frames (DPR.CATG=SCIENCE) from the OB are contained in this directory. Find a description of raw FORS1 science data here.

<OBS_ID>/sci_proc

Here you find the processed science data. Note that not all setups are presently pipeline-supported. Find a description of processed FORS1 science data here

All reduced SCIENCE data are renamed. The renaming scheme can be found here.

<OBS_ID>/cal_raw

If measured, raw calibration frames (DPR.CATG=CALIB) produced by the OB are contained in this directory (not the ones from the calibration plan!).

Calibrations measured as part of the regular calibration plan are stored under the GEN_CALIB directory. Find a description of FORS1 raw calibration data here.

<OBS_ID>/cal_proc

The pipeline products of the raw attached calibrations are delivered here. Find a description of FORS1 product calibration data here. The CALIB products are renamed. The renaming scheme can be found here.

<OBS_ID>/logs

This directory has logging information about processing and packing of your data:

• Association Blocks (extension .ab)
• logs of the pipeline processing (extension .rblog)

Association Blocks (ABs) are ASCII files which contain all the information required to pipeline-process and pack data. This information includes the reduction recipe, the input raw file(s), the calibration products needed for processing, and the names of the final products. ABs provide the logical link between raw files, product files, and associated calibration files. They are described here.

The pipeline processing log is a record of the science reduction process, with a detailed log of reduction steps, results etc.

GEN_CALIB

This directory collects all calibration frames that are associated to your science data, and that have been measured as part of the regular calibration plan. It also contains their pipeline products and calibration frames of a general nature (like static line tables). Calibrations that have been measured by user-defined OBs and that have been used for pipeline processing of science data may be included here in addition. The directory has three subdirectories, two of which have further fine-structure:

GEN_CALIB
raw	proc	gen
BIAS FLAT LAMP STD	BIAS FLAT LAMP STD

GEN_CALIB/raw

Raw calibration files. Find a description of FORS1 raw calibration data here.

GEN_CALIB/proc

Calibration products derived from the raw calibrations. Find a description of FORS1 product calibration data here. The CALIB products are renamed. The renaming scheme can be found here.

GEN_CALIB/gen

General calibration data of static nature.

Additional or missing raw calibration files may be retrieved anytime from the generic ESO Archive form.

Calibration data are public while SCIENCE data have a proprietary period of one year.

GEN_INFO

This directory hosts some general information which is described under list files. It has the following subdirectories:

DIRECTORY	CONTENT
ObservingReports	nightlogs, OB report (HTML files)
scripts	executable scripts (presently one: print_all_reports)

LIST FILES

In the SM data package directory you will find the following listings:
   

FILE	CONTENT	WHERE
ServiceMode.html	this file	top
product_codes.html	a table describing the naming scheme for product files	top
README.txt	short additional information	top
archive_<RUN_ID>.txt	list of all proprietary files (SCIENCE, attached CALIBs) as read from the archive	GEN_INFO
qc0_<RUN_ID>.txt	log containing the comparison between the constraint set (airmass, moon distance, etc) and the actual values	GEN_INFO
NIGHTLOG.html	set of html files with nightlog and OB information	GEN_INFO/ObservingReports
list_sciRaw_<OBS_ID>.txt	summary of the fits files in each directory (these files are provided in text [*.txt] and PostScript format)	<OBS_ID>/sci_raw
list_sciProc_<OBS_ID>.txt	summary of the fits files in each directory (these files are provided in text [*.txt] and PostScript format)	<OBS_ID>/sci_proc
list_calRawGen_<TYPE>.txt	summary of the fits files in each directory (these files are provided in text [*.txt] and PostScript format) , where <TYPE>=BIAS,FLAT,LAMP,STD	GEN_CALIB/raw/<TYPE>
list_calProcGen_<TYPE>.txt	summary of the fits files in each directory (these files are provided in text [*.txt] and PostScript format) , where <TYPE>=BIAS,FLAT,LAMP,STD	GEN_CALIB/proc/<TYPE>

The executable script print_all_reports under GEN_INFO/scripts will print all ps files in your package.

Archive summary file: archive_<RUN_ID>

While the above listings are about files on the DVDs, the archive summary listing is the result of a query to the ESO Archive. It is useful as a check on the completeness of the ServiceMode package. All files created by OBs which have been generated by the PI are listed here. The list includes all SCIENCE files, and the attached calibrations, and acquisitions, if applicable. This list is the result of a query to the archive.

archive list
keyword table

QC0 summary file: qc0_<RUN_ID>

This file contains a report of quality control parameters ('QC level 0' where level 0 stands for Quality Control without pipeline processing) for your raw SCIENCE files. These parameters are airmass, seeing, moon distance, and fractional lunar illumination. They have been measured on site (column 'msrd'). They are compared to the required values as defined in your OBs ('targt') and flagged (OK/NOK).

The list is intended to give a rough indication of whether or not the required constraints have been obeyed. They should not be interpreted in a too formal way, however. E.g., there may be cases where the seeing was worse than required, but this was compensated by a longer exposure time. Check the night reports for details.

Note that the seeing values reported here are DIMM seeing values, they are not measured on the frame. If the alarm flag ("NOK") is set in the SEEING column, the DIMM seeing value was larger than your seeing constraint during the indicated obseration. However, in many cases, the delivered seeing in the instrument focal plane is better than the DIMM seeing. Whenever possible, the on-site observer has measured the focal plane immediately after execution to determine the success or failure of your observation. Thus, your observation may have been completed within your specifications, even if the SEEING alarm flag is set. Please review the affected observation carefully.

 

QC0 summary file
keyword table

Night logs and OB logs

This is a set of html files with an extraction of night log information and OB information. All relevant information about the nights contained in your package is included, as well as information about each OB in your delivery.

Point your browser to GEN_INFO/ObservingReports/NIGHTLOG.html and navigate either per night (labeled as 1) or per OB (2).

The html files also come as stripped-down, printer-friendly versions. The files are organized to have a summary on top, and details below. The OB summary replace the previously delivered OB summary files.

You can use either the navigation bar to jump to a specific night/OB, or use the up/down arrows (night logs only) to browse sequentially. The OB navigation bar uses colour coding to give you a quick impression about OB grading.

There are additional links to ambient condition information.

NOTE:

• The external links (asm: seeing, sky transparency etc.) will only work with network connection.
• The asm links require java-enabled browsers.
• The navigation bars read best with style-sheets and java-enabled browsers.

 

Sample nightlog files
NIGHTLOG.html

 

DATA FILES

Data delivered as a part of the SM package include:

Science Raw Data

All the FITS files produced by user-created Observation Blocks and that could be executed are delivered (categories SCIENCE and ACQUISITION). Some of the OBs may have been executed more than once. In particular, if time permitted, those OBs which produced data clearly out of the specified constraints were tried to be re-executed. Hence there may be more raw files per OB than originally specified.

Appropriate calibration data (these may be raw and/or processed [master] calibration data) are always delivered.

Master calibration data, if available, are delivered according to the instrument mode of the science raw data. Only exception are BIAS and DARK files and a bad pixel table, which are not mode-specific. Their names contain information about their function and instrumental modes.

The following master calibration data are in general included:

• bias frames under BIAS,
• twilight sky flats (IMG) under FLAT,
• reduced photometric standard star exposures and zeropoint tables (IMG) under STD,
• normalized screen flats (LSS, MOS) under LAMP,
• wavelength calibration files (LSS, MOS) under LAMP,
• files to correct spatial distortion (MOS) under LAMP
• reduced spectroscopic standard star spectra (LSS/MOS) under STD.

Raw calibration data, again mode-specific, are delivered if they fulfil at least one of the following conditions:

• they are specific to your programme (i.e. have been created by your OBs),
• they have not been used to construct master calibration files,
• they are standard star files and your science data are IMG mode,
• the instrument mode selected by you is presently not pipeline-supported.

Additional or missing calibration files may be retrieved from the  ESO Archive.

Check the FORS1 pipeline pages, specifically for calibration data.

If you have measured in the IMG, LSS or MOS mode, you can expect to receive pipeline-reduced files. Generally, all SCIENCE taken in IMG, LSS and MOS mode are reduced if they fulfil the following conditions:
 

• they are measured in supported (standard) modes
• master calibration data were available from the same night or at least within a few nights,
• LSS data: the 1port readout mode has been used.

Quality checks for the data reduction are applied. The quality of flattening and removal of 4port structure is checked. The degree of overexposure is determined. No file is rejected on the basis of these checks, however.
 

Science data are processed by the pipeline with the best available calibration data. Please note that ESO is not assuming any responsibility in respect to the usefulness of the reduced data. The adopted reduction strategy may not be suitable for the scientific purpose of the observations.

Check the FORS1 pipeline pages, specifically for more details on science reduction.

Photometric data included for IMG mode

For imaging projects, photometric standards are observed on every clear night through the filters being used that night. Clear nights are defined as nights when no clouds are visible. However, we can not certify these nights as photometric.

When available,
 

• the raw standard star files (DPR CATG = STD),
• the reduced standard star files (PRO CATG = REDUCED_STANDARD_IMG),

Pipeline-generated zeropoints per night are obtained by averaging all available standard star measurements. They are available here.

NOTE: some of the STANDARD star files (raw or reduced) may contain saturated pixels. So check them carefully if you want to use them. Pipeline-produced zeropoints are checked for overexposure.

PROBLEMS AND ISSUES

Filename Length Problem

To display or manipulate the FITS files with older versions of IRAF (before 2.11), you can:

1. copy these FITS files to your hard-disk and rename them with filenames <= 32 characters in length,
2. create symbolic links with filenames <= 32 characters in length to your CD-ROM files.

The FITS files use the ESO HIERARCH FITS keyword extensions standard to all ESO telescopes. Note that IRAF treats all ESO HIERARCH header lines as COMMENT lines, i.e. IRAF and IDL cannot automatically interpret the information provided in ESO HIERARCH header lines.

Please note that the RA and DEC keywords are recorded in degrees. To translate these keywords so that they can be used by IRAF you have to use the asthedit task in the noao.astutil package. The help file for this task gives an example of how to translate the ESO format to the IRAF format.

Several keywords have been added to the files to make them understandable for IRAF and IDL. The main keywords are OBJECT, FILTER, GRISM, SLIT, AIRMASS, UT, ST, and IMAGETYP. These are translations from some of the hierarchical keywords. The tasks in ccdred and xccdred understand these keywords if the instrument file is set up correctly. The image sections for 4-port readouts are defined as well in the headers and are understood by the tasks in xccdred.

Data related issues

Find other actual issues here.

Send comments to <smoehler@eso.org>  Last update: Nov 14, 2006