VERSION:

P81 (April 2008)

GENERAL

Runs performed in Service Mode receive a set of data on media which presently are DVDs. A description on how to read media produced by the ESO/ST-ECF archive is available here.

This page contains an overview of the structure and content of the data packages for SINFONI.

For further information please look at: SINFONI Quality Control and Data Flow Operations Homepage

• General description SINFONI data management with a list of FAQs

DIRECTORY STRUCTURE

The top-level structure of the package for your VLT run is as follows:

For each observation block (OB) that has been executed on Paranal, you find all measured raw data (FITS files) in a directory named by the OB number (FITS key OBS.ID). If pipeline products exist, these are also added within the OB directory. The OB package structure is hoped to be more user-oriented than the previously used DATE structure.

The GEN_CALIB directory collects all calibration files (raw and products) that have been measured as part of the regular calibration plan, and calibration frames of a general nature (like static line tables). The GEN_INFO directory has general information.

The tree shown above is the logical structure, which means that this is the way the data have been organized before they have been put onto media. Depending on the size of your package, the directories may be distributed across more than one DVD. It is a good idea to create the original tree on your local disk and then copy all files from the media into this tree.

<OBS_ID>

For each executed observation block of your VLT run, a directory containing all measured data is created. All data under <OBS_ID> carry your run ID.

Note that some of your OBs may have been executed more than once. In particular, if time permitted, we tried to re-execute OBs which produced data clearly out of the specified constraints. Check out the NIGHTLOG.html file for details (go to "OB information"). All data from OBs that have been executed multiple times are found in the same directory.

Each OB directory is divided into subdirectories for acquisition frames, science frames, calibration frames, and log files. In most cases, they contain science and acquisition data, but there may also be OBs with attached calibration data. Depending on the nature of the OB, not all subdirectories may exist:

<OBS_ID>

acq cal_proc cal_raw logs sci_proc sci_raw

 

<OBS_ID>/sci_raw

All raw science frames (DPR.CATG=SCIENCE) for the OB are contained in this directory. For descriptions of raw science data look at: SINFONI pipeline pages.

<OBS_ID>/sci_proc

This directory contains the pipeline products derived from the science frames. Descriptions of the reduced science products can be found on the SINFONI pipeline pages.

All pipeline products (calibration and science) are renamed. The SINFONI renaming scheme is documented.

<OBS_ID>/cal_raw

If measured, raw calibration frames (DPR.CATG=CALIB) produced by the OB are contained in this directory. These could be:

• attached arc lamp or flat field exposures
• user-specified telluric standard stars
• user-specified psf calibration stars

Calibrations measured as part of the regular calibration plan can be found in the GEN_CALIB directory. Raw calibration data are described here: SINFONI

<OBS_ID>/cal_proc

Pipeline products derived from raw calibrations of the OB are collected here. Please look at the SINFONI calibration pipeline pages for descriptions of the products. The SINFONI renaming scheme is documented.

<OBS_ID>/logs

This directory has logging information about processing and packing of your data:

• Association Blocks (extension .ab)
• logs of the pipeline processing (extension .rblog)

Association Blocks (ABs) are ASCII files which contain all the information required to pipeline-process and pack data. This information includes the reduction recipe, the input raw file(s), the calibration products needed for processing, and the names of the final products. ABs provide the logical link between raw files, product files, and associated calibration files.

While Association Blocks are a technical piece of information, the reduction log is a somewhat more user-oriented account of the science reduction process, with a detailed log of reduction steps, results etc.

GEN_CALIB

This directory collects all raw calibration frames that are associated to your science data, and that have been measured as part of the regular calibration plan. It also contains their pipeline products and calibration frames of a general nature (like static line tables). Calibrations that have been measured by user-defined OBs and that have been used for pipeline processing of science data may be included here in addition. The directory has three subdirectories, two of which have further fine-structure:

GEN_CALIB
gen	proc	raw
	DET SPEC_ARC SPEC_FLAT SPEC_OTHER	DET SPEC_ARC SPEC_FLAT SPEC_OTHER

GEN_CALIB/raw

Raw calibration files. Raw calibration data are described on the SINFONI pipeline page.

GEN_CALIB/proc

Calibration products derived from the raw calibrations. Please look at the SINFONI calibration pipeline pages for descriptions of the products. The renaming scheme is available.

All calibration frames are divided into a few general types. This types can be found as subdirectories of the proc and raw directories:

TYPE/SUBDIRECTORY	SINFONI FLAVOR	CONTENT	DPR.TYPE
DET	day-time calibration	Detector dark frames	DARK
SPEC_ARC	day-time calibration	Spectroscopic arc lamp exposures	WAVE,LAMP
SPEC_FLAT	day-time calibration	Spectroscopic flat field exposures	FLAT,LAMP
SPEC_OTHER	bi-monthly	Detector linearity	LINEARITY,LAMP
SPEC_OTHER	bi-monthly	Distortion (NorthSouth Test)	DISTORTION,FIBRE,NS DISTORTION,FLAT,NS DISTORTION,WAVE,NS
SPEC_STD	night-time calibration	telluric standard star	STD SKY,STD
SPEC_STD	night-time calibration	PSF calibrator source	PSF-CALIBRATOR SKY,PSF-CALIBRATOR

GEN_CALIB/gen

General calibration data of static nature. You may expect the following files:

FILE NAME	PRO.CATG	DESCRIPTION
argonJ.tfits neonK.tfits xenonHK.tfits xenonargonH.tfits	REF_LINE_ARC	reference arc line tables
REF_BP_MAP.fits	REF_BP_MAP	reference bad pixel map
FIRST_COLUMN.tfits	FIRST_COL	the first column (a single number)
drs_setup_wave.tfits	DRS_SETUP_WAVE	band specific parameters for wavecal recipe

Additional or missing raw calibration files may be retrieved anytime from the  ESO Archive. Calibration data are public while SCIENCE data have a proprietary period of one year.

Bias frames usually come in sets of five raw files which are stacked to one master bias. They are provided only once per week since they are in general very stable.

...

GEN_INFO

This directory hosts some general information:

• A file archive_<RUN_ID> containing a report of all raw files with your run ID in the ESO Archive (.txt and .ps version);
• a file qc0_<RUN_ID> containing a report of quality control parameters ('QC level 0') for your raw SCIENCE files (.txt and .ps version) ;
• a directory ObservingReports; here you find night log information, including details about the grading of and the ambient conditions for your OBs.

You will find the following list files:  

FILE	CONTENT	WHERE
ServiceMode.html	this file	top
README.txt	short additional information	top
product_codes.html	list of abbreviations	top
NOTES.txt	some notes	GEN_INFO
NOTES_P75.txt	some notes, specific to observing period P75	GEN_INFO
archive_<run.id>.txt archive_<run.id>.ps	list of all proprietary files (SCIENCE, attached CALIBs) as read from the archive	GEN_INFO
qc0_<RUN_ID>.txt	log containing the comparison between the constraint set (airmass, moon distance, etc) and the actual values	GEN_INFO
NIGHTLOG.html	set of html files with nightlog and OB information	GEN_INFO/ObservingReports
list_sciRaw_<OB.ID>.txt, list_sciProc.txt, list_calRaw.txt, list_calProc.txt,	summary of the fits files in each OB directory (these files are provided in text [*.txt] and PostScript format)	<OBS_ID>/sci_raw, <OBS_ID>/sci_proc, <OBS_ID>/cal_raw, <OBS_ID>/cal_proc
list_calRawGen_<type>.txt, list_calProcGen_<type>.txt, list_calGen_gen.txt	summary of the fits files in the GEN_CALIB directories (provided in text [*.txt] and PostScript format)	GEN_CALIB/raw/DET, etc. GEN_CALIB/proc/DET, etc. GEN_CALIB/gen

Known problems (instrument, pipeline, quality)

2005-03-13 (P75) : Start of operations

2005-04-25 (P75) : The dark frames, with DIT>200 show in a sporadic manner a few ADU higher signal of the even columns in the area of slitlet #25, (~ column 1922 ... 1985). The same behavior has been found for telluric standard star observations, psf calibrations and science frames. As en electronic effect, it may occur in all kind of exposures of the SINFONI array. The effect is monitored in DARK frames. Slitlet #25 can therefore contaminated in a sporadic manner by this effect.

2005-04-25 (P75) The pipeline version 1.0.1 is used to reduce stacks of science frames. The dark products are not part of the cascade. Science data stacks with embedded SKY frames are well reduced. Science stacks without embedded SKY frames are reduced using one of the three offered reduction methods. We use a pair-wise subtraction of consecutive OBJECT frames to eliminate the SKY, before co-adding. Hence science products contain the negative spectra as well (similar to ISAAC LW chopping). If the user-specified jitter offsets are less than or of the order of the PSF, this method becomes critical.

2005-04-25 (P75) From science stacks without embedded SKY frames, that are pointing to the clear sky (no source), we derived typical sky subtraction residuals of about 60 ADU in the stacked product frames and about 200 ADU in the coadded product frames for DIT=600 sec. These residuals are due to sky emission line variations from exposure to exposure.

2005-04-28 (P75) Reconstructed and coadded science product cubes taken before 2005-03-30 suffer of two inconsistencies. a) slitlet #1 of individual cubes is not expanded from one spaxel to two pixel. b) the plane size of the cube is fixed to 80 x 80 pixels, hence large offset stacks will not fit completely in the fixed size cube (pipeline version 1.0.1).

2005-05-23 (P75) On 2005-04-04, several telluric standard stars saturated the chip. Hence a persistent signal occurred for about a week, mostly detectable in long DIT (e.g 300sec, 600sec, 900sec) dark frames. The persistence appears as several vertical stripes in all the raw frames, most striking in high BIT frames. The stripes are flagged as hot pixels in the hot pixel map. The largest gradient of fading occurred in the first three days after this event. All science raw frames taken up to 3-6 days after this event are more or less impact.

2005-05-23 (P75) The flat fields taken after 2005-04-01 show a flux reduction in slitlet #32 of about 40% correlated with a partial flux increase in slitlet #1 by 5-10%. Both slitlets build the lowermost and uppermost slide in the FOV.

2005-06-02 (P75) Not a problem, but more a limitation: Science products (PRO.CATG = COADD_OBJ) are only generated when the cross section of the final coadded cube does not exceed a certain number pixel. The upper limit is at about 128 x 128.

2005-06-15 (P75) The transmission of SINFONI is not a constant over the FOV. It varies up to 20%. For the 250 camera there is a minimum for slitlets #8, #9, and #10.

2005-07-20 (P75) The SEQ.CUMOFFSET keys are missing in telluric standard star frames taken with the NGS template.

2005-07-21 (P75) The si_rec_obsnod recipe of pipeline version 1.0.8 (and earlier version) cannot coadd science OBJ_OBS cubes for ADA.POSANG != 0.

2005-09-09 (P75) Since 2005-08-01 the dark frames show an increasing number of hot pixels in the region corresponding to slitlet #18.

2005-10-05 (P75) A saturated telluric standard star caused again a strong persistence effect (see 2005-05-23). Darks of DIT=600 are affected for three days.

2005-10-08, 12, 17 The PSF of telluric standard standard stars shows unregular refraction rings, or a triangular shape. Most prominent in the 0.025 H+K setting.

2006-05-30 (P77) Distortions reduced with pipeline version 1.3.5 are not optimal. All science and calibrations have beeen reprocssed with version 1.3.7, but not optimal DISTORTION products SLIT_POS prodcuts might reside in the system (e.g. in the master calibration database). The not quite optimal products can be easily identified in the MFLAT_STACKED_DIST and FIBER_NS_DIST. These prodcuts are corrected for optical distortion; at thr right rim of these frames, the region without signal should be less than 4 pixel wide. Pipeline products of version 1.3.5 show up to 20 pixel without signal.

2006-07-28 (P77) The si_rec_jitter recipe of the sinfoni pipeline V 1.3.7 in the case of SCIENCE stacks without SKY frames has the following behaviour: The closest in time OBJECT frame is used to generate a SKY frame to be subtracted. For the many SCIENCE OBs in 2006-06 in quasi-staring mode; the offset positions on the sky may occcure multiple times within the stack of frames. Hence it might occur that the same area of the night-sky is subtracted as SKY from the OBJECT frame, resulting in fake sky subtraction. Pipeline version V.1.3.8 checks in addition to the closest in time rule also the position on the sky when selecting an OBJECT frame to be used for SKY subtraction. All SCIENCE data acquired between 2006-06-22 and 2006-07-01 hav ebeen reprocessed using pipleine version V1.3.7 but with non-default sky subtraction method=2. All SCIENCE data acquired after 2006-07-01 have been processed using pipeline version V1.3.8 using the above described check to handle non-jittered staring stacks.

A detailed description of all problems is given in the pipeline section of theSINFONI QC web pages

Known IRAF Problems

• Filename Length Problem. To display or manipulate the FITS files with older versions of IRAF (before 2.11), you can:
  - copy these FITS files to your hard-disk and rename them with filenames <= 32 characters in length;
  - create symbolic links with filenames <= 32 characters in length to your DVD files.
  
  
• Header Interpretation Problem. ESO FITS files use the ESO HIERARCH FITS keyword extensions standard to all ESO telescopes. Note that IRAF treats all ESO HIERARCH header lines as COMMENT lines, i.e. IRAF and IDL cannot automatically interpret the information provided in ESO HIERARCH header lines. The problem may be solved using the tool hierarch28. Find information about this tool here.
  
  
• RA, DEC. Please note that the RA and DEC keywords are recorded in degrees. To translate these keywords so that they can be used by IRAF you have to use the asthedit task in the noao.astutil package. The help file for this task gives an example of how to translate the ESO format to the IRAF format.

Stand-alone FITS handling tools

Find information about FITS header handling tools (e.g. dfits, fitsort, hierarch28) here.