2011-06-08:a correction of the bad lines introduced into SINFONI raw data during instrument read-out is now corrected using the pipeline version 2.2.7.
2011-04-01:an improved wavelength calibration parameter set has been defined for SINFONI's J-band.
2010-10-01: the SINFONI pipeline version used during this period is version 2.2.3
2010-01-01: SINFONI data cubes are now corrected for atmospheric dispersion. This is applied to all grism s (J, H, K, and H+K) and for the pixel scales 25 and 100 mas. The correction exists in SINFONI pipeline versions 2.1.0+ and exists in the sinfo_rec_jitter recipe. The atmospheric dispersion correction relies on additional fits tables that provide the size of the correction based on airmass, waveband, and pixel scale. These files are available in the pipeline distribution (2.1.0 or later) and at the bottom of this page.
2009-07-01: packages are now created for both Service Mode and Visitor Mode.
2009-05-01: packages can now be downloaded via the ESO User Portal
2008-10-16: all packages now contain additional log information. These include the quality control scoring, any Observing Block comments, and data reduction comments (here)

DATA PACKAGES: GENERAL

Data packages have been delivered for period P87 (April-September 2011) and before. For new data, acquired after the begin of October 2011, data packages are no longer created. Users can access their raw data in electronic form through the ESO User Portal.

Data for VLT pre-imaging runs are processed and delivered as before.

For completeness, the structure of the historical packagesis described below.

Science data have been processed by the pipelines with the best available calibration data. Please note that ESO is not assuming any responsibility in respect to the usefulness of the reduced data. The adopted reduction strategy may not be suitable for the scientific purpose of the observations.

The top-level structure of the data package is as follows:

For each observation block (OB) that has been executed on Paranal, you find all measured raw data (FITS files) in a directory named by the OB number (FITS key HIERARCH.ESO.OBS.ID). If pipeline products exist, these are also added in the OB directory.

The GEN_CALIB directory collects all those calibration files (raw and products) that have been measured as part of the regular calibration plan, and calibration frames of a general nature (like static line tables). The GEN_INFO directory has general information, like data reports and night logs.

The tree shown above is the logical structure, which means that this is the way the data have been organized before they have been put onto media. Depending on the size of your package, the directories may be distributed across several media. It is a good idea to create the original tree on your local disk and then copy all files from the media into this tree.

<OBS_ID> (e.g. 179211)

For each executed observation block of your run, the package contains a directory with all measured data from that OB. All data under <OBS_ID> carry your run ID.

Note that some of your OBs may have been executed more than once. In particular, if time permitted, observatory staff try to re-execute OBs which produced data clearly out of the specified constraints. Check out the NIGHTLOG.html file for details (go to "OB information"). All data from OBs that have been executed multiple times are found in the same directory.

Each OB directory is further subdivided into subdirectories for science frames, calibration frames, and log files. In many cases, there will be science data only, but there may also be OBs with attached calibration data:

<OBS_ID>

acq sci_raw sci_proc cal_raw cal_proc logs

<OBS_ID>/acq

All acquisition frames (DPR.CATG=ACQUISITION) from the OB are contained in this directory. This directory only exists if such data exist.

<OBS_ID>/sci_raw

All raw science frames (DPR.CATG=SCIENCE) from the OB are contained in this directory.

<OBS_ID>/sci_proc

Here you find the pipeline-processed science data. The naming scheme can be found here.

<OBS_ID>/cal_raw

If measured, raw calibration frames (DPR.CATG=CALIB) produced by the OB are contained in this directory ("attached calibrations"). These are the ones which have been taken upon user's request in addition to the ones from the calibration plan.

Calibrations measured as part of the regular calibration plan are stored under the GEN_CALIB directory.

<OBS_ID>/cal_proc

The pipeline products of the raw attached calibrations are delivered here.

The CALIB products are renamed. The naming scheme can be found here.

<OBS_ID>/logs

This directory holds logging information about processing and packing of your data:

• Association Blocks (.ab)
• association logs (.alog)
• logs of the pipeline processing (extension .rblog)
• scoring results (.html) (optional)
• extraction from the nightlog, OB grade, QC0 report (.qcm)
• a reduction comment (.cmt) (optional)

Description:

• Association Blocks (ABs) are text files which contain all the information required to pipeline-process and pack data. This information includes the reduction recipe, the input raw file(s), the calibration products needed for processing, and the names of the final products. More ...
• Association logs are delivered since P80. They are a simplified version of ABs, designed to provide the association information essential for the user. More ...
• The pipeline processing log is a record of the science reduction process, with a detailed log of reduction steps, results etc.
• The scoring report is intended to give some feedback about the data quality. It is still experimental. More ...
• The nightlog file is an extracted version (per OB) of the summary qc0 report and the NIGHTLOG.html file (see below). More ...

That directory in addition holds QC plots, if available.

---

GEN_CALIB

This directory collects all calibration frames from the regular calibration plan that are associated to your science data. It also contains their pipeline products, and calibration frames of a general nature (like static line tables). Calibrations that have been measured by user-defined OBs and that have been used for pipeline processing of science data may be included here in addition.

The directory has four subdirectories (gen, logs, proc, raw), two of which have further fine-structure:

GEN_CALIB
raw	proc	gen	logs
cal1 cal2 cal3 cal4	cal1 cal2 cal3 cal4

GEN_CALIB/raw

Raw calibration files. These divide into raw file types (e.g. BIAS, FLAT etc.; see instrument specific section below).

GEN_CALIB/proc

Calibration products derived from the raw calibrations. These divide into file types like the raw calibration files, see instrument specific section below.

The CALIB products are renamed. The naming scheme can be found here.

GEN_CALIB/logs

Association Blocks, association logs and processing logs for the calibration files under GEN_CALIB. There might also be scoring logs (.html files).

GEN_CALIB/gen

General calibration data of static nature.

Additional or missing raw calibration files may be retrieved anytime from the generic ESO Archive form, or from the instrument specific forms.

Calibration data are public immediately while SCIENCE data normally have a proprietary period of one year.

---

GEN_INFO

This directory hosts some general information. It has the following subdirectories:

DIRECTORY	CONTENT
ObservingReports	nightlogs, OB report (HTML files), association report
scripts	executable scripts (presently one: print_all_reports)

 

The data package contains the following report files:

FILE	CONTENT	WHERE
README.html	the package portal page: point your browser here to find all information	top
ServiceMode.html	this file	top
product_codes.html	a table describing the naming scheme for product files	top
archive_<RUN_ID.txt	list of all proprietary files (SCIENCE, attached CALIBs) as read from the archive	GEN_INFO
qc0_<RUN_ID>.txt	list of all SCIENCE files, containing the comparison between the user constraint set and the actual values	GEN_INFO
NIGHTLOG.html	set of html files with nightlog, OB and association information	GEN_INFO/ObservingReports
list_sciRaw_<OBS_ID>.txt etc.	summary report of the fits files in each directory (these files are provided in text [*.txt] and PostScript format)	all data directories

The executable script print_all_reports under GEN_INFO/scripts can be used to print all postscript files in your package.

Archive report: archive_<RUN_ID>

While the above listings are about files in the package, the archive report is the result of a query to the ESO Archive. It is useful as a check on the completeness of the data package. All files created by OBs which have been generated by the PI are listed here. The list includes all SCIENCE files, and the attached calibrations, and acquisitions, if applicable.

archive report
keyword table
sample file

---

QC0 report: qc0_<RUN_ID>

This report is sent only for Service Mode runs.

This file contains a report of quality control parameters ('QC level 0' where level 0 stands for Quality Control without pipeline processing) for your raw SCIENCE files. These parameters are airmass, seeing, moon distance, and fractional lunar illumination. They have been measured on site (column 'msrd'). They are compared to the required values as defined in your OBs ('targt') and flagged (OK/NOK).

The list is intended to give a rough indication of whether or not the required constraints have been fulfilled. They should not be interpreted in a too formal way, however. E.g., there may be cases where the seeing was worse than required, but this was compensated by a longer exposure time. Check the night reports for details.

Note that the seeing values reported here are DIMM seeing values, they are not measured on the frame. If the alarm flag ("NOK") is set in the SEEING column, the DIMM seeing value was larger than your seeing constraint during the indicated obseration. However, in many cases, the delivered seeing in the instrument focal plane is better than the DIMM seeing. Whenever possible, the on-site astronomer has measured the focal plane immediately after or during execution to determine the success or failure of your observation. Thus, your observation may have been completed within your specifications, even if the SEEING alarm flag is set. Please review the affected observation carefully and check the night reports for details.

QC0 report
keyword table
sample file (.txt)

---

Night logs, OB logs and Association report

This is a set of HTML files with night log information, OB grading information and data association information. All relevant information about the nights contained in your package is included here, as well as information about each OB in your delivery.

Point your browser to GEN_INFO/ObservingReports/NIGHTLOG.html (or start from the package portal page, README.html) and navigate per night (labeled as 1), per OB (2) or per set of files (3).

The HTML files also come as stripped-down, printer-friendly versions. The files are organized to have a summary on top, and details below.

You can use either the navigation bar to jump to a specific night/OB/set of files, or use the up/down arrows (night logs only) to browse sequentially. The OB navigation bar (2) uses colour coding to give you a quick impression about OB grading. There are additional links to ambient condition information.

The association report (3) organizes your data and their association. It has two main levels: the OB (observing block), and the AB (association block) which collects raw file(s) and associated information like product files, calibration files, log files etc. This report gives you an impression how the data in your package are logically linked, while the listings in each directory give you a table of contents. File names in the association report may show up several times, e.g when a calibration file has been used for processing more than one science file.

NOTE:

• The external links (like the ASM links: seeing, sky transparency etc.) will only work with network connection.
• The ASM links require java-enabled browsers.
• The navigation bars read best with style-sheets and java-enabled browsers.

Sample nightlog files
NIGHTLOG.html

Known IRAF problems

• Filename Length. To display or manipulate the FITS files with older versions of IRAF (before 2.11), you can:
  - copy these FITS files to your hard-disk and rename them with filenames <= 32 characters in length;
  - create symbolic links with filenames <= 32 characters in length to your DVD files.
  
  
• Header Interpretation. ESO FITS files use the ESO HIERARCH FITS keyword extensions standard to all ESO telescopes. Note that IRAF treats all ESO HIERARCH header lines as COMMENT lines, i.e. IRAF and IDL cannot automatically interpret the information provided in ESO HIERARCH header lines. The problem may be solved using the tool hierarch28. Find information about this tool here.
  
  
• RA, DEC. Please note that the RA and DEC keywords are recorded in degrees. To translate these keywords so that they can be used by IRAF you have to use the asthedit task in the noao.astutil package. The help file for this task gives an example of how to translate the ESO format to the IRAF format.

Stand-alone FITS handling tools

This page contains an overview of the structure and content of the SINFONI data package. A detailed description of the Service Mode package for SINFONI data is found here.

For further information about SINFONI data and QC have a look at:

• SINFONI Quality Control and Data Flow Operations Homepage
• SINFONI data management with a list of FAQs.
• SINFONI data types: science raw | science products | calibration raw | calibration products
• SINFONI pipeline processing
• SINFONI quality control parameters and trending
• General information about SINFONI (Spectrgraph for INtegral Field Observations in the Near Infrared)
• the SINFONI Archive form
  

The naming scheme for SCIENCE product files can be found here.

The SINFONI raw and processed calibrations are defined by the calibration plan.

More about calibrations ...

The GEN_CALIB directory for SINFONI data is structured as follows:

All calibration frames are divided into a few general types for each setup used for the SCIENCE measurements. These types can be found as subdirectories of the proc and raw directories:

An example:

Your science data have been measured in OBJECT_NODDING mode, with the S4_H+K grism (fits header: ESO HIERARCH INS SETUP ID) at the 250 mas pixel scale (fits header: ESO HIERARCH INS OPTI1 NAME). There have been daytime calibrations for this mode and configuration, plus flat files, standard and psf stars for the night. You may expect the following calibration data (in order of appearance):

under <OBS_ID>/logs:

• the observation and processing logs for the specific observing Observing Block.

under GEN_CALIB/logs:

• the Association Blocks and processing logs for the files calibration files found in GEN_CALIB/proc

under <OBS_ID>/sci_raw:

• the observed SCIENCE raw created from each of your Observing Blocks

under <OBS_ID>/sci_proc:

• the pipeline SCIENCE products files created from each of your Observing Blocks

under GEN_CALIB/raw:

• raw North/South distortion frames (consisting of FIBRE_NS (~75 frames), FLAT_NS (2 frames), and WAVE_NS (2 frames))
• raw standard star frames STD and associated sky frames SKY_STD (~2-3 of each)
• raw DARK frames (~3 frames)
• raw lamp flats (FLAT_LAMP) (~10 frames)
• raw wavelength calibration frames (WAVE_LAMP) (~2 frames)

under GEN_CALIB/proc:

• the pipeline calibration product files created from the above raw CALIBs. These include:

• master dark
• master flat
• master bad pixel frame
• slitlet distance table
• wavelength map
• slit map
• distortion
• coadded standard and standard spectra
• master psf

under GEN_CALIB/gen:

• these are calibration data of a static nature. The following files will be included:

Known problems (instrument, pipeline, quality)

from 2009-12-09 (P84) sinfoni pipeline version 2.1.0: This pipeline version incorporates a major improvement in the sinfo_rec_jitter recipe. For all grism (J, H, K, and H+K) and for the pixel scales 25 and 100 mas, the data cubes are now corrected for atmospheric dispersion. Depending on the airmass and ADA rotation angle the source centroid can shift from one wavelength end of the cube to the other by several pixels. For the largest wavelength range cube (i.e. for the H+K band) and at the smallest pixel scale, this shift could be as large as 6-8 pixels. This shift is now corrected within sinfo_rec_jitter and is applied to the telluric STD star cubes, the PSF standard star cubes, and to all science cubes. For the 250 mas pixel scale, no correction is applied. This is due to the fact that the shift from atmospheric dispersion is less than 1 pixel at this pixel scale. The atmospheric dispersion correction relies on additional fits tables that provide the size of the correction based on airmass, waveband, and pixel scale. These files are available in the pipeline distribution (2.1.0).

2009-01-01 to PRESENT DATE: Important note for science observations with a large number of input frames:
The sinfoni pipeline coadds its input frames into data cubes in computer memory. Therefore, if your OB contains a large number of input images, or has large offsets, the default parameter that defines the maximum allowed cube size may quickly be exceeded. In general, pipeline versions older than 2.0.8 has the maximum allowable cube size <= 130 x 130 x 21 (X * Y dimension * number of input frames), depending on the amount of RAM available, and on the pipeline parameter configuration. If the science OB is larger than this, will produce a pipeline processing warning and create no product files. To increase the default maximum cube size, edit the parameter:

sinfo.objnod.mosaic-max-size in the pipeline configuration file sinfo_rec_jitter.rc.

The maximum allowed size for the cube mosaic is computed as 100 * sinfo.objnod.mosaic-max-size. Therefore if you projected cube size is 150 x 150 x 50, you will need a sinfo.objnod.mosaic-max-size of at least 11250. For the sinfoni pipeline versions 2.0.8 and newer, the default value of sinfo.objnod.mosaic-max-size has now been set to 14196.

2007-11-13 to PRESENT DATE: Occasionally, the SINFONI distortion frames will fail. This is not apparent at the time the OB is executed, but will only become evident when the distortion OB is pipeline processed. This failure manifests itself in fibre traces that are missing in some of the pseudo slits, do not cover the full length of the detector, or are too faint. As a result the pipeline cannot fit the full fibre traces and fails. These faulty distortions are found in roughly 10% of all distortions obtained. These bad distortions are detected by QC and a request is then made to Paranal to repeat them. To prevent users from downloading faulty distortions, the associated raw frames are generally hidden in the archive.

2008-08-01 (P81) Between the dates of 2008-08-01 and 2008-08-17, SINFONI data showed an increase in noise level located in the lower right hand side of the detector (affecting, primarily, slitlets numbers 23, 24, 25, and 26). This can be seen in the darks, flats, and science images, but was most apparent in the number of bad pixels detected from the dark and flat data reduction routines. If your data is affected by this, you can surpress the number of detected hot pixels by either using darks from outside of this affected period, or processing the darks with higher noise rejection thresholds (ie. set the dark recipe parameters bp_noise.thresh_sigma_factor to a higher value, and bp_noise.low_rejection and bp_noise.high_rejection to a lower value).

2006-07-28 (P77) The si_rec_jitter recipe of the sinfoni pipeline V 1.3.7 in the case of SCIENCE stacks without SKY frames has the following behaviour: The closest in time OBJECT frame is used to generate a SKY frame to be subtracted. For the many SCIENCE OBs in 2006-06 in quasi-staring mode; the offset positions on the sky may occcure multiple times within the stack of frames. Hence it might occur that the same area of the night-sky is subtracted as SKY from the OBJECT frame, resulting in fake sky subtraction. Pipeline version V.1.3.8 checks in addition to the closest in time rule also the position on the sky when selecting an OBJECT frame to be used for SKY subtraction. All SCIENCE data acquired between 2006-06-22 and 2006-07-01 hav ebeen reprocessed using pipleine version V1.3.7 but with non-default sky subtraction method=2. All SCIENCE data acquired after 2006-07-01 have been processed using pipeline version V1.3.8 using the above described check to handle non-jittered staring stacks.

2006-05-30 (P77) Distortions reduced with pipeline version 1.3.5 are not optimal. All science and calibrations have beeen reprocssed with version 1.3.7, but not optimal DISTORTION products SLIT_POS prodcuts might reside in the system (e.g. in the master calibration database). The not quite optimal products can be easily identified in the MFLAT_STACKED_DIST and FIBER_NS_DIST. These prodcuts are corrected for optical distortion; at thr right rim of these frames, the region without signal should be less than 4 pixel wide. Pipeline products of version 1.3.5 show up to 20 pixel without signal.

2005-10-08, 12, 17 The PSF of telluric standard standard stars shows unregular refraction rings, or a triangular shape. Most prominent in the 0.025 H+K setting.

2005-10-05 (P75) A saturated telluric standard star again caused a strong persistence effect (see 2005-05-23). Darks of DIT=600 are affected for three days.

2005-09-09 (P75) Since 2005-08-01 the dark frames show an increasing number of hot pixels in the region corresponding to slitlet #18.

2005-07-21 (P75) The si_rec_obsnod recipe of pipeline version 1.0.8 (and earlier version) cannot coadd science OBJ_OBS cubes for ADA.POSANG != 0.

2005-07-20 (P75) The SEQ.CUMOFFSET keys are missing in telluric standard star frames taken with the NGS template.

2005-06-15 (P75) The transmission of SINFONI is not a constant over the FOV. It varies up to 20%. For the 250 camera there is a minimum for slitlets #8, #9, and #10.

2005-06-02 (P75) Not a problem, but more a limitation: Science products (PRO.CATG = COADD_OBJ) are only generated when the cross section of the final coadded cube does not exceed a certain number of pixels. The upper limit is at about 128 x 128.

2005-05-23 (P75) The flat fields taken after 2005-04-01 show a flux reduction in slitlet #32 of about 40% correlated with a partial flux increase in slitlet #1 by 5-10%. Both slitlets build the lowermost and uppermost slide in the FOV.

2005-05-23 (P75) On 2005-04-04, several telluric standard stars saturated the chip. Hence a persistent signal occurred for about a week, mostly detectable in long DIT (e.g 300sec, 600sec, 900sec) dark frames. The persistence appears as several vertical stripes in all the raw frames, most striking in high BIT frames. The stripes are flagged as hot pixels in the hot pixel map. The largest gradient of fading occurred in the first three days after this event. All science raw frames taken up to 3-6 days after this event are more or less impact.

2005-04-28 (P75) Reconstructed and coadded science product cubes taken before 2005-03-30 suffer of two inconsistencies. a) slitlet #1 of individual cubes is not expanded from one spaxel to two pixel. b) the plane size of the cube is fixed to 80 x 80 pixels, hence large offset stacks will not fit completely in the fixed size cube (pipeline version 1.0.1).

2005-04-25 (P75) From science stacks without embedded SKY frames, that are pointing to the clear sky (no source), we derived typical sky subtraction residuals of about 60 ADU in the stacked product frames and about 200 ADU in the coadded product frames for DIT=600 sec. These residuals are due to sky emission line variations from exposure to exposure.

2005-04-25 (P75) The pipeline version 1.0.1 is used to reduce stacks of science frames. The dark products are not part of the cascade. Science data stacks with embedded SKY frames are well reduced. Science stacks without embedded SKY frames are reduced using one of the three offered reduction methods. We use a pair-wise subtraction of consecutive OBJECT frames to eliminate the SKY, before co-adding. Hence science products contain the negative spectra as well (similar to ISAAC LW chopping). If the user-specified jitter offsets are less than or of the order of the PSF, this method becomes critical.

2005-04-25 (P75) The dark frames, with DIT>200 show in a sporadic manner a few ADU higher signal of the even columns in the area of slitlet #25, (~ column 1922 ... 1985). The same behavior has been found for telluric standard star observations, psf calibrations and science frames. As en electronic effect, it may occur in all kind of exposures of the SINFONI array. The effect is monitored in DARK frames. Slitlet #25 can therefore contaminated in a sporadic manner by this effect.

2005-03-13 (P75) : Start of operations