[ ESO logo ]
The UVES reprocessed
data set

UVES REPROCESSING
QC HOME
Processing
[best viewing with style sheets and javascript enabled]

Quick links: data selection | master calibrations | pipeline | reduction | limitations | acknowledgements

UVES reprocessing

Note (2013-10-09): this first release of UVES echelle reprocessed data is superseded by the new release published on the ESO Data Products Spectral Query Form. Users should download UVES data only from that site. Documentation of the new release is provided here.

This data set consists of UVES echelle data processed by the ESO UVES pipeline (version 3.2). It has been executed by the Quality Control (QC) Group being part of the Data Flow Department.

The project provides a uniform reprocessing of UVES echelle point source data from the very beginning of operations. Quality assessment, quality control and certification have been integral parts of the project.

The immediate goal of this project has been to remove instrument signature from the raw spectra and provide extracted, wavelength-calibrated and flux-calibrated spectra ready for scientific analysis, or for more advanced processing. This advanced processing could involve the determination of radial velocities, the co-addition of spectra etc.

[ disclaimer ] UVES SCIENCE data have been processed by the pipeline with the best available calibration data. Please note that ESO is not assuming any responsibility in respect to the usefulness of the reduced data. The adopted reduction strategy may not be suitable for the scientific purpose of the observations.

[ disclaimer ] Pls note a major quality issue with the redmost setups (usually the setup centered at 860 nm), due to residual fringes in the extracted spectra. More ...


top Data selection

For UVES reprocessing, we have selected the largest possible homogeneous data set. UVES data types being part of the data set are:

  • point source ECHELLE data: they are described by FITS keys [HIERARCH.ESO.]DPR.TYPE= ECHELLE and DPR.TECH=OBJECT,POINT or OBJECT;
    these form the largest part of all UVES,ECHELLE data in the archive (roughly 52,000 raw files up to 2007-03-31).

Currently the following types of UVES data are not included in the product data set:

  • data using the image slicers (ECHELLE,SLICER) and/or the absorption cell (ECHELLE,ABSORPTION-CELL)
  • ECHELLE data from extended objects (DPR.TECH=OBJECT,EXTENDED)
  • data from the FLAMES/UVES instrument mode.

In general, no distinction has been made between Visitor Mode (VM) and Service Mode (SM) data, nor between standard settings and non-standard settings a priori. Hence, many of the available products have never been QC processed before and never been sent to PIs before.

For the data reduction, only already existing calibration solutions ("master calibrations") have been used. These have been processed by the QC group, reviewed, certified (if acceptable) and archived continuously as part of the day-to-day data flow operations. These calibration solutions have a high and homogeneous quality.

Science data have only been processed if master calibrations could be found in the archive. For certain "non-standard" settings master calibrations were not produced in the first years of UVES operations (until about 2003). These are e.g. 1x2 or 2x3 binnings, or central wavelengths other than 346, 390, 437, 520, 564, 580, 600, 860 nm.

Implicitly, this decision imposes an important additional selection criterium on science data processing. In order to produce an acceptable quality, the science data need to find archived master calibrations within a defined time window. This is usually +/- 3 days. This window matches the calibration plan at the Observatory, and it is motivated by the assumption that within that time, the ambient conditions are stable enough to remove instrument signature. While the Observatory always provided raw calibration data within that condition, the strategy to process them on the QC side was evolving with time.

Due to these effects, certain UVES ECHELLE data have not been reprocessed for the following reasons:

- Up to 2003-12, most data taken in Visitor Mode do not have master calibrations stored in the archive. After that date, VM data usually had master calibrations produced, and these science data have been reprocessed.

- The special fast read-out mode 625 kHz has been used mostly in VM, but rarely also in SM. No master calibrations so far are archived, and no science products are available.

- Generally, the first year of UVES operations (2000-02 until 2001-03) has seen fewer archived master calibrations than later years. This means, on average, a higher probability of so far unprocessed UVES data, because of missing master calibrations in the configured time window.

The selected time range is 2000-02-18 (start of UVES science operations) until 2007-03-31. All UVES raw data from that time range, and meeting the above criteria, have been processed.

top Master calibrations

The following types of master calibrations have been used for the science processing:

type (described by the FITS key pro.catg) name (first part) * content
MASTER_BIAS_{ccd}

UV_MBIA

master bias: created from 5 raw bias frames; removes bias level and structure
ORDER_TABLE_{ccd} UV_PORD order table: contains a description of the echelle order position, used for extraction
LINE_TABLE_{ccd} UV_PLI1 / PLI2 /PLI3 line tables (for each third of the extraction slit), giving the dispersion solution for the extracted spectra
MASTER_FLAT_{ccd} UV_MFLT master flat: created from three raw flats; used for: removing gain noise, removing the echelle function, removing slit noise
MASTER_RESPONSE_{ccd} UV_MRSP contains a response curve used for flux calibration; derived from selected sets of standard star measurements, collected for most (but not all) standard settings over typically a year (optional)
EXTCOEFF_TABLE   used to correct for atmospheric extinction (optional)

ccd = BLUE | REDL | REDU
* naming convention has slightly evolved

top Pipeline

The UVES pipeline v3.x has major improvements with respect to previous versions. The most important ones concern optimal extraction:

  • virtual resampling is used as a numerical fit to the cross-dispersion profile in the high-S/N regime
  • analytical fitting is used in the low-S/N regime
  • the pipeline does an initial quick S/N analysis and then adapts to the best choice.

The result is a robust and self-adaptive treatment of the optimum extraction which gives a stable signal tracing and a significant reduction of extraction ripples in comparison to earlier versions of the software. More ...

The pipeline software, and the users manual are available for download here.

Each pipeline product has the version used stored in the header key "HIERARCH ESO PRO REC1 PIPE ID". The UVES products published here used versions 3.0 to 3.2.

top Reduction steps

The main reduction steps have been:

  • de-bias
  • subtract interorder background
  • find and extract orders (using optimal extraction which also subtracts the sky signal)
  • extract master flat field with the same profile, divide by extracted flat field
  • apply dispersion solution and rebin -> extracted but not merged spectrum (UV_SWCA)
  • merge orders -> reduced spectrum + errors (UV_SRED + UV_SERR; UV_SSKY)
  • if master response curve exists: apply -> flux-calibrated, extinction-corrected spectrum + errors (UV_SFLX + UV_SFXE)

The codes in brackets stand for the products included in the data sets (details below).

The final product is the flux calibrated spectrum UV_SFLX, having physical units on both axes. However, this is not always available since the required master response curves exist only for the most commonly used setups. Therefore, the "reference" product is the fully reduced but not flux-calibrated spectrum (UV_SRED) which is always available in the data set. Both types of products have their associated error files.

The wavelength-rebinned spectra have not been corrected for barycentric and heliocentric motion. If you want to apply these corrections, their values may be found in the header, as keys HIERARCH.ESO.QC.VRAD.BARYCOR and HELICOR, resp.

Flux calibration available:

The master response curves (RC) for flux calibration have been compiled and maintained by QC Garching for the following standard settings (wavelength in nm):

  • 346, 390, 437 BLUE;
  • 564, 580, 860 REDL and REDU.

All other central wavelengths do not have response curves and therefore no flux calibrated products.

More on UVES flux calibration ...

Find more about the UVES products under "products" (here).

top Reduction limitations and problems

Find here a record of the currently known limitations and problems of the UVES science reduction recipe. It is, most likely, incomplete.

Residual fringes in the red: Many red spectra (in particular those taken with the standard setup at central wavelength 860 nm) display arteficial ondulations in the flat-fielded and extracted spectra. Their strength increases towards redder wavelengths. They have a typical spatial scale of 0.2-0.3 nm and a typical amplitude of roughly 10%. They result from residual fringes, i.e. an imperfect fringe cancellation.

The pipeline default flat-fielding method is called "extract". It was also used here and consists of the following sequence: 1. optimum extraction of science spectrum; 2. optimum extraction of the flat-field, using the same spatial profile; 3. divide both by each other. This method does not completely suppress fringes, as was realized after finishing the reprocessing project. An alternative method, called "pixel", essentially avoids the residual-fringes problem. It first divides the flat-field into the science spectrum pixel by pixel (hence the name), and then provides the extraction. The figure below demonstrates the difference:

Residual fringes problem. The upper spectrum shows the result spectrum of the default flat-fielding method ("extract"), as compared to the lower (scaled and shifted) spectrum obtained with the "pixel" method. The upper spectrum displays the residual fringes as 2-3 A wide ondulations at the 10% level. The lower spectrum is free from these artefacts.

As a drawback, there is only a rather simple order-merging mechanism available with the pixel method, which truncates the signal from the orders at specified positions, thereby causing larger interorder gaps in the redmost spectral parts. But generally this method yields greatly improved S/N for red spectra. It is currently available with the public release 3.4.0+ , but not with the archived product spectra.

Ripples: Under certain conditions small-scale ripples appear in the extracted spectra. Especially affected are high-S/N data with a small FWHM (good seeing). The effect scales roughly as (S/N)/FWHM2. It may be due to inappropriate virtual sampling of the cross-dispersion profile, or due to the non-linear response of the CCD. By default the cross-dispersion profile is sampled with 1/5 pixel size. If the profile is only a few pixels wide this results in a quasi-periodic sampling error due to the finite size of the numerical grid. If due to the extraction algorithm, this effect can be avoided by a smaller sampling size. Pipeline versions 3.2.x and later have a self-adaptive mechanism which finds and sets the appropriate sampling size after a coarse initial estimate of S/N. Earlier pipeline versions do not have this mechanism and have been executed always with sampling size 1/5. Those data are therefore more subject to the ripple effect.

An alternative in those cases is AVERAGE extraction which does not have sampling problems. However the products then may include cosmic rays. This current project is an automatic processing scheme, so there was no option to dynamically decide between AVERAGE and OPTIMUM extraction, but with an exported pipeline the user may decide to use this alternative extraction scheme.

Flux-calibrated spectra not always available: master-response curves are required to proceed from the extracted spectra to the flux-calibrated spectra. These are not available for all settings but only for the standard settings (the ones used in Service Mode) centred on: 346, 390, 437, 520, 564, 580, 600, 860 nm.

Wavelength range of flux-calibrated spectra: some of the master response curves used for flux calibration have the initial and the final 25 Angstrom cut off, hence the flux-calibrated spectra have a slightly smaller wavelength range than the extracted ones.

Flat-field signatures: the 346 nm spectra have relatively strong signature of the flat field extraction (repeating ondulations on the scale of one echelle order) (here)

Quite rarely, the 860 REDU spectra have flat-field related artefacts at the begin of echelle orders, right after the order gaps (here).

Flat-fielding and wavelength dispersion before 2001-10-01: in the early history of UVES data processing, the proper association between calibrations and science was done with all but one relevant instrument parameter: slit width. This means that master flats (UV_MFLT), and the dispersion tables (UV_PLI1,2,3), were correctly created and archived but only for one value of slit width per setting, although maybe raw science and calibration data with several such values were existing. In other words: raw calibration data were complete from the beginning of UVES operations, but master calibrations in the archive are incomplete, for certain dates before 2001-10-01. With the operational setup of the UVES reprocessing project, master calibrations are not recreated but just downloaded from the archive. Therefore, the correct match of slit width and calibrations cannot be guaranteed for science data before that date. Usually, this has little impact on the data quality, but in extreme cases (e.g. science data taken with the 0.3 arcsec slit, master_flat with 1.5 arcsecs) this may affect the extraction quality and may explain artefacts, especially with a period of one echelle order. Check here for a particularly impressive example.

Order gaps: The 860 REDU spectra have incomplete spectral coverage (echelle orders are truncated by the chip). This is a feature of the instrument.

Object centering: if the object is not well-centered in the slit, extraction may fail or produce all kinds of artefacts, including extraction ripples (here).

Object not point-like: if the object has a complex structure (binary, background, extended object), the extraction may fail or produce artefacts. The algorithm assumes a point source, with a near-gaussian profile shape (although the profile is measured and fitted, for stability, with a more complex function). It locks to the brightest signal in the slit. Complex sources should better be extracted with the EXTENDED method and treated individually.

The certification process has not been very sensitive to complex object structure. It may have happened that corrupted product spectra of an extended object, or a binary object, have passed certification without comment!

Emission-line objects: without a traceable continuum, the pipeline may produce artefacts. In many cases such products can be recognized automatically by an exceptionally small (< 0.5") or large (> 3") FWHM, and a comment is provided. Still the spectra may be useful since the emission lines may be extracted correctly (e.g. see here). But such data have to be inspected with care.

Radial velocity correction: The wavelength-rebinned spectra are not corrected for barycentric and heliocentric motion. If you want to apply these corrections, their values may be found in the header, as keys HIERARCH.ESO.QC.VRAD.BARYCOR and HELICOR, resp.


top Acknowledgements

This project was led and conducted by Reinhard Hanuschik. It was made possible only by key contributions from a bunch of people at ESO. Special thanks go to: Jonas Larsen and Andrea Modigliani for their superb support with the UVES pipeline; to John Pritchard for advice on many aspects of state-of-the-art UVES QC operations; to Dieter Suchar and his staff for doing an excellent job to set up the processing platform; to Alain Smette and Cedric Ledoux for suggesting significant improvements to the optimal extraction algorithm; and to Remco Slijkhuis for developing and installing the prototype product ingestion tool.


[QC group logo]  
[ESO][Index][Search][Help][News]