Quick links: overview | release content | data organization | data selection | release notes | master calibrations | pipeline description | data reduction | data quality | known features&issues | file types | file structure | file size | acknowledgement text | appendix
ESO Phase 3 Data Release Description
UVES Echelle science products
Important note: We have recently reprocessed all data until 2007-03-31. The reprocessed data have a recently discovered artefact fixed. While it occurred rarely, it was potentially dangerous because it could be mistaken for a spectral emission line feature. The reprocessed data with longer exposure times also benefit from a much better cosmic ray suppression. Both improvements are due to fixing a badly chosen processing parameter, reduce.extract.kappa. Until 2007-03-31, its value was 20, while its more appropriate value 10 (which is also the pipeline default) was and is used for later data. These later data are therefore not affected by the issues and have not been reprocessed.
A third improvement affects all RED data acquired until 2002-04-10. The pipeline versions used for processing the master calibrations until that date had a bug with the error computation for the RED upper CCD, resulting in SNR values being too low. This is also fixed with the new release.
The reprocessed data come as version 2 (with the version 1 data still available on request, for comparison). We have decided to reprocess all UVES data until the date 2007-03-31, no matter if affected by the reported issues or not. This means: all UVES data from 2000-03-11 until 2007-03-31 now come as version 2, all UVES data acquired on 2007-04-01 and later come as version 1.
Abstract. This is the release of reduced 1D spectra from the UVES spectrograph, ECHELLE mode (as opposed to the FIBRE mode), point sources (as opposed to extended sources), taken without or with the image slicer. Data taken with the absorption cell are also included in the data release. This release includes re-processed UVES data, as well as current and future UVES data not processed before at ESO. The processing scheme is as homogeneous as possible.
The selected data cover the vast majority of the entire UVES data archive, from the begin of operations in March 2000 until present. The data have been reduced with the UVES pipeline, versions uves-5.1.3 and higher, recipe uves_obs_scired. All data have their instrument signature removed (de-bias, flat-fielded), with the exception of the absorption-cell data which have the signature of the iodine cell not removed. Then the data have been extracted, wavelength-calibrated and order-merged. Whenever possible they have been flux-calibrated. The pipeline output products have been converted to the standard binary table, following the ESO 1D spectroscopic standard.
The processing is performed by the Quality Control Group using a dedicated workflow tool in an automated process. The pipeline processing uses the archived, closest-in-time, quality-controlled, and certified master calibrations. It is important to note that the reduction process itself is automatic, while the quality assessment and certification of the master calibrations was (and still is) human-supervised.
The overall data content is not fixed but grows with time as new data are being acquired and processed (approximately with monthly cadence and with a delay of 1 or 2 months).
The data format follows the ESO 1D spectroscopic standard for so-called phase-3 data products. Each spectrum is a multi-column binary table. There is one product file for each parent raw file. The typical observing pattern of a simultaneous observation in the blue arm and in the red arm is propagated to the data products: there is one blue product and one red product if the original observation used both arms.
This data release offers science-grade data products, with the instrumental signature removed, flux-calibrated (if possible), with error estimates and a list of known shortcomings.
These data products are considered to be ready for scientific analysis. They are expected to be useful for any kind of high-resolution spectroscopic research, including line profile studies and radial velocity studies. For studies of energy distributions the investigator should keep in mind that the UVES instrument is a high-resolution spectrograph, designed for stability and throughput but not for high-precision flux calibration. There are slit losses, and many observations did not care about photometric conditions.
Data have been pipeline-processed with the best available calibration data. However, please note that the adopted reduction strategy may not be optimal for the original scientific purpose of the observations, nor for the scientific goal of the archive user.
The UVES_ECHELLE release is a stream release. The overall data content is not fixed but grows with time as new data are being acquired and processed (approximately with monthly cadence and with a delay of 1 or 2 months). The data is tagged with "UVES_ECHELLE" in the ESO archive user interface.
The first data were published under UVES_ECHELLE in September 2013, with UVES data from the begin of operations (March 2000 ) until May 2013. Since then the data have been added in a processing stream in about monthly intervals. In 2017 an issue with the RED data taken acquired until 2007-03-31 was discovered. The entire UVES data (BLUE and RED) until that date were reprocessed and are available as version 2. All UVES data acquired after that date were not affected by the issue and come as version 1.
The downloaded data come with their technical archive names. The README file coming with every download contains the information necessary to properly rename the files and establish their association. The fundamental unit of the renaming process is to capture column #2 in the README file (technical name starting with ADP) and column #3 (original name starting with UV_) and use something like 'mv $2 $3'.
Data selection is entirely rule-based. It is organized along the following criteria:
Note that the largest fraction of UVES point-source data comes as 'normal' ECHELLE (about 80-90%). Second largest fraction is ECHELLE,SLICER (about 10%). The remaining few percent divide into ABSORPTION-CELL data, with or without the SLICERs. Physical slicers come in three different varieties which however has no impact on the data products.
A priori, no selection is made on the basis of observing mode (visitor or service), nor on settings. UVES ECHELLE settings are defined by the combination of binning, central wavelength and dichroic filter (more...).
De facto we can process only those science data for which certified master calibrations can be found in the archive. These master calibrations were also processed by the Quality Control Group but much earlier in time, close to the date of acquisition in order to provide quick quality feedback to the Observatory. The operational selection pattern for master calibrations evolved over the years. It reflects pipeline evolution and calibration plan changes, and also manpower constraints. In the early years of UVES, the processing focus was on calibrations taken in standard settings (based on central wavelength) and for Service Mode (SM) science data. The fast read mode (4port, 625kHz) was not supported either. Later, the calibration scheme was extended to non-standard settings (at best-effort basis), ignoring the Service/Visitor Mode (VM) distinction.
Therefore the completeness ratio for the UVES Echelle science data in this stream is inhomogeneous. For the early years (until 2002) almost no VM data are included, and no SLICER data. Later, the fraction is close to complete. Processed SLICER data are available as of 2003-04-13. Find the list of run IDs with incomplete coverage, or no coverage at all, in the Appendix.
Note that the fact that a certain UVES raw file does not have a product in this release does not imply a quality issue with the raw data. It is more likely that this raw file falls into the category of VM science taken in the early years of UVES in non-standard settings or in the fast read mode.
Such data would most likely process fine manually.
In general, the master calibrations were processed with different (earlier) pipeline versions than the science data in this stream. This is not considered an issue since the calibration recipes of the UVES pipeline did not strongly evolve with time.
Science data with the PROG.ID starting with 60. or 060. have been de-selected, considering them as test data. Data taken at daytime (with obviously wrong ‘SCIENCE’ tag) have been ignored. Otherwise the metadata tag ‘SCIENCE’ has been blindly accepted from the raw data (originally defined by the PI), thus including sometimes standard stars (click here for an example). This is often evident from the OBJECT metadata key. Also, there are rare cases when test observations were executed under the SCIENCE label. Most very short exposures with no signal fall into that category but have not been suppressed.
All data (both with and without the SLICERs) have been processed up to the wavelength calibration step. The last step, the flux calibration, was undertaken with master response curves (see below). These do not exist for all settings, not even for all standard settings. The products which could be flux-calibrated have the highest reduction level (FLUXCAL=ABSOLUTE, ORIGFILE names start with UV_SFLX), while the others have FLUXCAL=UNCALIBRATED, names start with UV_SRED (if taken without slicer and without absorption-cell), UV_SREA (without slicer, with absorption-cell), UV_SRES (with slicer, without absorption-cell), or UV_SREB (with slicer and absorption-cell). There are SLICER data which could be flux-calibrated (taken e.g. in the 580 setting), their ORIGFILE name also starts with UV_SFLX.
There is no raw data selection based on quality. Likewise, we have not considered OB grades: the observations might have any grade between A and D, or X. The availability, or non-availability, of a particular file in this release does not infer any claim about the data quality. E.g. saturated pixels are not flagged by the pipeline. The user is asked to carefully check the data for those cases.
Each product file has exactly one parent raw file. Multiple observations within the same OB have not been combined. Data from multiple visits of the same target have not been combined either.
The data reduction used the standard UVES pipeline recipe uves_obs_scired. Find a description of the UVES science recipe in the User Manual.
All recipe parameters used were default, except for:
Note that for these red data (central wavelength ≥ 760nm) the flux scale in the data products needs to be corrected due to a processing bug, see 'Known features and issues'.
The extraction method (reduce.extract.method) was ‘optimal’ for the normal (non-sliced) ECHELLE data. Because of the multiple stellar signal in case of SLICER data (3-5 depending on the slicer used), these were extracted with the ‘linear’ method. In ‘optimal’ mode, the pipeline uses an initial SNR estimate, based on which it selects the appropriate cross-order extraction profile (Gauss, Moffat, Virtual). Find more details in the User Manual, section 11.1.13.
The processing parameter reduce.extract.kappa (threshold for cosmic ray rejection) was set to 20 by mistake, in the earlier version (1.0) of this data stream, instead of the much better standard value of 10. The REDU part of many UVES spectra was affected by this wrong choice, for data acquired until 2007-03-31. The bad choice resulted in virtually no rejection of cosmic events, and in some spectral artefacts in unfavourable (though rare) cases. This has been fixed in the version 2 of the data stream which is the default. If in doubt, check the header for the processing parameter reduce.extract.kappa.
ccd = BLUE | REDL | REDU
This list is the same for data taken with and without the image slicers.
Information about the UVES pipeline (including downloads and manual) can be found here. The QC pages contain valuable information about the UVES Echelle data, their reduction and the pipeline recipes.
The main reduction steps are the following (see the displays below; find more details in the QC pages and in the Pipeline User Manual). The bias level and structure is removed, then, depending on the flat-fielding method, either the science data are divided by the flat and then extracted using the order table (‘pixel’ method), or the other way round (‘extract’). Upon extraction, the optimal extraction algorithm determines the sky background and removes the cosmics. SLICER data (being extracted with method 'linear' meaning a simple summing up of the total signal in the extraction slit) have no sky background determined: the observing method is suitable for bright sources and short exposure times, hence sky background can be neglected. Then the extracted orders are wavelength-calibrated (using the line tables), rebinned, and merged. This is the final step, if no master response curve is available, otherwise that calibration solution is used to flux-calibrate the extracted spectrum. If so, the spectrum is delivered in physical units. Otherwise it comes in arbitrary units (counts) and contains those instrument or atmospheric signatures which cannot be removed by the flat field (chromatic coating efficiencies, atmospheric extinction) or have been added by the flat field (spectral response of the calibration lamp).
Note that in the case of the absorption-cell data, the signature of the iodine cell has not been removed. (For that purpose there are flat-field data available from the ESO archive with the iodine cell turned on. It was felt that the reduction with these calibration data is too delicate to be executed in an automatic way.)
The reduction of the SLICER data follows exactly the same steps, and uses the same types of calibrations, as the reduction for the non-sliced ECHELLE data. The only exception is the extraction method (linear).
The flux calibration has been applied for the following central wavelengths: 346, 390, 437, 564, 580, 760, and 860 nm.
The UVES Echelle products are wavelength calibrated. No corrections for barycentric or heliocentric motion have been applied. The corresponding values have been calculated by the pipeline from the observation and are stored in the header (HIERARCH.ESO.QC.VRAD.BARYCOR or HELICOR, in km/s).
No correction for telluric absorption lines has been applied. The wavelength scale is in air.
In some cases the PIs have specified attached arclamp or flat calibration during the night, immediately after the science data. We do not guarantee that these special calibration data have been used for the reduction. More likely is that the standard daytime calibrations have been applied.
The UVES pipeline creates the following product files:
While these pipeline product files are separate (image) fits files, they are finally converted into a single binary table fits file which is the delivered data product, with the wavelength scale as first column and all other products as further columns (see ‘File Structure’ below).
Contrary to the product files of the UVES pipeline (which generates one product for each detector), the converted product from the RED arm (as downloaded from the archive) is a single file. The two non-overlapping red segments have been merged in a single spectrum with non-equal sampling (and no degradation). There is always an unavoidable gap between the two spectral segments due to the geometrical gap between the two red detectors.
There is some internal quality control on the pipeline process, monitoring:
This information has largely been used to improve and fine-tune the processing process. An individual one-by-one inspection of the products has been unaffordable.
SNR. The signal-to-noise ratio of the spectra is measured by the ratio of FLUX/ERR (or FLUX_REDUCED/ERR_REDUCED giving the equivalent result). Its chromatic slope is as complex as the spectrum itself. As representative numbers, the pipeline calculates an average per echelle order (stored in the header per order as 'HIERARCH ESO QC ORDnn OBJ SN' for BLU spectra, and as 'HIERARCH ESO QC CHIPi ORDnn OBJ SN' for RED spectra). From these numbers, a median has been calculated (omitting the respective first and last order) which is the number quoted as SNR on the query interface and in the header. Note that, as a peculiarity in the red wavelength ranges, those files reduced with the reduce.ffmethod=pixel method have a higher SNR in the spectral regions with overlapping echelle orders (see the lower panel labelled "S/N" here).
Nature of source. The pipeline applies some assumptions on the source (these assumptions apply to the non-sliced case): :
Flux calibration. The flux calibration is good enough for obtaining a coarse energy distribution but is not of spectrophotometric quality. The slit losses are unknown, and many observations did not care about photometric conditions. The master response curves have been derived from carefully selected individual standard star observations, but they are applied to science data sometimes taken a long time before or after, in many cases years apart. There has been no attempt to correct for these long-term effects.
The flux-calibrated spectra are provided in units of 10
Bug in the flux scale. In the UV_SFLX products with settings RED760 or RED860, the flux scale in the data product is wrong by a certain factor. This mismatch exists for data processed until 2017-12-31, but is fixed for later processing dates.
These are the affected columns in the UV_SFLX products with RED760 or RED860:
The data with correct flux scale are marked by the FITS key 'FLUXCORR=TRUE'. This key does not exist in uncorrected data. Unfluxed data products are not affected. Fluxed products from set-tings with shorter central wavelength are not affected.
There are two different factors per setting, depending on the wavelength range. These are the factors:
The two different wavelength ranges, and their correction factors, originate from the two different CCDs and their pipeline products. The SNR ratio is not affected, neither are the other columns. The reason for the bug is that the master response curves as used for the flux calibration are based on standard star products originally processed with a different flat-fielding method (reduce.ffmethod=extract) than the science data (reduce.ffmethod=pixel). Both methods differ by the quoted scaling factors. Its application to the data products (fits files) was overlooked.
Wavelength range of flux-calibrated spectra: some of the master response curves used for flux calibration have the initial and the final 25 Ångström cut off, hence the flux-calibrated spectra have a slightly smaller wavelength range than the extracted ones.
Spectral gaps. For spectra of the RED arm, increasingly larger gaps occur beyond about 880 nm, related to the ‘pixel’ flat fielding algorithm. These gaps would be much smaller with the ‘extract’ algorithm which however has strong artifacts from the fringes in the flat field. It was felt that the spectral gaps are of minor impact in general. An investigator interested in the spectral information in those gaps might want to manually reduce the data with the ‘extract’ method.
Ozone absorption bands. In the extreme blue part of the 346 nm spectra (shorter than 320 nm) there are the ozone (O3) absorption bands causing some ondulations (here).
Light leak. Some spectra taken with the old RED upper CCD (upgraded in 2009) and with a long exposure time (3000s and more) are contaminated by a light leak which is visible in the raw frames (see figure below). Under unfavourable conditions this artefact propagates to spectral artefacts. In case of doubts we recommend to inspect the raw files where the light leak is easy to spot.
Flat-field signature. Quite rarely, the 860 REDU spectra have flat-field related artifacts at the begin of echelle orders, right after the order gaps (here).
TUCD1 key filled wrongly. The UVES wavelength scale refers to air wavelength. The TUCD1 key was filled incorrectly with "em.wl" which stands for vacuum wavelengths, for all data downloaded before about 2017-08. The content of TUCD1 has now been corrected to "em.wl;obs.atmos", and all data downloaded in 2017-08 and later are fine.
Flat-fielding and wavelength dispersion before 2001-10-01: in the early history of UVES data processing, the proper association between calibrations and science was done with all but one relevant instrument parameter: slit width. This means that master flats (UV_MFLT), and the dispersion tables (UV_PLI1,2,3), were correctly created and archived but only for one value of slit width per setting, although maybe raw science and calibration data with several such values were existing. In other words: raw calibration data were complete from the beginning of UVES operations, but master calibrations in the archive are incomplete, for certain dates before 2001-10-01. With the operational setup of the UVES reprocessing project, master calibrations are not recreated but just downloaded from the archive. Therefore, the correct match of slit width and calibrations cannot be guaranteed for science data before that date. Usually, this has little impact on the data quality, but in extreme cases (e.g. science data taken with the 0.3 arcsec slit, master_flat with 1.5 arcsec slit) this may affect the extraction quality and may explain artifacts, especially with a period of one echelle order. Check here for a particularly impressive example.
Possible artefacts in the initial part of the RED spectra. This issue has been fixed in the version 2 data products covering the date range 2000 until 2007-03-31. It does not exist in the data acquired later.
In the version 1 data, for the first 10-20 nm of many RED settings (corresponding to the initial 2-4 echelle orders), there is the danger of spectral artefacts. These are due to the wrong choice of the processing parameter kappa. We have found these features in particular in the 564, 580 and 860 nm settings. The purpose of this parameter is to recognize and suppress artefacts like cosmics or bad pixels. With the choice of 20 instead of 10, it would suppress outliers only beyond 20 sigma. This resulted in the extraction algorithm mistaking the bad columns of the REDL CCD with the echelle signal, and also in accepting all cosmics. Now, with the version 2 data products, these artefacts have disappeared and also the long red exposures have a much cleaner signal, with proper suppression of cosmics.
Object centering. If the object is not well-centred in the slit, spectral extraction may fail or produce all kinds of artifacts, including extraction ripples (here). The position of the object is stored in each product, in the keys HIERARCH.ESO.QC.ORDnn.OBJ.POS (in pix) where nn is the echelle order number. Its mean value is stored as parameter qc_pos in the QC1 database table uves_science_public.
Emission-line objects. Without a traceable continuum, the pipeline may produce artifacts. In many cases such products can be recognized automatically by an exceptionally small (< 0.5") or large (> 3") FWHM. Still the spectra may be useful since the emission lines may be extracted correctly. Find an example here.
Absorption-cell data. Data taken with the iodine cell (almost always taken with the 600nm setting) have its absorption signature still in the extracted spectrum, here.
Unfortunately not all data with the absorption cell have been tagged properly. Mostly in visitor-mode nights, it might have happened that such data have the DPR.CATG filled with ECHELLE while they should have ECHELLE,ABSORPTION-CELL. The key INS OPTI1 NAME is likely filled with IN in those cases while otherwise it is OUT.
Resolving power. The spectral resolving power as displayed on the archive interface is a nominal value based on the slit width. It has been derived from the QC monitoring of health check arclamp exposures. The current and historical measurements are available here (use also the links 'FULL' and 'history') and on the interface to the QC1 database. The arclamp-based values are a lower limit to the resolving power actually applicable to a given science spectrum. That number, while not being directly measurable (unless telluric lines are available for a FWHM measurement), depends on the slit width and the prevailing seeing. For short exposures under seeing conditions better than the slit width the resolving power could be actually better than the nominal value.
There exist five different file types, depending on the reduction level (flux calibrated or not) and on the product category:
The following naming convention applies to the ORIGFILE product name:
e.g. UV_SFLX_58146_2001-09-13T09:12:30.326_RED564d1_2x2_21.fits has the components:
Due to the data format of UVES raw data, the UVES products with d1 or d2 in their name always come as a BLU/RED twin taken (almost) simultaneously (although not necessarily with identical exposure times).
The UVES products come as binary fits table with multi-column format (http://www.eso.org/sci/observing/phase3.html). The columns are labeled as follows:
a) non-SLICER data:
b) SLICER data:
In the case of the red spectra, the header contains keywords from both pipeline products (remember the pipeline delivers one spectrum per detector, and these are merged into the single binary fits table), distinguished by a field 'CHIP1' or 'CHIP2', resp., in the keyword name.
The UVES products have about 3 MB size for the unbinned (1x1) RED arm, and 2 MB for the unbinned BLUE arm. Binned products (2x2) come at half this size. Files are always uncompressed.
According to the ESO data access policy, all users of ESO data are required to acknowledge the source of the data with an appropriate citation in their publications. Find the appropriate text here.
All users are kindly reminded to notify Mrs. Grothkopf (firstname.lastname@example.org) upon acceptance or publication of a paper based on ESO data, including bibliographic references (title, authors, journal, volume, year, page-numbers) and the programme ID(s) of the data used in the paper.
The following run IDs got either no data products at all, or are covered incompletely. The processing was not possible because no master calibrations exist for these data. The following operational reasons might apply: