Quick links: overview | release content | data organization | data selection | release notes | data reduction | pipeline description | master calibrations | data quality | known issues&features | reprocessed data | file types | file structure | file size | acknowledgement text
ESO Phase 3 Data Release Description
XSHOOTER Echelle science products
Abstract. This is the release of reduced 1D spectra from the XSHOOTER spectrograph, ECHELLE,SLIT mode (as opposed to the ECHELLE,IFU mode which is not included). All spectra have been reduced under the assumption of point-like sources. This release is an open stream release, it includes so far archived XSHOOTER data and will be continued into the future. The processing scheme is as homogeneous as possible.
The selected data cover the vast majority of the entire XSHOOTER data archive, from the begin of operations in October 2009 until present. The data have been reduced with the XSHOOTER pipeline, version xshoo-2.3.12 and higher. All data have their instrument signature removed: they have been de-biased, flat-fielded, wavelength-calibrated, order-merged, extracted, sky-subtracted and finally flux-calibrated. Telluric absorption has not been corrected for. The pipeline output products come in the ESO 1D standard binary table, along with some ancillary files.
The processing is performed by the Quality Control Group in an automated process. The pipeline processing uses the archived, closest-in-time, quality-controlled, and certified master calibrations. It is important to note that the reduction process itself is automatic, while the quality assessment and certification of the master calibrations was (and still is) human-supervised. The overall data content grows with time as new data are being acquired and processed (approximately with monthly cadence and with a delay of 1 or 2 months).
The data format follows the ESO 1D spectroscopic standard for phase-3 data products. Each spectrum is a multi-column binary table. There is one product file for each set of input raw files. The set of input raw files is template-based for the NODDING and OFFSET data, and file-based for STARE data. The XSHOOTER spectral products always come per arm: UVB, VIS, or NIR.
This data release offers science-grade data products, with the instrumental signature and sky background removed, flux-calibrated, with error estimates and quality flags, and with a list of known shortcomings. They are considered to be ready for scientific analysis. They are expected to be useful for any kind of medium-to-high resolution spectroscopic research, including abundance and line profile studies, and radial velocity studies. For studies of energy distributions the researcher should keep in mind that the XSHOOTER instrument is a medium-to-high resolution spectrograph, designed for stability and throughput. It is usually not operated to achieve high-precision spectrophotometry, unless the wide slit is used. Slit losses have to be expected, and they may differ from arm to arm. There has been no attempt to correct for telluric absorption lines; this might affect certain science applications.
The telluric standard stars, measured frequently during most nights in order to provide a representative reference for telluric absorption lines, have been included in the data release as stand-alone science-grade spectra. They can serve two purposes: providing high-quality reference spectra for telluric correction, and providing science-grade spectra of early-type and solar-type stars, often repeated at various epochs.
Data have been pipeline-processed with the best available calibration data. However, please note that the adopted reduction strategy may not be optimal for the original scientific purpose of the observations, nor for the scientific goal of the archive user.
The XSHOOTER release is a stream release. The overall data content is not fixed but grows with time as new data are being acquired and processed. The data are tagged as "XSHOOTER" in the ESO archive user interface.
The first XSHOOTER data were published in May 2014. They have been filled since then into the archive in a chronological manner. New data are being added at roughly monthly intervals and with a delay of 1 or 2 months, when all master calibrations for the corresponding time interval are available.
Recently we have added the spectra of telluric standard stars. They have been processed in exactly the same way as the data taken as science data. For future dates, their spectra will be added in monthly intervals as well.
The downloaded data come with their technical archive names:
The first useful step to do is the organization of the data on the local disk. The README file coming with every download contains the information necessary to properly rename the files and establish their association. The fundamental unit of the renaming process is to capture column #2 in the README file (technical ADP name) and column #3 (original name, starting either with XS_ for the fits and txt files, or with r.XSHOO/r.SHOO for the graphics) and use something like 'mv $2 $3'. The file type is in column #4, find the possible values listed here. The spectrum is of type SCIENCE.SPECTRUM while the other files are of type ANCILLARY.<subtype>. Once you have renamed the files, they carry the names as also listed in the ASSON1, ASSON2 etc. header keys of the first extension of the spectrum.
Then, the easiest way to identify all files belonging together is the timestamp in their name, e.g. 2009-10-05T07:45:26.840, which is common to all file names belonging together.
Data selection is rule-based. It is organized along the following criteria:
No selection has been made made on the basis of the observing mode (visitor or service), nor on settings. XSHOOTER ECHELLE settings are defined by the combination of arm (UVB, VIS, or NIR), binning (1x1, 1x2, or 2x2), and slit width.
The MAPPING data have not been included because they are 2D scans of extended objects and require user decisions for the background subtraction and the extraction strategies.
We have processed only those data for which certified master calibrations exist in the archive. Depending on their type, these master calibrations exist at daily frequency or less. They were all processed by the Quality Control Group but earlier in time, close to the date of acquisition, in order to provide quality feedback to the Observatory. All master calibrations used here were certified, meaning checked for quality and proper registration of instrument effects. For XSHOOTER, the selection pattern for master calibrations was complete from the begin of operations on, covering both Service Mode (SM) and Visitor Mode (VM) science data and all settings.
Note that the fact that a certain set of XSHOOTER raw files does not have a product in this release does not necessarily imply a quality issue with the raw data. There are sometimes acquisition patterns for which the processing jobs can be predicted to fail (e.g. NODDING with one raw file; see “Rejected or failed processing”). In some of those cases the raw data would probably process fine with a fine-tuned strategy.
In general, the master calibrations were processed with different (earlier) pipeline versions than the science data in this stream. The most significant change in the XSHOOTER pipeline was related to version xshoo-2.0 and higher, reflecting the results of a major science-grade review in 2012 and 2013. This change was implemented in 2013-01. All science data acquired later than that date have been processed with essentially the same pipeline version as their master calibrations. All science data acquired earlier have still been processed with the current pipeline version but their master calibrations were processed with earlier versions. See “Master calibrations” for a discussion of the main differences.
Science data with the PROG.ID starting with 60. or 060. have been de-selected, considering them as test data. On the other hand, telluric standard stars have (almost) always the PROG.ID 60.A-9022(C) and have always been included in the processing scheme. Users interested in searching only for XSHOOTER telluric standards should use this PROG.ID on the interface.
Data taken at daytime (with obviously wrong ‘SCIENCE’ tag) have been ignored. Otherwise the header tag ‘SCIENCE’ has been blindly accepted from the raw data (originally defined by the PI), thus including sometimes standard stars intended by the PI for use as flux calibrator (or as telluric standard star). This is often evident from the OBJECT header key. Also, there are rare cases when test observations were executed under the SCIENCE label. Some very short exposures with no signal fall into that category but have not been suppressed.
All data have been processed up to the flux calibration step. The flux calibration was done with master response curves.
There is no data rejection based on quality. Likewise, we have not considered OB grades (as given by the observatory staff to assess the match with user-defined constraints, for SM data) for selection: the observations might have any grade between A and D (if taken in SM), or X (in VM). The availability, or non-availability, of a particular file in this release does not infer any claim about the data quality.
For the observing modes NODDING and OFFSET, we have combined all raw files taken in one template and from one arm in a single product file. Note that we have combined only within a template; multiple observations of the same OB, or observations across OBs, have not been co-added. In the automatic quality checks of the products, we have flagged those rare cases when a template was aborted and an additional quality issue occurred (see section ‘Data Quality’).
For observing mode STARE, we have processed every single input raw file separately. While processing of STARE data could also be done with all files from a single template stacked together, it is not always reasonable to do so, since there might be cases when the template sequence was designed to follow a time variability pattern. Since it is impossible to automatically recognize this case, it was decided to process the STARE data always one by one. A manual co-addition or stacking of these spectra by the user is straightforward.
We have included the telluric standard stars in the current release. They are acquired by the ob-servatory staff during nighttime, often in slots about every 2 hours. They match the science data of the previous 2 hours in setup (read mode, slit width, DIT for NIR) and airmass (within about +/- 0.2). If observatory-provided, they are always taken in NODDING mode (until 2013-04 always in STARE mode). The strategic goal is to provide high-SNR spectra with about the same telluric absorption signature as the science data. The rules about time and airmass match are motivated by proximity of the PWV (precipitable water vapour) value causing a certain telluric absorption line depth. The setup match rule is followed in order to match the line width.
While the flux-calibrated reduced spectra of these standard stars might be useful as a first step towards a telluric correction of the science data, we also think they might be useful as stand-alone, high-SNR spectra of early-type, or solar-type, stars. Most of these targets are taken from the Hipparcos Catalogue. Because of their special acquisition pattern, there are some quality remarks, see the section 'Data quality'.
Depending on the observing mode, the data reduction uses the standard XSHOOTER pipeline recipes xsh_scired_slit_<mode>, with mode being one of stare, nod or offset. Find a description of these science recipes in the Pipeline User Manual (follow the XSHOOTER link; there you will also find the link to the Reflex tutorial, the current version is under ‘XSHOOTER Reflex Tutorial’).
Find the pipeline version used for processing in the header of the product file, under “HIERARCH ESO PRO REC1 PIPE ID”, or in the QC database, key pipe_id. The version for the initial dataset (the historical batch until 2013-12-31) was xshoo/2.3.12dev1 (despite its name, this was a stable version).
All recipe parameters, including the extraction methods, were set by the pipeline defaults. The only exceptions are purely technical (generate-SDP-format=TRUE, to generate the output format of the 1D spectra; and dummy-association-keys=4 for the technical ASSOC/ASSON keys). Find more details in the User Manual, sections 10.15-10.17 (for version 12.1).
The main reduction steps of the XSHOOTER Echelle science and telluric standard star data are the following:
STARE observations. Each STARE frame is reduced individually; multiple STARE observations from the same template are not stacked. Reduction steps are:
The localization was done with the method MANUAL (localize-slit-position=0.0, localize-slit-hheight=2.0). It is the most robust method but not always the correct one (if the object is off-centre). The position of the peak signal can easily be checked on the preview plot #2 (for VIS arm data).
NODDING observations. They get combined within the same template, per arm; the pipeline assumes constant transparency and equal exposure times for all input frames. Reduction steps are:
Note that in case of odd number of input files, the pipeline rejects the last one and processes all others. Nodding with one input frame always fails.
The pipeline versions earlier than xshoo/2.7.0 had a bug with incorrect handling of the nod throw. This is the displacement between the two nodding positions. It is 5 arcsec by default but users can modify that value in the phase-2 OB design. The pipeline always assumed 5.0 arcsec, independent of the actual value.
The effect was that for nod throw < 5 arcsec, the negative parts of the nodded signal were (partly) included in the extraction mask, and the extracted fluxes - and the corresponding SNR - were compromised. Data with nod throw smaller than 3.5 arcsec are strongly affected (amounting to a few percent of all data), the other data only marginally if at all (most data have the default 5.0 arcsec).
All nodding data with nod throw <= 3.5 arcsec have been reprocessed and are available as corrected versions. Data with a higher nod throw have not been reprocessed since the impact is marginal. Data with a nod throw larger than 5 arcsec are not affected. Data acquired after 2016-11-01 (any nod throw) have been properly processed right away. Data in other observing modes are not affected.
OFFSET observations. SKY and OBJECT frames get combined within the same template, per arm; the pipeline assumes constant transparency and sky brightness, and equal exposure times for all input frames. Reduction steps are:
The wavelength scale is in air.
Information about the XSHOOTER pipeline (including downloads and manual) can be found under the URL http://www.eso.org/sci/software/pipelines/. The QC pages contain information about the XSHOOTER data, their reduction and the pipeline recipes.
Please check Sect. 7 of the User Manual for the description of calibration data.Table 1. Set of master calibrations used for data reduction
* arm = UVB|VIS|NIR
For almost every night used for XSHOOTER science, there exists a flux standard measurement. On the other hand XSHOOTER is neither designed for, nor operated under strictly photometric conditions, and for that reason the flux calibration cannot aim at photometric accuracy, but is geared towards removing the remaining instrumental signature (mainly from the spectral energy distribution of the flat lamp). For this purpose master response curves are sufficient, which are carefully compiled from individual (nightly) response curves, giving an accuracy of the flux scale in the science products of about +/- 10% differentially (meaning the instrumental effects are removed with that accuracy). Still the photometry could be off by larger amounts, due to uncontrolled transparency variations and slit losses. Because of the increasing instability of the UVB flat-field lamp, the concept of master response curves was given up and replaced by nightly response curves for all data taken after 2015-03-31. See section 'Flux calibration' for more details.
Wavelength scale. The XSHOOTER Echelle products have a topocentric wavelength scale; no corrections for barycentric or heliocentric motion have been applied. The corresponding values have been calculated by the pipeline and are stored in the header (HIERARCH.ESO.QC.VRAD.BARYCOR and …HELICOR, in km/s). The wavelength scale refers to air.
Telluric absorption lines. No correction for telluric absorption lines has been applied. For almost all spectra, one or more telluric standard stars have been measured, within 2 hours of the observation and within 0.2 in airmass. The telluric correction is not provided by the pipeline because it would require assumptions and judgements on the science spectrum.
For most of the telluric standard stars a fully reduced spectrum is also available in this release. The best way to identify them on the archive page is by PROG_ID=60.A-9022(C), and in the data header by the key IS_TELL=T. The user may want to select the best suitable one and apply standard techniques to process it into a template for telluric absorption line correction with the same spectral resolution and about the same PWV value as the science data.
You could also download the raw data from the ESO archive through the 'calSelector' service, along with the master calibration files, process them and apply the product to the science spectra. Alternatively, you may want to visit the ESO skytools web page for appropriate tools.
Be aware that a telluric standard star spectrum and a science spectrum, even though formally matching in slit width and complying with the operational match rules (within 2 hours in time, and 0.2 in airmass), not necessarily match in FWHM of the telluric lines. If the exposure-time integrated seeing for a science or a standard star spectrum was better than the slit width, the effective resolution will depend on the seeing rather than on the slit width.
Bad pixel map. We used static bad pixel maps for the processing, selected per arm and binning. (The pipeline also supports dynamic bad pixel maps derived from flats, their advantage being a closer time match, at the price of far less controlled quality.) These are from 2013 and have been carefully reviewed for the pipeline versions xshoo-2.x.
Normal daytime calibrations and AFC frames (automatic flexure compensation). During the lifetime of XSHOOTER three different association schemes were used for ORDER_EDGE, DISP_TAB, and XSH_MOD_CFG_OPT master calibrations:
A careful investigation of the shifts in X and Y direction to be compensated has shown that an effective difference in quality was not noticeable. Hence we have decided to keep the historical association scheme and process the data accordingly.
Master calibration names and recipe parameters used for reduction. The product header contains a list of all used master calibrations, look for keys “HIERARCH ESO PRO REC1 CAL<n> NAME” and “... CATG”, with the index n. The pipeline parameters and their values are listed as “HIERARCH ESO PRO REC1 PARAM<n> NAME” and “... VALUE”.
Products. The XSHOOTER pipeline creates a large number of intermediate and final product files. The final product in the spectroscopic data format combines information from the following products:
These products are combined into the final binary table fits file which is the delivered main data product, with the wavelength scale as first column and all other products as further columns (see ‘File Structure’ below).
There is also an ancillary fits file:
There are further ancillary files for each product:
All XSHOOTER spectra come separately by arm, like the raw data. Typically there are three spectra (from the UVB, VIS and NIR arm), but there are exceptions:
The three products have essentially no spectral gap. The UVB product has the nominal signal truncated beyond 556nm, to suppress a spurious pseudo-absorption at 570nm arising from artifacts in the flat field (the suppression is controlled by the parameter cut-uvb-spectrum=TRUE).
Rejected or failed processing. Under the following conditions the pipeline processing always fails, and no data products exist (but such data could possibly be processed interactively, e.g. with Reflex):
The possible reasons for the existence of such data could be:
Furthermore there are occasionally processing failures. A typical case is a high number of nodding input frames (20 or more) for which the pipeline sometimes, after long iterations, did not converge. No major attempts were made to understand and fix the situation, unless it was obvious.
Note that only in very rare cases we have rejected a product that was successfully created by the pipeline. Even in case of heavy saturation we have decided to deliver the spectrum to the archive, since the saturated pixels are marked in the QUAL column, and the unsaturated regions might still be useful. Also, products with large parts of negative flux have not been rejected. Instead the QCFLAGs have been set accordingly (see Section ‘Data Quality’, Table 3).
Only in the following cases we have rejected a product:
One of the main improvements of the XSHOOTER pipeline over time was the flux calibration. Initially (before pipeline version xshoo/2.x) the response curves, as derived from almost nightly flux standard stars, had a poor algorithmic quality (e.g. uncontrolled splines) and were almost useless. With the review and improvements of version xshoo-2.x, these issues have been solved, and the response curves are of good quality and stability.
For the years 2009 until 2015, the response curves showed very good stability, and we decided to use master response curves. The main reason is that we are aiming at a flux calibration, not at a photometric calibration. XSHOOTER data have been taken under conditions which were not photometrically controlled, and depending on the slit width used, they suffer from slit losses. Without the goal of a photometric calibration, the use of daily, close-in-time response curves would only be reasonable if instrument components were observed to vary on short timescales. This was not the case until about 2015-03.
After that date, however, presumably due to ageing flat lamp, the stability of the UVB arm response curves degraded. In particular the region at 370 nm, close to the Balmer jump, is dominated by the transition between the two flat-field lamps and is particular sensitive to variability. It turned out that the stability criterion of 10% was often violated after that date.
We have therefore decided to switch the flux calibration to the nightly response curves for all data taken on and after 2015-04-01. This approach provides optimal protection against short-term variability of the UVB lamps.
Although this instability affected only the UVB arm, we decided for consistency to switch the flux calibration to the nightly scheme for all three arms. The data from 2015-04-01 until 2016-11-01 that were already processed, archived and available for download, have been reprocessed. Data from 2016-11-02 on are processed with the same schema. There was no reason to reprocess the pre-2015-04-01 data, they are unaffected by this issue.
Master response curves. The procedure for the master response curves follows the scheme developed for UVES. These are based on carefully compiled individual response curves. The selection criteria were:
In order to make use of the xshoo-2.x improvements, we started the compilation with data from 2013-01 and later. The master response curve for the NIR arm, averaged from all compiled response curves, and its +/- 10% envelope, are displayed in Figure 1.
We then took this master response curve and compared it to individual response curves from the earlier epochs. Again the good (in the above sense) response curves were selected interactively and averaged. Their number amounts to several hundred individual curves, per arm. That average, together with the +/-10% envelope, is plotted for the UVB arm in Figure 2.
Then the existing master response curve was multiplied by a fudge factor (to account for differences in the normalization procedure for the master flats at that time), it is displayed as the blue line. It is obvious that adopting that master response curve (originally defined for the 2013 time range) also for the earlier epochs is possible within the +/-10% range. It has small systematic variations, possibly due to instrumental drifts (change of coating properties or flat-field lamp energy distribution) and possibly also to different pipeline versions, but it is well-defined in terms of spline robustness and also step size (in particular in the VIS and NIR telluric windows).
We have tested the validity of our approach with XSHOOTER observations of the flux standard star Feige 110. These data have been observed in NODDING mode, with the widest slit size 5”, and have been processed with the corresponding science recipe, including the final step of the flux calibration with the master response curves for the three arms. A comparison of the flux calibrated XSHOOTER spectrum (which should in this case have photometric quality) to the tabulated energy distribution is reassuring (Figure 3). We did also a comparison to a flux-calibrated UVES spectrum of the same star (Figure 4). Again, the agreement is very good.
Versions of master response curves. We have identified several periods of validity for the master response curves, separated by breakpoints. One breakpoint occurred on 2013-01-15 and is related to the onset of using the pipeline versions xsh-2.x from that date on for the calibrations, in particular for the creation of the master flats. Those versions use a different normalization procedure than the versions xsh-1.x used before. Because both the science data and the standard stars (used for computing the response curves) are flat-fielded, the master response curve and the flat-fields used need to be aligned in terms of pipeline algorithms.
The second breakpoint applies to the NIR arm only and reflects the exchange of the NIR flat lamp, along with its significantly changed chromatic energy distribution.
Table 3. Versions of master response curves
Slit losses. In some cases PIs have taken their spectra with both the wide (5”) slit (for photometry) and a narrow slit (for resolution). In these cases it is possible to measure the slit losses, for the prevailing seeing and transparency conditions.
Because of their photometric quality (at least with no slit losses), we have marked the wide-slit data with the quality bit 0 for good photometry, while in all other cases they are flagged as 1 for unknown photometric quality (depending on seeing, slit losses could still be negligible).
The flux-calibrated spectra are provided in units of erg/cm^2/s/Å.
Master calibrations. All used master calibrations have been quality-reviewed and certified at the time of acquisition, as part of the closed QC loop with the Observatory which also includes trending. Check out for more under the XSHOOTER link of http://www.eso.org/qc/ALL/daily_qc1.html. The most important parameters for the quality of the products are the SNR of the master flats, and the rms of the dispersion solution. The SNR of the master flats was always high enough to be dominated by the fixed-pattern (gain) noise, which is important to not compromise the SNR of the science data. The rms of the wavelength dispersion was higher in the first few years (with pipeline versions xsh-1.x) than later, due to a lower number of lines found (Figure 5).
SNR. There is a column “SNR” in the product table that is calculated from the signal and the corresponding error. It has no independent information but is provided for convenience. Its mean value is written as header key SNR. There are also the header keys HIERARCH.ESO.QC.FLUXn.SN (n=1…2 or 3) that describe the SNR in various spectral windows, defined in the comment of the key.
SPEC_RES. The header key SPEC_RES contains the nominal resolving power, as derived from the arclamp calibrations of the same slit width. The actual resolution of the science data has not been measured but could actually be higher, depending on seeing conditions and exposure time. For a point-like source, the actual resolution is between the value corresponding to the seeing disk, exposure-time integrated, and the value corresponding to the slit width. A better estimate would be available from the half-width of the telluric absorption lines.
Telluric standard stars. Although processed in the same way as the science spectra, spectra of the telluric standard stars have some characteristic quality features. The goal of the nighttime staff is to achieve a high-SNR spectrum while trying to avoid saturation. With the actual flux per pixel depending on the currently prevailing extinction and also on seeing, it is often unavoidable to have a first attempt ending up in saturated, or underexposed, spectra. Then, follow-up exposures are taken to optimize the flux per pixel. Optimization is usually done per arm, with the exception of the UVB arm which is not relevant for telluric line correction and is not optimized. The outcome of this procedure is often a sequence of spectra of the same target with exposure times getting shorter, or longer, or – if already optimized for this arm but not yet for the other arm – repeated identical exposure times. We offer all of these spectra and leave the final judgement and selection to the user who should keep in mind that even heavily saturated (certain regions only, not everywhere) telluric spectra exist in the archive. On the other hand many of the optimally exposed telluric standard star data have a very high SNR and offer the full scientific potential of exquisite spectra of early-type, or solar-type, stars.
QC flag. The header key “QCFLAG” in the XSHOOTER products describes automatically assigned quality flags. It is composed of seven binary bits. For each of it, the value 0 means “no concern”:Table 4. QC flags
The final QC flag is composed of all binary values. For instance, 0000000 is a pristine product, 1011100 is a product that might have some slit losses, and also has negative flux indicating a localization or extraction issue. The seeing check #1 is a rather coarse quality check and meant to be indicative only (the two header keys TEL.FWHM.START and ...END refer to the first raw file). The same is true for the airmass check (#2); the underlying values are also from the first raw file. Taken together, they indicate mainly how reliable the flux scale is. With #1 and/or #2 being 1, it becomes likely that the flux scale for spectra from the different arms may show jumps. Only if check #3 returns 0 the flux calibration is likely to be free from slit losses, but still not necessarily be of photometric quality since the transparency is not strictly controlled for XSHOOTER observations.
Flags #4 and #5 assess the extraction quality. They are not entirely independent but not redundant either. Their underlying values are averaged across the whole spectral range. Flag #6 indicates possible saturation issues, while #7 is a technical indicator of template abortion, which might alert the user to unusual observing conditions that might explain abnormal product properties. Template abortion almost always indicates degradation of observational conditions, or instrumental problems, and results often in data sequences which are incomplete at least for good automatic processing, or violate quality assumptions of the pipeline. In case there were no quality issues discovered other than flag #7 being set, we have delivered the product to the archive. If flag #7 was found to be set, and another quality issue out of #4, #5 or #6 was found, we have decided to not deliver the product since these products were always found to be useless.
Science products. There has been some internal quality control on the pipeline processing of the science data, monitoring:
This information has largely been used to improve and fine-tune the reduction process. An individual one-by-one inspection of the products has not been done.
Quality flags. The XSHOOTER pipeline supports pixel-based quality flags (not to be confused with the QC flag that applies to the whole file), propagated through the entire calibration and reduction procedure. They come as co-added bit codes and are available per wavelength bin in the spectrum table as column “QUAL” (see figure below). The possible values for the bit codes are defined in the User Manual (sect. 11.3). Some important values are listed here:List of important quality flags
The logarithm of the quality flags is plotted in blue as lowest panel in the preview plots, labeled “log Q”. The bit 2^22 has been suppressed in the plots (but not in the data) in order to avoid the plots being swamped.
Previews. Originally developed as quick-look plots for process quality control, we felt they might be also useful for the archive user as preview plots of the spectra. They are delivered as ancillary files along with the main spectra. There are two plots:
1. the main preview plot (Figure 7), one per arm;
Process quality control. The quality of the data reduction is monitored with quality control (QC) parameters, which are stored in a database. The database is publicly accessable through a browser and a plotter interface.
The QC parameters are used to monitor the reduction quality. The most obvious check is the SNR versus signal control plot (the signal being expressed as the mean across the entire reduced spectrum, before flux calibration, and the SNR being the mean over the entire spectral range). An example for NODDING, VIS arm is displayed in Figure 9. The blue crosses represent the low gain mode which is often used to obtain very high SNR of bright targets by combining many input spectra. For that mode, the record SNR found so far is in excess of 1000, quite remarkable for a product that has been automatically pipeline-processed without any fine-tuning. The red filled circles represent the “workhorse” high-gain mode, which is able to deliver SNR of up to 300 or more. The spread in the achieved SNR for a given value of mean_reduced is likely to come from the spectral slope of the products. This is illustrated by the next figure which displays these parameters for three selected observing runs. Their SNR slopes are well confined, due to their targets having roughly the same spectral slopes.
This list might evolve with time. Please check also Sect. 6 of the Pipeline User Manual and Sections 7.5 and 7.6 of the Reflex Tutorial (http://www.eso.org/sci/software/pipelines/, go to ‘XSHOOTER Reflex Tutorial’). There is also the FAQ page about data reduction (go to the XSHOOTER section of http://www.eso.org/sci/data-processing/faq.html).
Localization and extraction. Sometimes STARE data might have the object off-center, with the effect that sky and target get confused by the pipeline which uses pre-defined localization and extraction windows (see Figure 11). The spectrum then has negative values. It might still be useful after correcting for the wrong sign. The method used for localization of STARE targets is MANUAL (despite the name the method is automatic) which is the most robust one, but sometimes the wrong choice. Check the User Manual.
Incorrect STARE mode: before the MAPPING template was introduced (December 2015), the STARE mode was sometimes used for effectively the same arrangement of scans across extended sources. These data cannot easily be identified. Their products are very likely of poor quality because of poor background subtraction and wrong extraction strategy. The user should check suspicious STARE products for their cross-dispersion profile and for other products from the same OB: if they exist, and if they show different sources at varying positions, the products are mapping data and should be ignored.
For NODDING and OFFSET data the extraction method assumes a well-centered point source. If this assumption is true, the cross-dispersion profile is well-defined as a single positive signal and two correlated negative signals (Figure 12 a,b). Quite some spectra actually contain complex sources (multiple targets, complex background, extended sources, Figure 12c). This is one of the main sources for abnormal spectra (negative flux, negative SNR). The pipeline does not recognize these situations, and the user should always carefully check the 2D spectra in case of doubts.
On the other hand the pipeline can safely extract spectra without continuum (from an emission-line object), with a cross-dispersion profile in Figure 9d. Such profile is not unusual.
The extraction issue with NODDING data having a nod throw of less than 5 arcsec has been fixed (at least for the data being strongly affected, the ones with nod throw <= 3.5 arcsec; see 'Data reduction and calibration'). Users should replace their downloaded spectra if processed with pipeline versions earlier than xshoo/2.7.0. The value of the nod throw can be read from the key "HIERARCH ESO SEQ NOD THROW".
In Figure 13 and Figure 14 we illustrate the effect of bad centering of a point source. The example is a telluric standard star, but the argument applies to a science target in the same way.
OFFSET data depend critically on the observer’s definition of the Observing Block (Figure 15). The pipeline blindly trusts the labels OBJECT and SKY in the raw data. The user should always check carefully if the OFFSET products have a cross-dispersion profile like in Figure 15 a,b (well behaved) or e.g. in 15c (ill-defined). Remember that OFFSET data without OBJECT, or without SKY, are suppressed.
It is not possible to tell from the header if the source is point-like or not.Furthermore it is impossible to check whether the OBJECT and SKY labels are correctly set by the PI. In case of doubt, the user should always check the 2D ancillary files and the QC plot for the VIS arm displaying the cross-dispersion profile.
Saturation and gaps. Saturated (or more generally, bad) pixels in the raw data get ignored by the pipeline. If there are too many, so that no good pixels can be found by the pipeline for a given wavelength bin, the flux in this wavelength range is set to zero. In the QUAL extension of the 2D ancillary spectra, these regions can be easily spotted: dark pixels have bit code 512, pixels affected by cosmics have bit code 16 or 32; bit code 4096 indicates saturated pixels. (the saturated pixels have the bit-code 4096 for “saturated pixels”). Find the complete list of bit codes in the Pipeline User Manual. In the extracted main output file the QUAL column has the bit-code 2^19=524288 (“missing data”) (see Figure 16). Outside of these gaps, the data might still be useful.
Partial saturation. If spectral regions have some saturated pixels but are not entirely saturated, patterns might occur as illustrated in Figure 17 (blue circles). These ripples tend to display at regular intervals of about 30 nm in the UVB regions longer than about 400 nm.
Spectral slopes in the VIS arm. Another artefact typical of high-SNR (telluric) spectra is displayed in Figure 18 (red circle). These bumps are caused by uncontrolled variations in the energy distribution of the flat-field lamp. If existing they display in the first two spectral orders, and the continuum slope in that region is incorrect. Often they do not show up in the :FLUX_REDUCED spectrum. More in the Reflex tutorial sect 7.5, "Instrument Response".
Spikes in NIR spectra. Lots of artificial spikes exist in many NIR spectra. They are due to a pipeline issue, namely an insufficient handling of the bad pixels at the steps of re-sampling and frame-stacking. This pipeline issue will be fixed in the future.
Negative fluxes. In the NIR, negative fluxes might appear in raw data and are the result of saturation (Figure 19). Affected spectral pixels are flagged by bit 2^21 = 2097152, and the impact on the spectrum are troughs embedded between peaks (unsaturated regions). Also, it might happen in rare cases that the reset anomaly causes negative pixels which then are perfectly valid, except for the wrong sign.
Wrong OB grades in ancillary text files. If in Service Mode an OB is executed twice (or in general more than once) during the same night, this is typically done because during the first attempt the ambient conditions unexpectedly degraded, resulting in a C grade. A second attempt might then be made, ideally resulting in a B or A grade. In such a case, the grade (as given in the ancillary text file of the product) is always recorded from the last execution. This is wrong in the typical case of a grade C-A pair and due to a bug in the script. The grade as retrieved from the nightlog tool NLT is correct.For observations repeated in different nights this bug does not appear, and the quoted grades are correct.
APERTURE key filled wrongly. Due to a software bug, the APERTURE key in the products was filled with the slit length (always 11 arcsec) instead of the slit width, if processed with pipeline version xshoo/2.6.0 or earlier.
TUCD1 key filled wrongly. The XSHOOTER wavelength scale refers to air wavelength. The TUCD1 key was filled incorrectly with "em.wl" which stands for vacuum wavelengths, for all data downloaded before about 2017-08. The content of TUCD1 has now been corrected to "em.wl;obs.atmos", and all data downloaded in 2017-08 and later are fine.
Pick noise. At certain times (e.g. during most of 2010-03) the VIS arm detector showed a strong pick noise pattern that the pipeline cannot remove from the data. The impact is a high-frequency noise that is easily spotted in the spectral windows plot (2). See an example in Figure 20.
Steps at begin of order. Occasionally there are a few steps at the begin of an order, apparently induced by an imperfect flat-fielding (Figure 21).
Ozone absorption bands. In the extreme blue part of the UVB spectra (shorter than 320 nm) there are the ozone (O3) absorption bands visible in very high-SNR spectra (Figure 22).
2D extracted spectrum not flux-calibrated. The ancillary 2D file (name starts with XS_SRE2, product category is SCI_SLIT_MERGE2D_<arm>) is not flux-calibrated although such a product would be useful and could be created by the pipeline. There is no particular reason why that product was added instead of the flux-calibrated 2D product, except for that it was assumed that this product would be of no general interest and useful only for QC purposes (like checking where the signal is relative to the slit geometry). If a user wishes to have a flux-calibrated version of the 2D file, he should calculate the ratio of the columns :FLUX (the flux-calibrated 1D signal) and :FLUX_REDUCED (the uncalibrated 1D signal). That ratio function should then be expanded into a 2D file and get multiplied into the SCI_SLIT_MERGE2D file, resulting in a flux-calibrated 2D spectrum.
Intensity jump at 2.25 μ. In the reddest part of the NIR arm data, the sky background might have a gradient along the slit. Then the assumption by the pipeline of a flat sky background is no longer valid, which results in a sudden jump of the signal which is however an artefact and entirely due to this pipeline effect (Figure 23). Best is to ignore data beyond 2.25 μ if this step occurs. This feature does not occur at high S/N, nor for spectra acquired in NODDING mode. It might also occur as a downward jump.
Slit losses. With the requirement that telluric standard stars should match their "parent" science data in airmass (within 0.2), there are occasionally telluric standard star spectra taken at high airmass but without careful alignment taking into account atmospheric dispersion effects. A particularly strong example of the resulting slit losses is displayed in Figure 24.
Most XSHOOTER spectra have product version number 1. In some cases we have reprocessed the spectra which then have version number 2. This highest version is offered in the phase3 archive by default, but the previous versions are usually available on demand. (Find the pipeline version as fits key "HIERARCH ESO PRO REC1 PIPE ID".)
The primary XSHOOTER Echelle product is the flux-calibrated spectrum, in binary spectroscopic data format:
The primary product has an ancillary fits file delivered with it (useful for quality assessment; in 2D image format):
The following naming convention applies to the ORIGFILE product name (the ancillary fits file has the same name, with XS_SRE2 replacing XS_SFLX): e.g.
The same scheme is also used for the telluric standard star spectra. Their OB ID is almost always 200193515, i.e. purely technical. Their RUN_ID is 60.A-9022(C), also purely technical. Both values mark these data as being provided by the Observatory. The header key IS_TELL=T (for 'true') unambiguously distinguishes the telluric spectra from the spectra processed from files intended as science observations (having no such key).
The ancillary files (always delivered with the primary product) have the following ORIGFILE names:Naming conventions of ANCILLARY files
The user may want to read the ORIGFILE header key and rename the archive-delivered fits files.
The primary XSHOOTER product XS_SFLX comes as binary fits table with multi-column format (http://www.eso.org/sci/observing/phase3.html).Internal structure of the XSHOOTER spectra.
The difference between FLUX and FLUX_REDUCED is the flux calibration. The FLUX_REDUCED data might become useful if the user wishes to apply his own response curve instead of the master response curve. The SNR column is provided for convenience. The QUAL column contains the quality flag per wavelength bin, propagated throughout the reduction.
The (unbinned) XSHOOTER products have about 1.5 MB size. The ancillary 2D files have the same spectral coverage and come as 15-29 MB files (depending on arm). Files are always uncompressed.
According to the ESO data access policy, all users of ESO data are required to acknowledge the source of the data with an appropriate citation in their publications. Find the appropriate text here.All users are kindly reminded to notify Mrs. Grothkopf (email@example.com) upon acceptance or publication of a paper based on ESO data, including bibliographic references (title, authors, journal, volume, year, page-numbers) and the programme ID(s) of the data used in the paper.