Quality Control and Data Processing

FITS key QC1 database: table, name definition class* HC_plot** more docu

[calculated by QC procedure] xshooter_format..flux_ThAr_norm normalized average flux of selected ThAr lines [ADU/sec] (UVB/VIS) HC [docuSys coming]

[calculated by QC procedure] xshooter_format..flux_Ne_norm normalized average flux of selected Ne lines [ADU/sec] (NIR) HC [docuSys coming]

[calculated by QC procedure] xshooter_format..flux_Xe_norm normalized average flux of selected Xe lines [ADU/sec] (NIR) HC [docuSys coming]

[calculated by QC procedure] xshooter_format..flux_Ar_norm normalized average flux of selected Ar lines [ADU/sec] (NIR) HC [docuSys coming]

QC.NLININT.THAR xshooter_format..nlin_ThAr number of selected ThAr lines (UVB/VIS) HC [docuSys coming]

QC.NLININT.NE xshooter_format..nlin_Ne number of selected Ne lines (NIR) HC [docuSys coming]

QC.NLININT.XE xshooter_format..nlin_Xe number of selected Xe lines (NIR) HC [docuSys coming]

QC.NLININT.AR xshooter_format..nlin_Ar number of selected Ar lines (NIR) HC [docuSys coming]

*Class: KPI - instrument performance; HC - instrument health; CAL - calibration quality; ENG - engineering parameter
**There might be more than one.

Scoring&thresholds Arc lamp flux

The median spectral resolution derived from wavelength calibrated arc lamp SLIT spectra is monitored in the upper right box of these plots UVB, VIS and NIR.

The number of arc lines found in the through IFU arc lamp exposure is monitored in the upper right box of these plots UVB, VIS and NIR.

Both the number of found arc lines and the normalized flux are monitored. Relaxed thresholds are applied.

Date event

2011-08-01 we monitor the flux of carefully selected arc lines obtained from xsh_predict instead of the average flux for all lines found from xsh_wavecal. This allows us to monitor the different arc lamps in the NIR arm separately.

2011-02-20 Xe lamp in NIR arm replaced; Ne lamp in NIR arm aligned correctly

2011-02-15 Ne lamp in NIR arm replaced

2010-12-17 Ne lamp in NIR arm dies

2010-11-20 pipeline derives higher flux for NIR

2010-11-08..2010-11-20 pipeline problems with number of accepted lines in NIR, affecting derived line flux (xsh_wavecal)

2010-10-08 new pipeline version (1.1.6) with different algorithm to determine line intensity (follows line tilt); strong increase in measured NIR line intensity even though raw data did not change (xsh_wavecal)

2010-06-01 UVB lamp provides about 20% more flux; due to the inclusion of fainter lines the average intensity drops (xsh_wavecal)

2010-06-01 VIS lamp provides about 30% more flux; due to the inclusion of fainter lines the average intensity drop (xsh_wavecal)

2010-01-26 new NIR lamp provides brighter lines

Algorithm Arc lamp flux

The physical model is fit to the through slit (not pinhole) arc line Echelle spectrum. The flux is measured from identified lines.

Flux of the flat lamps

FITS key QC1 database: table, name definition class* HC_plot** more docu

[calculated by QC procedure] xshooter_flat..flux_1 maximum flux in raw flat, normalized by exposure time HC [docuSys coming]

QC.FLUX.MAX xshooter_flat..counts_1 maximum counts in raw flat (D2 lamp in UVB) HC [docuSys coming]

[calculated by QC procedure] xshooter_flat..flux_2 maximum flux in raw QTH flat ENG [docuSys coming]

[calculated by QC procedure] xshooter_flat..counts_2 maximum counts in raw QTH flat (UVB only) HC [docuSys coming]

*Class: KPI - instrument performance; HC - instrument health; CAL - calibration quality; ENG - engineering parameter
**There might be more than one.

Scoring&thresholds Flux of the flat lamps

For the UVB arm, we monitor the frame counts in ADU and the lamp flux in ADU/sec for both lamps QTH and D2 separately for IFU and one of the slits (1.0'') in hg, 1x1 read mode.
For the VIS arm the 0.9'' slit, and IFU with hg, 1x1 read mode is trended.
The NIR arm flat counts and lamp flux is monitored for 0.9'' slits and IFU.

The counts are scored to avoid non-linearity and saturation and low SNR.

Algorithm Flux of the flat lamps

From time to time flat field lamps age and fail. The highest replacement frequency occurs for the UVB-arm QTH lamp.
Between November 1, 2009 and August 20, 2010 the NIR flat field lamp provided less flux in the K band than before and after. The effect of this on processed data can be clearly seen in the plot below, comparing a telluric standard star observed and processed before and after that change.

Figure A, NIR-arm, impact of NIR-arm flat field lamp on telluric STDs.
The upper part of the last orders in the K band shows 10% vignetting due to a small design error of the M8 baffle inside the NIR cryostat. This cancels out with flat fielding, but results in higher noise for this region.

The counts are basic direct reads from the raw frames.

Spectrum of flat lamp

One aspect of the flat field lamp is the spectrum F(lambda) is produces. The spectrum is an ingredient of the response function, which itself is used to calibrate XSHOOTER science data. So far the flat spectrum of all four lamps are parameterized and monitored. Main events that contribute to the changes of the flat field lamp spectra are lamp aging and the physical exchange of lamps.

On 2015-10-24, the first flux standard star was acquired with the effmon template, meaning this observation was acquired under strict photometric conditions. The calibration product 'response function' and the flat field spectrum with setup slit = 5.0x11 and binning =1x1 (UVB and VIS only) from that date which are used to process the flux standard star from 2015-10-24 serve as a reference in the following.

Since the xshooter flat field recipe of the pipeline cannot generate a flat spectrum, 5.0x11 slit 1x1 binning flats are processed by the science recipe using the reference flat and the reference response function (from 2015-10-24) in order to obtain a flux calibrated spectrum of the flat field lamp. The science recipe is configured for not searching a source but integrating the lamp counts over the full slit length in this particular case. The comparison between the flux calibrated reference flat spectrum (from 2015-10-24) and the current flux calibrated spectrum is parameterized via

current_spectrum / reference_spectrum = a + b * ( lambda - lambda_0 )

where parameter a is the relative overall flux of the spectrum, corresponding to the lamp power and parameter b is a first order chromatic deviation (the slope between the reference flat spectrum and current flat spectrum). lambda_0 is the central wavelength of the instrument arm.

On 2015-10-24, there is a=1 and b=0 by definition. Parameter b is monitored and scored.

Notes:

UVB-arm: the D2 lamp spectrum [305 - 360] nm and the QTH lamp spectrum [ 370 - 500] nm is fit separately. The reddest order of the QTH lamp illuminated order are not used for the fit due no known variations of the dichroic. A variation in the slope of delta b = 1e-3 [1/nm] corresponds to flux increase of 3% at the red edge of the D2 illuminated UVB-arm spectrum and a flux decrease of 3% at the blue edge of the D2-lamp illuminated UVB-arm spectrum. For the QTH-lamp illuminated part (, wich covers 140nm instead of 60nm for the D2 part), a delta b = 1e-4 [1/nm] corresponds to 7% flux increase at the blue edge.
VIS-arm: The bluest orders are not used for the fit due no known variations of the dichroic: [ 560 - 1020] nm. A variation of delta b = 1e-4 [1/nm] corresponds to 2.5% flux change at the spectrum edge.
NIR-arm: the full range is used for the fit [1000 - 2460] nm.

FITS key QC1 database: table, name definition class* HC_plot** more docu

[calculated by QC procedure] xshooter_flat..flat_spec_a (a&b*lam) fit to current/reference spectrum: ENG [docuSys coming]

[calculated by QC procedure] xshooter_flat..flat_spec_a2 for UVB-arm QTH-lamp ENG [docuSys coming]

[calculated by QC procedure] xshooter_flat..flat_spec_b spectral gradient HC [docuSys coming]

[calculated by QC procedure] xshooter_flat..flat_spec_b2 for UVB-arm QTH-lamp HC [docuSys coming]

*Class: KPI - instrument performance; HC - instrument health; CAL - calibration quality; ENG - engineering parameter
**There might be more than one.

Scoring&thresholds Spectrum of flat lamp

The gradient of the flat field spectrum b is monitored. For the UVB-arm both spectral part, QTH lamp based and D2 lamp based are monitored separately.

The spectrum gradient is not scored since 2018-09

Algorithm Spectrum of flat lamp

The spectrum gradient was introduced in 2017 and 5'' flats have been reprocessed back to 2013.
2019-01: The flat field lamp exchanges, which might introduce new slopes on the flat spectrum, are not longer registered and no longer marked in the plot.

2018-09: the VIS-arm lamp flux decreases and the spectral gradient becomes redder.

Parameters a and b are the best fit parameters of the formula given above.

Slit and IFU function

In 2017-02 slit vignetting in IFU and SLIT flat fields was detected in the UVB-arm. The slit function is parameterized by the slit rms and the illumination gradient. For every master flat (any SLIT and the IFU; 1x1, 1x2 and 2x2 binning) the median of the 21 central rows is examined:

FITS key QC1 database: table, name definition class* HC_plot** more docu

[calculated by QC procedure] xshooter_flat..slit_rms RMS of slit function HC [docuSys coming]

[calculated by QC procedure] xshooter_flat..slit_b gradient of slit function HC [docuSys coming]

[calculated by QC procedure] xshooter_flat..slit_db gradient error HC [docuSys coming]

*Class: KPI - instrument performance; HC - instrument health; CAL - calibration quality; ENG - engineering parameter
**There might be more than one.

Scoring&thresholds Slit and IFU function

The rms of the slit function, which increases for slopes and dust grains on the slit, the gradient of the slit function and the error in the gradient derived from the linear fit are monitored.

The gradient and the rms are scored.

Algorithm Slit and IFU function

2017-02-01 first slit vignetting events detected
2017-02-15 QC1 report and trending put into operation
2018-08-23 Outliers in the slit function gradient and the rms over the slit function are usually correlated with events of low ambient temperature (see BIAS plot) and high humidity (see slice plot).

The cross section of a central order of the master flat is analysed.

ADC analysis via tech. IFU flats

Given the ongoing problems on the reliability of ADC positions and the difficulty to measure the ADC positions directly, ADC-related QC1 parameters have been introduced in 2012. From the daily acquired health check IFU flats the counts on all three slices are measured in a red order and in a blue order of the Echelle spectrum. The relative illumination between the three slices is monitored. So far a symptom is monitored and not the cause. The illumination ratios are dependent on the ADC positions. Also high humidity periods can have an impact on the measured QC1 parameters (see trending plots in 2015-03 to 2015-04).

The ADCs are repaired in 2017, the monitoring continues.

In a first step the median counts are measured in six dedicated windows:

S1R (slice 1 in a red order),
S2R (slice 2 in the same red order, central slice),
S3R (slice 3 in the same red order),
S1B (slice 1 in a blue order),
S2B (slice 2 in the same blue order, central slice),
S3B (slice 3 in the same blue order.

The QC parameter slice_rms is the rms of the three red order slice counts.
The QC parameter slice2_rms is the rms of the three blue order slice counts.

The QC parameter flux_order_ratio is a relative difference between the two flux averages:

fr = ( S1R + S2R + S3R ) / 3.0, (average counts in the three red order slices)

fb = ( S1B + S2B + S3B ) / 3.0, (average counts in the three blue order slices)

flux_order_ratio = (fr - fb) / fr (relative difference)

Since 2014-07-31

The QC parameter slice_s_rms is the relative difference between the two outer slices in the red order.

slice_s_rms = ( S1R - S3R ) / ( SR1 + SR3 ) * 2.0 ;

The QC parameter slice2_s_rms is the relative difference between the two outer slices in the blue order.

slice2_s_rms = ( S1B - S3B ) / ( S1B + S3B ) * 2.0 ;

When one outer slice becomes brighter than the other outer slice the sign of the QC parameter will change.

FITS key QC1 database: table, name definition class* HC_plot** more docu

[calculated by QC procedure] xshooter_flat..slice_rms flux rms between 3 slices in reference order ENG [docuSys coming]

[calculated by QC procedure] xshooter_flat..slice2_rms flux rms between 3 slices in another reference order ENG [docuSys coming]

[calculated by QC procedure] xshooter_flat..slice_s_rms flux rms between the blue and the red slice in reference order ENG [docuSys coming]

[calculated by QC procedure] xshooter_flat..slice2_s_rms flux rms between the blue and the red slice in another reference order ENG [docuSys coming]

[calculated by QC procedure] xshooter_flat..flux_order_ratio norm flux ratio between two reference orders (a-b)/a ENG [docuSys coming]

*Class: KPI - instrument performance; HC - instrument health; CAL - calibration quality; ENG - engineering parameter
**There might be more than one.

Scoring&thresholds ADC analysis via tech. IFU flats

See here

Thresholds have been set to operational meaning values.

Algorithm ADC analysis via tech. IFU flats

Date event

2010-07-18-2010-08-02 The Atmospheric Dispersion Compensator (ADC) of the UVB arm was stuck and did no longer adapt to changes in airmass or observing mode (SLIT vs. IFU). This resulted in strong vignetting of IFU data and in slit losses for SLIT data. Below you find Figure B comparing observations of the same telluric standard stars with (red) and without (black) that problem (airmass noted besides date). The spectra were scaled to overlaps between 350nm and 400nm.

2012-05 The UVB and VIS ADCs (atmospheric dispersion compensators) became more and more unreliable and are since 2012-05 at a fixed position. The impact of the fixed ADC position on science observations is described in the XSHOOTER User Manual. Several smaller drifts have been observed thereafter. The only way to monitor the stability of the ADC position is to monitor the flux ratio between the three slices of an IFU flat field. This monitoring is in operations to preserve the health of the instrument.

2012-06-01 new QC1 parameters, retrieved from QC script

2013-05-27 in the UVB-arm, blue orders are illuminated by the D2 lamp (< 350 nm) and red orders are illuminated by the QTH lamp (> 350 nm). Hence the UVB-arm flux_order_ratio QC parameter measures the flux ration between two different lamps and ADC effects together. Both effects are not deconvolved. The UVB-arm flux_order_ratio QC parameter is therefore not scored. The bi-modal distribution of the UVB-arm flux ratio is due to warm-up effects.

2014-08-01 two more QC1 parameters introduced

2014/2015 Extreme ambient conditions have shown, that the slice related QC parameters for UVB and VIS are dependent on the temperature of the optical elements and on the humidity within the optical path. While the humidity at the observatory drop over days, the humidity inside the istrument can take weeks to evaporate, as can bee monitored in the ADC QC parameters. As a consequence ADC QC parameters are blind for measuring ADC alignments in high humidity periods.

2018-04 Due to a shutter problem in the VIS-arm, flats showed the following behavior: one or more raw flats showed a significant higher illumination, while the remaining raw flats retained their counts. In some cases the higher illuminated flats were saturated. The problem is not directly visible in master flats, as the pipeline recipe rejects saturated exposures pixel by pixel when combining the master flat. A new QC parameter raw_med_std, which measures the standard deviation from the sample of the five raw flat median counts, was introduced.

2017-05 The morning HC IFU flats in the VIS-arm are subject of vignetting. This can be seen in the bi-modal distribution of the ADC monitoring plots.

Figure B, ADC problem in 2010.

The windows currently (2012-07-06) in use are (x1, y1, x2, y2) coords given in pixels of the master flat product:

    UVB S1R: '1452 1500 1471 1600'
    UVB S2R: '1477 1500 1493 1600'
    UVB S3R: '1501 1500 1518 1600'
    UVB S1B: ' 513 2330 532 2380'
    UVB S2B: ' 539 2330 557 2380'
    UVB S3B: ' 564 2330 582 2380'

    VIS S1R: '1652 2050 1666 2100'
    VIS S2R: '1674 2050 1684 2100'
    VIS S3R: '1697 2050 1711 2100'
    VIS S1B: ' 426 2605 444 2757'
    VIS S2B: ' 452 2605 470 2757'
    VIS S3B: ' 477 2605 496 2757'

    NIR S1R: '622 1133 632 1175'
    NIR S2R: '638 1133 647 1175'
    NIR S3R: '655 1133 665 1175'
    NIR S1B: '125 1080 135 1111'
    NIR S2B: '143 1080 152 1111'
    NIR S3B: '160 1080 169 1111'

The values are derived from the daily health check IFU master flats. The counts in several windows of the master echelle flat are retrieved and analysed according to the formalism given above.

Short term stability

In 2018 shutter problems in the VIS arm resulted in flat, of which not all of the five raw flats showed the same exposure level and of which some of the raw flats showed a minor gradient in the slit function. For this reason the QC procedures have been extended to check for the coherence of the count level within the raw flats.

FITS key QC1 database: table, name definition class* HC_plot** more docu

[calculated by QC procedure] xshooter_flat..raw_med_std raw frame med counts std HC [docuSys coming]

*Class: KPI - instrument performance; HC - instrument health; CAL - calibration quality; ENG - engineering parameter
**There might be more than one.

Scoring&thresholds Short term stability

The standard deviation of the sample of five raw flat counts is monitored in the upper right box for the VIS-arm only.

Thresholds have been set to operationally meaningful values.