 |
 |
 |
 |
 |
 |
 |
 |
||||||
 |
HST Data Handbook for NICMOS |
|||||
|
||||||
3.3 Basic Data Reduction: calnica
The calnica task operates on individual NICMOS datasets and performs the job of removing the instrumental signature from the raw science data. The calnica task also tries to identify cosmic ray hits and combines the multiple readouts in MULTIACCUM observations.
The inputs to calnica are the raw science (*_raw.fits) files. The output of calnica is usually a single file containing the calibrated science data (*_cal.fits). For MULTIACCUM mode datasets there is an additional intermediate output file (*_ima.fits) which contains the calibrated data from all the intermediate readouts. The _ima.fits data are fully calibrated up to, but not including, the cosmic ray rejection. The format of the input and output science data files are identical, so that the output data can be reused as input to calnica, if desired. One could, for example, process a science data file through some subset of the normal calibration steps performed by calnica, examine or modify the results, and then process the data through calnica again, performing other calibration steps or using alternate calibration reference files. One example of such a procedure would be reducing data where there are significant changes in the quadrant bias level from readout to readout in a MULTIACCUM sequence. In section 4.1, we discuss this common NICMOS data anomaly, and in section 4.1.5 we describe one technique for treating it using the biaseq task. At present, the use of this routine requires multiple, re-entrant applications of calnica in order to partially process the images before and after the use of the biaseq task.
Figure 3.1 shows the portion of a calibrated NICMOS science file header containing the switches and reference file keywords that pertain to the processing performed by calnica. The accompanying flow chart (figure 3.2) shows the sequence of calnica calibration steps, the input data and reference files and tables, and the output data file. Each calibration step is described in detail in the following sections.
Figure 3.1: Partial NICMOS Header
Figure 3.2: Calnica Processing Flow
ZSIGCORR (Zero-Read Signal Correction)
At the beginning of a NICMOS observation the detector pixels are reset to a bias level and then read out to record that bias level. There is an interval of approximately 0.2 seconds that elapses between the time each pixel is reset and then read. Because NICMOS does not have a shutter, signal from external sources starts to accumulate during that 0.2 second interval. When the initial (or "zeroth") read is later subtracted from subsequent readouts, any signal in the zeroth read will also be subtracted. For very bright sources, the amount of signal in the zeroth read can be large enough to lead to inaccurate linearity corrections, as well as the failure to detect saturation conditions, in the NLINCORR calibration step, because the linearity correction and saturation checking both depend on the absolute signal level accumulated in a pixel.For MULTIACCUM observations, the ZSIGCORR step is used to estimate the amount of source signal in the zeroth read and to supply corrections to the NLINCORR step for that signal. The ZSIGCORR step estimates the amount of signal in the zeroth read by first measuring the amount of signal that arrived in each pixel between the zeroth and first reads, and then scaling that signal to the effective exposure time of the zeroth read (nominally 0.203 seconds). Pixels that have an estimated zeroth read signal greater than 5 times their ERR value are assumed to contain detectable signal; those below this threshold are ignored. The user may set a different zero-read detection threshold by using the zsthresh task parameter for calnica. The estimated zeroth read signal is then passed, on a pixel-by-pixel basis, to the NLINCORR step, so that it can account for that signal when applying linearity corrections and saturation checking on the zeroth-read subtracted images with which it works. The ZSIGCORR step also performs saturation checking on the zeroth and first readout images.
Note that this technique will not work well for pixels covered by targets that are so bright that the signal is already beginning to saturate in either the zeroth or first readouts.
Pixels that are determined to have detectable signal in the zeroth read are marked in the DQ images of the output *_ima.fits file with a data quality flag value of 2048. The ZSIGCORR routine uses the MASKFILE, NOISFILE, DARKFILE, and NLINFILE reference files.
The ZSIGCORR routine is implemented in calnica versions 3.0 and higher. It was implemented in the standard OPUS calibration pipeline on 11 November 1997, and archived data from before that time does not have the ZSIGCORR step applied. If you are concerned about accurate flux measurements for bright sources in NICMOS observations taken before that time, you may wish to reprocess the data using the latest version of calnica (see section 3.5), or to retrieve the data again from the HST Archive via OTFR, which will automatically apply the ZSIGCORR step. ZOFFCORR (Subtract Zero-Read Image)
The ZOFFCORR step of calnica performs the subtraction of the zeroth read from all readouts in a MULTIACCUM file. This step is performed for data generated by the MULTIACCUM readout mode only. For ACCUM and BRIGHTOBJ readout modes, the subtraction of the zeroth read is performed on-board, because the images returned to the ground are formed by taking the difference of initial and final non-destructive detector readouts.The pipeline will subtract the zeroth read image from all readouts, including the zeroth read itself. Furthermore, the self-subtracted zeroth-read image will be propagated through the remaining processing steps and included in the output products, so that a complete history of error estimates and data quality (DQ) flags is preserved. After this step is performed, the science data are in the same form as the raw science data from any other observing mode and are processed the same way throughout the remaining steps of calnica. No reference files are used by this step.
MASKCORR (Mask Bad Pixels)
Flag values from the static bad pixel mask file are added to the DQ image. This uses the MASKFILE reference file, which contains a flag array for known bad (hot or cold) pixels. There is one MASKFILE for each detector. In general, only truly "defective" pixels are included in the MASKFILE reference images available from the STScI calibration database. There are other bad pixels, however, which you may wish to mask out, such as those pixels affected by "grot" (see section 4.1.5). If you wish, you can create a new MASKFILE which includes additional bad or suspect pixels, and reprocess your data using this new mask.BIASCORR (Wrapped Pixel Correction)
NICMOS uses 16-bit analog-to-digital converters (ADCs), which convert the analog signal generated by the detectors into signed 16-bit integers. Because the numbers are signed and because the full dynamic range of the converter output is used, raw pixel values obtained from individual detector readouts can range from -32768 to +32767 DN. In practice the detector bias level is set so that a zero signal results in a raw value on the order of -23000 DN. In ACCUM and BRIGHT OBJECT modes, where the difference of initial and final readouts is computed on-board, the subtraction is also performed in 16-bit arithmetic. Therefore, it is possible that the difference between the final and initial pixel values for a bright source could exceed the dynamic range of the calculation, in which case the final pixel value will wrap around the maximum allowed by the 16-bit arithmetic, resulting in a negative DN value. Given the level at which the NICMOS detectors saturate, and the analog-to-digital conversion factor, the maximum "real" pixel value that is expected is on the order of +40000 DN. Such a value will be wrapped to about -26000 DN by the on-board difference calculation.The BIASCORR step searches for pixel values in the range -23500 to -32768 DN and adds an offset of 65536 DN to these pixel values to reset them to their original real values. The BIASCORR step only affects ACCUM and BRIGHTOBJ mode observations, although it is applied to all NICMOS data sets. For MULTIACCUM data, it should have no effect.
No reference files are used by this step.
NOISCALC (Compute Statistical Errors)
Errors for MULTIACCUM, ACCUM and BRIGHTOBJ modes are computed in the calnica pipeline. The NOISCALC step performs the task of computing an estimate of the errors associated with the raw science data based on a noise model for each detector. Currently the noise model is a simple combination of detector read noise and Poisson noise in the signal, such that:where
rd is the read noise in units of electrons, adcgain is the analog-to-digital conversion gain factor (in electrons per DN) and counts is the signal in a pixel in units of DN. Noise is computed in units of electrons, but the result is converted to units of DNs for storage in the error image. The detector read noise is read pixel-by-pixel from the NOISFILE reference image and depends on the read rate of the observation (fast or slow), as well as the number of initial and final reads (NREAD). Separate NOISFILEs are required for each combination of read rate and NREAD. The data quality flags set in the DQ image of the NOISFILE are propagated into the DQ images of all image sets (imsets) being processed.
Because the noise calculation is performed before dark subtraction has taken place, the noiseless electronic signal component known as "shading" (see DARKCORR below) is still present in the data. In calnica versions 3.3 and later the NOISCALC step estimates the level of the shading signal in the data by computing column or row statistics in the DARKFILE reference file. The computed shading estimate is subtracted from the signal in the science image when computing Poisson noise on the detected counts. This yields a more accurate noise estimate than what was produced in earlier versions of the pipeline.
Throughout the remaining steps in calnica, the error image is processed in lock-step with the science image, getting updated as appropriate. Errors are mostly propagated through combination in quadrature. For MULTIACCUM data sets, the ERR array for the final calibrated image (*_cal.fits) is populated by the CRIDCALC step of calnica based on the calculated uncertainty of the count rate fit to the MULTIACCUM samples.
In general, the values in the error images should only be regarded as an estimate of the data uncertainties. The precise pixel noise values in NICMOS images are difficult to compute a priori because many factors may contribute, sometimes in unpredictable ways (see, e.g., the discussions of data anomalies such as cosmic ray persistence in chapter 4). DARKCORR (Dark Current and Bias Shading Subtraction)
Dark images taken with NICMOS contain three distinct, additive signal components: the so-called "shading", amplifier glow, and the true dark current. The shading is a noiseless signal that appears as gradient across a detector quadrant and is due to the fact that the bias level on the pixels is gradually changing as they are being read out. The amplitude of the shading signal is a function of the time since a pixel was last read out. The amplifier glow is signal produced by a small amount of infrared radiation from the detector readout amplifiers. The amplitude of the amplifier glow is directly proportional to the total number of readouts in an observation. The true detector dark current signal is quite small for the NICMOS arrays and is linearly dependent on the total exposure time of an observation.Because the shading and amp glow signals depend on factors other than the exposure time of an observation, it is not possible to apply a simple scaling of a single dark reference image to match the exposure time of the science data that is being calibrated. Therefore, a library of dark current images is maintained for each of the three cameras, covering all of the predefined MULTIACCUM sample sequences, and a subset of ACCUM exposure times and NREAD values (see the NICMOS Instrument Handbook). The reference dark file appropriate for the exposure sequence used in MULTIACCUM, or the exposure time and NREAD values used in ACCUM, is determined by the OPUS generic conversion process when it populates the DARKFILE reference file keyword in the primary header of raw data files. The calnica DARKCORR step subtracts the dark reference images, readout-by-readout for MULTIACCUM observations, from the science data. Error estimates of the dark current, stored in the ERR images of the DARKFILE, are propagated in quadrature into the ERR images of all processed science imsets. Data quality (DQ) flags set in the DARKFILE are also propagated into the DQ images of all processed imsets.
For BRIGHTOBJ mode data, dark subtraction is skipped by default in calnica, because in general the short exposure times should result in insignificant dark current relative to the object signal. In practice there may be bias components with non-zero amplitude (e.g., akin to "shading") which are present in BRIGHTOBJ mode data. At present, however, there is no standard procedure for removing these. Given the very limited use of BRIGHTOBJ mode for on-orbit science, we will not discuss its reduction further here.
Chapter 4 includes a more detailed discussion of NICMOS dark and bias components, their properties and behavior, including irregularities which are not well handled by the standard processing pipeline, and which require special care on the part of the user. Chapter 4 also discusses recent updates to the dark reference files available from STScI, including the new dark generator WWW tool. Proper removal of additive instrumental signatures (i.e. dark and bias) can be one of the most important steps in achieving high quality, science grade NICMOS data reductions, and we recommend that the user read the relevant sections of chapter 4 in detail. NLINCORR (Linearity Correction)
The linearization correction step corrects the integrated counts in the science image for the non-linear response of the detectors. The observed response of the detectors can conveniently be represented by 2 regimes:The NLINCORR step applies the linearity correction to pixels with signal below their defined saturation levels. However, it applies no correction to pixels in the high signal regime, but rather flags them in the DQ image as saturated (DQ value = 64). This step uses the NLINFILE reference file, which consists of a set of images containing the c1, c2, and c3 correction coefficients and their variances at each pixel. The [NODE,2] extension of the NLINFILE sets the saturation value for each pixel. Error estimates on the correction applied to non-saturated pixels are propagated into the ERR images of all imsets processed. Data quality flags set in the NLINFILE are also propagated into the processed DQ images. There is one NLINFILE per detector.
- At low and intermediate signal levels the detector response deviates from the incident flux in a way that is correctable using the following expression:
- where c1, c2 and c3 are the correction coefficients, F is the uncorrected flux (in DN) and Fc is the corrected flux. In practice the coefficient c1 is set to 1, so that the total correction increases from a value of 1 starting at the zero signal level.
- At high signal levels-as saturation sets in-the response becomes highly non-linear and is not correctable to a scientifically useful degree; the saturation level is about 30,500 DN, with a standard deviation of about 2,000 DN.
Early versions of NICMOS non-linearity correction used a linear correction scheme, rather than the 2nd-order parameterization that is now employed.1 Starting in calnica v3.3, the NLINCORR step was updated to accommodate the 2nd-order correction, but is backwards-compatible such that old NLINFILEs using the linear correction may still be used if desired. New reference files have been created that include these higher order corrections. Additionally, the nonlinearity reference files include a [NODE,1] extension. This sets the data value below which no nonlinearity correction is applied. It now appears instead that the NICMOS arrays are somewhat nonlinear at all count levels. In the new NLINFILEs, therefore, the [NODE,1] values are uniformly set to 0.0. Because calnica v3.3 was released after the end of NICMOS Cycle 7 operations, all Cycle 7 and 7N data retrieved from the HST archive prior to 26 September 2001 were processed by OPUS using the older non-linearity corrections. If you think that your data may benefit from the newer, more accurate linearity corrections, you should reprocess the images (see section 3.5), or retrieve them again using OTFR, which will automatically process them using the new nonlinearity corrections.
1 The correction term in the nonlinearity equation given above is quadratic. This is then multiplied by the uncorrected flux, yielding an effectively cubic relation between uncorrected and corrected values.
BARSCORR (Bars Correction)
Some NICMOS images will have pairs of bright and dark columns or rows, which have come to be known as "bars". The bars are believed to arise from electrical cross-talk in the detector lines during the readout of one camera when another of the cameras enters the auto-flush idle state. The bars manifest themselves as a noiseless DC offset of a few DNs along a pair of columns or rows, with the pattern replicated exactly in all four image quadrants. They are discussed and illustrated in section 4.2 of this Handbook.Versions 3.3 and higher of calnica use the BARSCORR routine to remove the effects of the bars from MULTIACCUM observations. The routine scans pairs of columns or rows (depending on the camera) in each readout of the MULTIACCUM observation and identifies those with median signals more than 2
No reference file is used by this step.
FLATCORR (Flat Field Correction)
In this step the science data are corrected for variations in gain between pixels by multiplying by an (inverse) flatfield reference image. This step is skipped for observations using a grism because the flatfield corrections are wavelength dependent. This step uses the FLATFILE reference file, which contains the flatfield image for a given detector and filter (or polarizer) combination. Error estimates and DQ flags contained in the FLATFILE are propagated into the processed images. There is one FLATFILE per detector and filter combination.UNITCORR (Convert to Count Rates)
The conversion from raw counts to count rates is performed by dividing the science (SCI) and error (ERR) image data by the exposure time (TIME) image data. No reference file is needed.PHOTCALC (Photometric Calibration)
This step provides photometric calibration information by populating the photometry keywords PHOTMODE, PHOTFLAM, PHOTFNU, PHOTZPT, PHOTPLAM, and PHOTBW with values appropriate to the camera and filter combination used for the observation. The photometry parameters are read from the PHOTTAB reference file, which is a FITS binary table containing the parameters for all observation modes. The values of PHOTFLAM and PHOTFNU are useful for converting observed count rates to absolute fluxes in units of erg/s/cm2/Angstrom or Jy, respectively (see section 5.3). PHOTCALC does not alter data values (which remain in units of counts or counts per second), but simply populates header keywords with the appropriate calibration information.CRIDCALC Cosmic Ray Identification and Signal Accumulation)
This step identifies and flags pixels suspected of containing cosmic ray (CR) hits. For MULTIACCUM mode observations, this step also combines the data from all readouts into a single image. In MULTIACCUM mode, the data from all readouts are analyzed pixel-by-pixel, iteratively computing a linear fit to the accumulating counts-versus-exposure time relation and rejecting outliers from the fit as CR hits. The default rejection threshold is set to 4Once all outliers have been identified, a final countrate value, and its uncertainty, are computed for each pixel using only non-flagged samples. The result of this operation is stored as a single imset in the output *_cal.fits file in which the number of unflagged samples used to compute the final value for each pixel and the total exposure time of those samples is reflected in the SAMP and TIME images, respectively. The variance ascribed to the final mean countrate is the uncertainty in the slope of the counts-versus-time relation at each pixel location, and is recorded in the ERR image of the *_cal.fits file. Pixels for which there are no unflagged samples, e.g., permanently hot or cold pixels, will have their output SCI, ERR, SAMP, and TIME values set to zero, with a DQ value that contains all flags that were set.
CRIDCALC is only applied to MULTIACCUM images. For data taken in ACCUM or BRIGHTOBJ mode, both the raw and calibrated images will contain cosmic rays and should be treated as with ordinary CCD data.
BACKCALC (Predict Background)
This step computes a predicted background (sky plus thermal) signal level, based on models of the zodiacal scattered light and the telescope plus instrument thermal background. This step uses the BACKTAB reference table which contains the background model parameters. Results of these predictions, along with direct estimates of the background level from the data themselves, are written to the BACKEST1, BACKEST2, and BACKEST3 header keywords. The image data are not modified in any way. At the time of this writing, this step has not yet been implemented. If there are future changes to the calibration procedures or software regarding the BACKCALC step, these will be reported in the Space Telescope Analysis Newsletter (STAN) and posted on the NICMOS website.WARNCALC (User Warnings)
In this step various engineering keyword values from the *_spt.fits files are examined and warning messages are generated if there are any indications that the science data may be compromised due to unusual instrumental characteristics or behavior. At the time of this writing, this step has not yet been implemented. Any future changes to the software implementing WARNCALC will be reported in the Space Telescope Analysis Newsletter (STAN) and posted on the NICMOS website.
|
|||||
| Space Telescope Science Institute
http://www.stsci.edu Voice: (410) 338-1082 help@stsci.edu |