Development Timeline

Look at what I am working on right now

OVERVIEW: PIPELINE, VERIFICATION and PRODUCTION TIMELINE

1. Timeline start: June 20th
2. Finalize pipeline development of code V2
3. Verification against previous pipeline
   1. Sky level
   2. Photometry
   3. Astrometry
   4. Header content
   5. weight_map
   implementation of new metric system
   1. Verification against previous pipeline
4. implementation of new pystack module (waiting for Luc)
   1. Verification of Sextractor input parameters
5. Identification and run on test suite
6. Foreach dataset in test suite
   1. Photometry and SKY verification
   2. Astrometry verification
   3. Product header and packaging verification
7. Timeline: End of September
8. RUN during October

DETAILS: Pipeline development

RAW file production

1. We will use the cached RAW files produced almost one year ago: Wed Jun 2 12:48:17 PDT 2004
2. I will override the existing collection of raw files:
3. Space have been allocated: Wed Jun 2 12:49:02 PDT 2004

   Association identification

   1. Maintained by ST-ECF
   2. replicated to CADC in hstecf database
   3. WAITING: full list of INPUT parameters describing the stack (ECF) and required by the pipeline (ECF)

   Association statistics (17 May 2004)

   1. Number of associations in table: 29967
   2. Number of associations in CADC science: 25412
   3. Number of associations without calibrated filters: 419 (with POL )

   Cross Correlation

   1. No work have been done to this module
   2. WAITING: For the new metric task so distorsion could be remove before the entire WFPC2 pipeline process (STScI/HOOK)

   IMSTACK

   1. Change into a pyraf module
   2. WAITING: Change in pystack to accomodate low S/N stacking considering mean based statistics (SIMARD)
   3. The check the noise properties for the extreme cases listed below

   Astrometric correction

   1. Change into a pyraf module: Done
   2. Implement proper logging in the header when failures (no stars, etc...) (DURAND)

   Statistical Image classification

   1. Change into a pyraf module: Done
   2. Implement proper loggin mechanism: (DURAND)
   3. Check algorithm: (DURAND)

   Source extraction

   1. What to do when filter don't have a calibrated zero point? (SCHADE)
   2. Change into a pyraf module: DONE
   3. Implement proper logging mechanism (DURAND)
   4. Check values extracted from source extraction (DURAND)
   5. SYNC extraction parameters with voSextractor (same config file?) (DURAND)

   AD Put and header augmentation

   1. Change into a pyraf module: NOT DONE
   2. WAITING: Decision about replacing existing version or adding new version to AD (SCHADE)
   3. WAITING: Decision to run voPublish or NOT. (SCHADE)

   DETAILS: Pipeline Verification

   Overall image quality

   1. Suppose to be STScI responsibilities. Since Mike Corbin departure, we haven't got any new names for this task (STScI)

   Better sky evaluation

   1. start work on getting potential sky trends in respect to the position of HST with earth limb and zodiacal light

   Special Science verification stacks

   1. u2j20g04b
   2. u62z2003b
   3. u5hh0602b
   4. u36n0202b
   5. u4fi010ab: Wrong zero point and presence of cosmic rays
   ACS cases
   1. j6fy05010
      1. J6FY05LAQ EXP-CR1 yes
      2. J6FY05LMQ EXP-CR1 yes
      3. J6FY05M1Q EXP-CR2 yes
      4. J6FY05M7Q EXP-CR2 yes
      5. J6FY05MEQ EXP-CR3 yes
      6. J6FY05MIQ EXP-CR3 yes
      7. J6FY05MRQ EXP-CR4 yes
      8. J6FY05MVQ EXP-CR4 yes
      9. J6FY05010 PROD-DTH yes
      10. J6FY05011 PROD-CR1 yes
      11. J6FY05012 PROD-CR2 yes
      12. J6FY05013 PROD-CR3 yes
      13. J6FY05014 PROD-CR4 yes
   2. j6m910030
      1. J6M910FQQ EXP-DTH yes
      2. J6M910FUQ EXP-DTH yes
      3. J6M910FYQ EXP-DTH yes
      4. J6M910G2Q EXP-DTH yes
      5. J6M910030 PROD-DTH yes
      J8F644010
      1. J8F644CIQ EXP-CR1 yes
      2. J8F644CLQ EXP-CR1 yes
      3. J8F644CPQ EXP-CR2 yes
      4. J8F644CTQ EXP-CR2 yes
      5. J8F644010 PROD-DTH yes
      6. J8F644011 PROD-CR1 yes
      7. J8F644012 PROD-CR2 yes

   VO considerations

   Parameters implementation

   Please see following section

   Pipeline execution status

   1. Potentially dependant of the Distributed processing system. I do have a fall back mechanism which is quite efficient using Sybase as the controller

   Population

   1. Populate a table with al the parameters to NULL
   2. When running pipe, populate the values
   3. Augment the header before storage

   ---

   Fixed parameters for WFPC V2

   The count ratio is a correction after flat field since the flat were computed with gain 14 in a (200:600) centered window.

   parameter	Gain	PC1	WF2	WF3	WF4
   Noise	7	5.24	5.51	5.22	5.19
   noise	15	7.02	7.84	6.99	8.32
   gain	7	7.12	7.12	6.90	7.10
   gain	15	13.99	14.50	13.95	13.95
   14/7	ratio	1.987	2.003	2.006	1.955

   WFPC2 parameters for V2

   Parameter Name	Chip dep	Dataset Dep.	Origin	Header Storage	Code to change	Done?	CVO NAME	UNIT	Verify?	required for pipe?
   ACQUISITION PARAMETERS
   APERTURE	NO (asn_maint)	YES	ORIG: hstdads..shp_data.shp_aperobj DB: asn_maint_mem..aperture	CEAPER	PREPROCESSING	YES		DIMENSIONLESS	NO	NO
   FILTER	NO (?)	YES	DB: asn_maint..filter DB: hstdads..science_sci_spec1234	CEFILTER	PREPROCESSING + photometric flag in asn_product (image_type and image_sex)	NO		DIMENSIONLESS	NO	YES
   FILTER NAME	NO	YES	LOOKUP: Standard filter name (UBVRI)	CEFNAME	PREPROCESSING	NO		DIMENSIONLESS	NO	NO
   GAIN	YES?	YES	DB: asn_maint_mem..nominal_gain	CEGAIN	PREPROCESSING + Waiting for flat field answer?	NO		NULL	NO	YES
   READNOISE	YES?	YES	CALC: LIB_ECF	CENOISE	PREPROCESSING	NO	CVONOISE	electrons	NO	YES
   GAIN STACK	YES?	YES	CALC: normalized GAIN * ncombine	CESGAIN	PREPROCESSING + Depends on nature (sum, average) final image?	NO	CVOGAIN	electrons	NO	YES
   OBSMODE	YES	NO	CALC: libecf..obsmode Header of the raw/calibrate file(OTFR)	CEOBSMO	PREPROCESSING	NO		DIMENSIONLESS	NO	YES
   PIXSCALE	YES	NO	CALC: libecf..sciap using aperture	CEPIXSC	PREPROCESSING	NO		arcsec	NO	YES
   SPATIAL MEASUREMENTS
   RESOLUTION	YES	NO	CALC: libecf..resolution $diameter=2.4e10; #diameter of the telescope $arc_rad = 206264.8; #number of arcsec/rad $rayleigh=1.22*$photplam * $arc_rad / $diameter; #1.22 * lambda/D $resolution = sqrt( ($rayleigh**2) + ($pixscale**2))	CEPSFW	PREPROCESSING	NO	CVOPSFW	arcsec	NO	YES
   POSITION ANGLE	YES	NO	CALC: libecf..rot ORIENTAT in Calibrated header is angle from NORTH to Y axis, counter clock wise TIME DEPENDENCY because of chip drifting around?	CEROT	PREPROCESSING	NO	CVOROT	DEGREES	NO	NO: have to check is sextractor is giving us the right orientation?
   GALACTIC	YES	NO	CALC: libecf..galactic Compute galactic coordinate(required by EB-M)	CEGAL	POSTPROCESSING	NO	CVOGAL	degrees	NO	NO
   BOUND COORDINATES?	YES	NO	CALC: libecf..wcs Coordinate for each corners(daniel)	CEOS[ga,EQ,EC]0n	POSTPROCESSING	NO	CVOS[ga,EQ,EC]0n	DIMENSIONLESS (degrees)	NO	NO
   E(B-V)	YES	NO	CALC: libecf..ebv Required DUST maps and software Dependencies on galactic	CEEBMV	PREPROCESSING	NO		DIMENSIONLESS	NO	NO
   HTM?	YES	NO	CALC: libecf..htm	CECELL	POSTPROCESSING	NO		DIMENSIONLESS	NO	NO
   WAVELENGTH MEASUREMENTS
   PHOTFLAM	YES	NO	CALC: libecf..photflam	CEPHOTFL	PREPROCESSING	NO		erg/sec/nu/cm**2	NO	YES
   WAVELENGTH-PIVOT	YES	NO	CALC: libecf..filter Manual PIVOT, PHOTPLAM	CEPIVWV	PREPROCESSING	NO	CVOPIVWV	nanometers	NO	YES
   WAVELENGTH-SPAN	YES	NO	CALC: libecf..filter RECTW	CEWSPANR	PREPROCESSING	NO	CVOWSPAN	nanometers	NO	NO
   WAVELENGTH-VOMAX	YES	NO	CALC: libecf..filter	CEWMAX	PREPROCESSING	NO		meters	NO	NO
   WAVELENGTH-VOMIN	YES	NO	CALC: libecf..filter	CEWMIN	PREPROCESSING	NO		meters	NO	NO
   ZEROPOINT-ST	YES	NO	CALC libecf..zero $zero=-2.5*log10($PHOTFLAM) - 21.1; 21.1 is the ST zero point for all #STMAG DANIEL m = -2.1log(DN/EXPTIME) + ZEROPOINT-ST	CEZERST	PREPROCESSING	NO		UNIT	NO	YES
   ZEROPOINT-AB	YES	NO	CALC libecf..zero DANIEL	CEZERAB	PREPROCESSING	NO		UNIT	NO	YES
   ZEROPOINT-VEGA	YES	NO	CALC libecf..zero	CEZERVE	PREPROCESSING	NO		UNIT	NO	YES
   TIME MEASUREMENTS
   TIME-MIN	NO	YES	DB: asn_maint..start_time_dmf	CETMIN	PREPROCESSING	NO	CVOTMIN	EEE dd-MMM-yyyy HH:mm:ss	NO	NO
   TIME-MAX	NO	YES	DB: asn_maint..start_time_dmf+asn_maint..time_span	CETMAX	PREPROCESSING	NO	CVOTMAX	EEE dd-MMM-yyyy HH:mm:ss	NO	NO
   TIME-SAMP	NO	YES	DB: asn_maint..exptime	CETSAMP	PREPROCESSING	NO	CVOTSAMP	seconds	NO	NO
   TIME-FILL	NO	YES	CALC: asn_maint..exptime/asn_maint..time_span DANIEL	CETFILL	PREPROCESSING	NO	CVOTFILL	DIMENSIONLESS	NO	NO
   Time Array	NO	YES	LOOKUP: Time array (start, stop, start, stop...)	CET001	PREPROCESSING	NO	CVOT001	DIMENSIONLESS	NO	NO
   FLUX MEASUREMENTS
   Flux in point sources	YES	YES	CALC: From sextractor/image_type	CEFSTA	POSTPROCESSING	NO	CVOFSTA	JANSKY	NO	NO
   Flux in extended sources	YES	YES	CALC: From sextractor/image_type	CEFGAL	POSTPROCESSING	NO	CVOFGAL	JANSKY	NO	NO
   Total flux	YES	YES	CALC:	CEFTOT	POSTPROCESSING	NO	CVOFTOT	JANSKY	NO	NO
   Flux in low frequencies	YES	YES	CALC: From sextractor/image_type	CEFLOWF	POSTPROCESSING	NO	CVOFLOWF	JANSKY	NO	NO
   Flux in high frequencies	YES	YES	CALC:	CEFHIGH	POSTPROCESSING	NO	CVOFHIGH	JANSKY	NO	NO
   Flux at S/N=10	YES	YES	CALC:	CEFLX10	POSTPROCESSING	NO	CVOFLX10	JANSKY	NO	NO
   STATISTICS MEASUREMENTS
   Number Extended Sources	YES	YES	CALC: From sextractor, less 0.5	CENEXT	POSTPROCESSING	NO	CVONEXTD	DIMENSIONLESS	NO	NO
   Number Point Sources	YES	YES	CALC: From sextractor, greater or equal 0.5	CENPOINT	POSTPROCESSING	NO	CVONPOIN	DIMENSIONLESS	NO	NO

   ---

   Filters

   Here are the filter characteristics for the WFPC2 V2 Alberto, we will use the querator filter definitions for mean, max and min instead of the actual CVO values based on this RECW. After discussion with Pat, we decided that this level of refinement was not needed...

   
   # Table 5.1: Current Values of PHOTFLAM and Zeropoint in the VEGAMAG system.
   # Instrument Filter1       PC      WF2     WF3     WF4
   # photflam1     VegaZP1         photflam2       VegaZP2         photflam3 VegaZP3       photflam4       VegaZP4
   # pivot1	pivot2	pivot3	pivot4
   # lambda_min Lambda_max Lambda_mean lanbda_width
   WFPC2 F122M   8.088e-15       13.768  7.381e-15       13.868  8.204e-15       13.752 8.003e-15        13.778    1795.8   1747.6   1828.9   1791.3   114.680000           137.119995           125.900002            22.440001 
   WFPC2 F160BW  5.212e-15       14.985  4.563e-15       15.126  5.418e-15       14.946 5.133e-15        15.002    1518.8   1515.2   1521.8   1518.6   121.745003           167.455002           144.600006            45.709999
   WFPC2 F170W   1.551e-15       16.335  1.398e-15       16.454  1.578e-15       16.313 1.531e-15        16.350    1831.2   1820.8   1838.2   1830.2   144.869995           188.330002           166.600006            43.459999
   WFPC2 F185W   2.063e-15       16.025  1.872e-15       16.132  2.083e-15       16.014 2.036e-15        16.040    1971.8   1969.5   1973.4   1971.6   175.029999           204.770004           189.899994            29.740000
   WFPC2 F336W   5.613e-17       19.429  5.445e-17       19.462  5.451e-17       19.460 5.590e-17        19.433    3359.5   3359.0   3359.2   3359.1   313.174988           350.225006           331.700012            37.049999
   WFPC2 F218W   1.071e-15       16.557  9.887e-16       16.646  1.069e-15       16.558 1.059e-15        16.570    2204.3   2202.2   2205.7   2204.1   193.304993           230.095001           211.699997            36.790001
   WFPC2 F255W   5.736e-16       17.019  5.414e-16       17.082  5.640e-16       17.037 5.681e-16        17.029    2600.1   2598.6   2601.0   2599.9   234.089996           274.910004           254.500000            40.820000
   WFPC2 F300W   6.137e-17       19.406  5.891e-17       19.451  5.985e-17       19.433 6.097e-17        19.413    2992.8   2989.0   2994.3   2992.    252.820007           325.579987           289.200012            72.760002
   WFPC2 F336W   5.613e-17       19.429  5.445e-17       19.462  5.451e-17       19.460 5.590e-17        19.433    3359.5   3359.0   3359.2   3359.1   313.174988           350.225006           331.700012            37.049999
   WFPC2 F343N   8.285e-15       13.990  8.052e-15       14.021  8.040e-15       14.023 8.255e-15        13.994    3431.9   3431.7   3431.9   3431.9   341.524994           343.875000           342.700012             2.350000
   WFPC2 F375N   2.860e-15       15.204  2.796e-15       15.229  2.772e-15       15.238 2.855e-15        15.206    3737.6   3737.4   3737.7   3737.6   371.980011           374.420013           373.200012             2.440000
   WFPC2 F380W   2.558e-17       20.939  2.508e-17       20.959  2.481e-17       20.972 2.558e-17        20.938    3982.8   3981.9   3982.7   3982.3   356.459991           425.940002           391.200012            69.480003
   WFPC2 F390N   6.764e-16       17.503  6.630e-16       17.524  6.553e-16       17.537 6.759e-16        17.504    3889.9   3889.9   3889.9   3889.9   386.549988           391.049988           388.799988             4.500000
   WFPC2 F410M   1.031e-16       19.635  1.013e-16       19.654  9.990e-17       19.669 1.031e-16        19.634    4092.7   4092.7   4092.6   4092.6   401.250000           415.950012           408.600006            14.700000
   WFPC2 F437N   7.400e-16       17.266  7.276e-16       17.284  7.188e-16       17.297 7.416e-16        17.263    4369.8   4369.8   4369.8   4369.8   435.640015           438.160004           436.899994             2.520000
   WFPC2 F439W   2.945e-17       20.884  2.895e-17       20.903  2.860e-17       20.916 2.951e-17        20.882    4312.1   4312.0   4311.7   4311.8   405.079987           451.519989           428.299988            46.439999
   WFPC2 F450W   9.022e-18       21.987  8.856e-18       22.007  8.797e-18       22.016 9.053e-18        21.984    4557.3   4557.5   4555.5   4556.4   394.744995           487.255005           441.000000            92.510002
   WFPC2 F467M   5.763e-17       19.985  5.660e-17       20.004  5.621e-17       20.012 5.786e-17        19.980    4670.3   4670.3   4670.3   4670.3   457.980011           474.619995           466.299988            16.639999
   WFPC2 F469N   5.340e-16       17.547  5.244e-16       17.566  5.211e-16       17.573 5.362e-16        17.542    4694.6   4694.6   4694.6   4694.6   468.149994           470.649994           469.399994             2.500000
   WFPC2 F487N   3.945e-16       17.356  3.871e-16       17.377  3.858e-16       17.380 3.964e-16        17.351    4865.4   4865.4   4865.4   4865.4   485.204987           487.795013           486.500000             2.590000
   WFPC2 F502N   3.005e-16       17.965  2.947e-16       17.987  2.944e-16       17.988 3.022e-16        17.959    5013.3   5013.2   5013.2   5013.3   499.855011           502.545013           501.200012             2.690000
   WFPC2 F547M   7.691e-18       21.662  7.502e-18       21.689  7.595e-18       21.676 7.747e-18        21.654    5483.9   5484.3   5483.3   5483.8   520.270020           568.929993           544.599976            48.660000
   WFPC2 F555W   3.483e-18       22.545  3.396e-18       22.571  3.439e-18       22.561 3.507e-18        22.538    5442.9   5445.6   5439.0   5442.2   459.070007           581.330017           520.200012           122.260002
   WFPC2 F569W   4.150e-18       22.241  4.040e-18       22.269  4.108e-18       22.253 4.181e-18        22.233    5644.4   5646.2   5642.1   5644.1   504.114990           600.684998           552.400024            96.570000
   WFPC2 F588N   6.125e-17       19.172  5.949e-17       19.204  6.083e-17       19.179 6.175e-17        19.163    5893.5   5893.5   5893.5   5893.5   586.849976           591.750000           589.299988             4.900000
   WFPC2 F606W   1.900e-18       22.887  1.842e-18       22.919  1.888e-18       22.896 1.914e-18        22.880    6001.3   6005.0   5996.8   6001.   497.750000           655.650024           576.700012           157.899994
   WFPC2 F622W   2.789e-18       22.363  2.700e-18       22.397  2.778e-18       22.368 2.811e-18        22.354    6187.9   6189.3   6186.4   6187.9   566.330017           659.869995           613.099976            93.540001
   WFPC2 F631N   9.148e-17       18.514  8.848e-17       18.550  9.129e-17       18.516 9.223e-17        18.505    6306.4   6306.4   6306.4   6306.4   629.054993           632.145020           630.599976             3.090000
   WFPC2 F656N   1.461e-16       17.564  1.410e-16       17.603  1.461e-16       17.564 1.473e-16        17.556    6563.8   6563.7   6563.8   6563.8   655.325012           657.474976           656.400024             2.150000
   WFPC2 F658N   1.036e-16       18.115  9.992e-17       18.154  1.036e-16       18.115 1.044e-16        18.107    6590.8   6590.8   6590.8   6590.8   657.674988           660.525024           659.099976             2.850000
   WFPC2 F673N   5.999e-17       18.753  5.785e-17       18.793  6.003e-17       18.753 6.043e-17        18.745    6732.3   6732.3   6732.3   6732.3   670.840027           675.559998           673.200012             4.720000
   WFPC2 F675W   2.899e-18       22.042  2.797e-18       22.080  2.898e-18       22.042 2.919e-18        22.034    6717.7   6718.3   6717.4   6718.1   626.924988           715.875000           671.400024            88.949997
   WFPC2 F702W   1.872e-18       22.428  1.809e-18       22.466  1.867e-18       22.431 1.883e-18        22.422    6917.1   6917.   6918.5   6918.8   619.969971           768.030029           694.000000           148.059998
   WFPC2 F785LP  4.727e-18       20.688  4.737e-18       20.692  4.492e-18       20.738 4.666e-18        20.701    8686.3   8668.5   8707.   8692.5   823.494995          1033.104980           928.299988           209.610001
   WFPC2 F791W   2.960e-18       21.498  2.883e-18       21.529  2.913e-18       21.512 2.956e-18        21.498    7872.5   7866.5   7880.6   7875.5   731.669983           862.130005           796.900024           130.460007
   WFPC2 F814W   2.508e-18       21.639  2.458e-18       21.665  2.449e-18       21.659 2.498e-18        21.641    7995.9   7983.5   8012.2   8001.6   732.400024           908.200012           820.299988           175.800003
   WFPC2 F850LP  8.357e-18       19.943  8.533e-18       19.924  7.771e-18       20.018 8.194e-18        19.964    9114.   9100.4   9128.8   9118.2   881.380005          1048.619995           965.000000           167.240005
   WFPC2 F953N   2.333e-16       16.076  2.448e-16       16.024  2.107e-16       16.186 2.268e-16        16.107    9544.9   9544.9   9544.9   9544.9   951.974976           957.224976           954.599976             5.250000
   WFPC2 F1042M  1.985e-16       16.148  2.228e-16       16.024  1.683e-16       16.326 1.897e-16        16.197    10221.   10216.   10225.   10222.   1013.150024          1074.250000          1043.699951            61.099998
   
   

   ---

   STATIC PARAMETERS

   Area Chip PC1: 0.000108506944444444
   Area Chip WF2: 0.000434027777777778
   Area Chip WF3: 0.000434027777777778
   Area Chip WF4: 0.000434027777777778
   
   Action: Remove vignetting from the surface
   

   ---

   VoSextractor mandatory parameters (any sextractor run)

   1. CVOGAIN
   2. CVOPSFW
   3. CVOSSAMP
   4. CVOWMIN
   5. CVOWMAX
   6. CVOWERR
   7. CVOTMIN
   8. CVOCROT
   9. CVOSERR
   10. CVOMAG0

   ap_diam_pix = (3.0/3600.0)*(1.0/hdrinfo['CVOSSAMP']
   -PIXEL_SCALE	= CVOSSAMP
   -SEEING_FWHM    = CVOPSFW
   -GAIN		= CVOGAIN
   -MAG_ZEROPOINT	= CVOMAG0
   -PHOT_APERTURES	= ap_diam_pix
   fjy = 10.0**(-0.4*(voSrc_propv['magab_aper']+48.60)+23.0)  = Flux in jansky
   
   crot = CVOCROT
   pa_image = float(sex.POS_ANGLE) - 90.0 - crot
   while pa_image > 360.0:
       pa_image = pa_image - 360.0
       while pa_image < 0.0:
   	pa_image = pa_image + 360.0
   	voSrc_propv['pos_angle'] = pa_image
   	voSrc_propve['pos_angle'] = abs(float(data[21]))
   

   ---

   DB SCHEMA

   Table 1: STACK parameters

   Note: Suggestion: Instead of having (1,2,3,4) for the 4 chips, this table should be chip based....

   Column name	Format	Explanation
   DATASET	varchar	Dataset name (Dataset_[pc1,_wf2,_wf3,_wf4])
   CEAPER	varchar	Aperture
   CEFILTER	varchar	Filter(s)
   CEFNAME	char(1)	Filter Name (UBVRI)
   CEGAIN	float	Gain
   CENOISE	float	Readout Noise
   CESGAIN	float	Final GAIN for the stack (ncombine)
   CEOBSMO	varchar	Obsmode sequence
   CEROT	float	Rotation (according to standard)
   CEOSGA	varchar	Bound galactic coordinates
   CEOSEQ	varchar	Bound equatorial coordinates
   CEOSEC	varchar	Bound ecliptic coordinates
   CEEBMV	float	Galactic dust absorption
   CEPHOTFL	float	Photflam
   CEPIVWV	float	Pivot Wavelength, photplam
   CEWSPAN	float	SPAN Wavelength
   CEWMAX	float	Max Wavelength
   CEWMIN	float	Min Wavelength
   CEZERST	float	ST MAG zero point
   CEZERAB	float	AB MAG zero point
   CEZERVE	float	VE MAG zero point
   CETMIN	float	Time min
   CETMAX	float	Time max
   CETSAMP	float	Time sampling
   CETFILL	float	Time filling factor
   CET001	varchar	Time array (vector)
   CEFSTA	float	Flux in point sources
   CEFGAL	float	Flux in extended sources
   CEFTOT	float	Total flux
   CEFLOWF	float	Flux in low frequencies
   CEFHIGF	float	Flux in high frequencies
   CEFLX10	float	Flux at S/N=10
   CENEXT	int	Number of extended sources
   CENPOINT	int	Number of point sources

   Table 1: ASN_PRODUCT

   Column name	Format	Explanation
   status	char	Status of the stack (P=processed, etc...)
   product_id	varchar(16)	Dataset_name
   family	varchar(16)	one of (calibrated,imtype.science,sextractor,astrom,preview)
   ingest_date	datetime	Pipe processing date
   location	varchar(8)	Production location
   received	char	If family is complete
   version	int	Version (1, or 2)
   product_date	datetime	asn_creation_date
   rashift	float	Astrometric correction for RA
   decshift	float	Astrometric correction for DEC
   nstar	int	Number of stars used for astrometric correction

   ---

   WFPC2 processing algorithm

   Processing steps for POD

   Get the POD from AD

   Processing steps for RAW

   Process the POD through POD2RAW, which is calling OTFR/OPUS

   Processing steps for calibration

   Process the RAW through CALWFPC

   Process through WARMPIX

   NEW TASK???: Geometric correction and recentering using the CCD position correction table

   Processing steps for PCROSS (if not already done)

   Chip by chip cross correlation.

   Vote system to get the most significant shift of all shifts

   Processing Steps for IMSTACK WFPC2 V2: Pass one (for J members)

   Flat processing

   MAP creation

   FLAT_map = FLAT with ( greater than 2.0 = bad since we want to ignore only the heavily vignetted pixels )

   Reversed Bad Pixels MAP

   C1H_map = 1 if C1H = 1024
   C1H_map = 0 if C1H other values but 0
   C1H_map = 1 for C1H = 0

   Final MAP

   MAP_j = FLAT_map_j * C1H_map_j

   Variance_J0

   Variance_j0 = [ Image_j0 / gain_j + R_j²/gain_j²] * (t_m/ t_j) * (gain_j / gain_stack) * MAP_j

   • Where Image_j0 is the j th image not flatfielded (the sub 0)
   • t_m is the average exposure time
   • j is the image counter
   • gain_stack is the final gain of the stack ( 7.0 or 14.0, mixed gain will be 7.0 )
   • R_j is the readout noise for the image at at gain_j

   Image_J0 = Image_j / Flat_j Since the flat are reversed normalized.

   by substitution, Variance_j = Flat_j² * Variance_j0
   because
   Variance_j = del(I)/del(I_j)² * Variance_j0

   Finally,
   Variance_j = Flat_j² * [ Image_j / ( Flat_j * gain_j) + R_j² / gain_j² ] * (gain_j / gain_stack) * (t_m/ t_j)* MAP_j

   Rewritten,
   Variance_j = [ (Flat_j * Image_j / gain_j ) + Flat_j² * R_j² / gain_j² ] * (gain_j / gain_stack) * (t_m / t_j) * MAP_j

   The science data is simply:
   Data_j = Image_j * (gain_j / gain_stack) (t_m/ t_j)

   Images and variances are then shifted

   This is rather complex. We also have to shift other image components like Flat_J and MAP_J since they are also needed after step 2.

   Question: Is the readout noise of the sifted image the same as the original

   We are using imshift in linear interpolation for shift (if shift value > 0.15 PC pixel)

   Action: Please check the origin of the max shift

   the the input data to pystack is

   Data_js and Variance_js
   where Data_js = Data_j shifted

   Then we need to shift the MAP_j and the Flat_j into MAP_js and Flat_js

   Then we proces these 2 input images lists with pystack and the results are a stack image with bad values masked to 10e06 called P₁

   Processing Steps for WFPC2 V2: Pass two (for J images)

   It is essentially the same equation than above but the have first to "descale" the P₁ image with the ration of gain and the ration of exposure time, i.e.: (gain_j / gain_stack) and (t_m/ t_j)

   The variance J2

   Variance_j2 = [ (Flat_js * P₁ / gain_j ) * (t_j/t_m) * (gain_stack / gain_j ) + Flat_js² * R_j²/ gain_j² ] * (gain_j / gain_stack) * (t_m/ t_j)* MAP_js

   The data J2 is simply

   Data_j2 = Data_js

   Then pystack pass 2

   The resulting stacked image and stack are composing the final product.

   Astrometric correction (using USNOB)

   Image caracterisation

   catalogue extraction

   Storage in AD

   ---

   GAIN manipulation

   From Source extractor for dummies V4

   Method	Effective Gain	Magnitude zeropoint	Input image
   1	gain * total_exp_time	Zeropoint(1 sec)	Image in count/sec
   2	gain	Zeropoint(1 sec) * log10(total_exp_time)	Sum of N frames
   3	N * gain	Zeropoint(1 sec) * log10(average_exp_time)	Average of N frames
   4	2 * N * gain / 3	Zeropoint(1 sec) * log10(average_exp_time)	Median of N frames

   Magnitude systems

   STMAG

   ZEROPT_second = -2.5 * log₁₀(PHOTFLAM) + PHOTZPT
   ZEROPT_exposure = -2.5 * log₁₀(PHOTFLAM) + PHOTZPT + 2.5 * log₁₀(exptime)

   PHOTZPT = -21.1 (always)
   m_stmag = -2.5 * log₁₀(counts/exposure) + ZEROPT_exposure
   m_stmag = -2.5 * log₁₀(counts/sec) + ZEROPT_seconds
   m_stmag = -2.5 * log₁₀( PHOTFLAM * DN / EXPTIME) + PHOTZPT

   AB mag

   m_AB = -2.5 log₁₀( Flux_nu ) - 48.594
   where flux_nu is in ergs cm^-2 s^-1 HZ^-1
   m_AB = -2.5 log₁₀( Flux_lambda ) - 2.402 - 5.0 * log₁₀( lambda )
   where flux_lambda is in ergs cm^-2 s_-1 A_-1
   and lambda in A
   F_nu = f_lambda * lambda² / c

   flam = 3x10^18 * fnu / lam^2
   fnu = (lam^2 * flam) / 3x10^18
   

   Converting an image to flux

   flux = DN * PHOTFLAM / exposure_time
   where flux is in cm^-2s^-1A^-1

   
   flux = DN * PHOTFLAM / exposure_time * PHOTPLAM**2 / 2.9972E18 * 1E+23
   where flux is in Jy

   JANSKY

   1 Jy = 1.0 * 10^-26 (W/m²/Hz) = 1.0E-23 erg/s/cm²/Hz

   FWHM

           seeing_fwhm = 1.22 * (photplam / 24000000000.0 ) * 206264.806247
   	seeing_fwhm = math.sqrt( (seeing_fwhm * seeing_fwhm) + (pixscale[chip] * pixscale[chip])  )
   	24000000000.0 = size of the mirror
   	206264.806247 = 180 * 3600. / pi
   
   
           $diameter=2.4e10; #diameter of the telescope
   
   	$arc_rad = 206264.8; #number of arcsec/rad
   
   	$rayleigh=1.22*$photplam * $arc_rad / $diameter; #1.22 * lambda/D
   
   	$resolution  = sqrt( ($rayleigh**2) + ($pixscale**2))
   

   CVORES

   cvosres = (( photplam / 10.0E10 ) / 2.4 ) * (2.063 * (10**5)) cvonyqr = cvosres / pixscale[chip] cvosres = cvosres / 3600.0 The Nyquist ratio (spatial_resolution/spatial_sample). Values less than 2.5 are undersampled.

   equations

   # [Y ABnu] = -2.5 * log([X ergs/cm^2/s/Hz]) - 48.594 # [Y ABnu] = -2.5 * log([X ergs/cm^2/s/A]) - 2.402 - 5.0 * log(lambda A) # [Y ABnu] = -2.5 * log([X W/m^2/Hz]) - 56.094 # [Y ABnu] = -2.5 * log([X photon/cm^2/s/A]) + 16.852 - 2.5 * log(lambda A) # [Y ABnu] = -2.5 * log([X photon/cm^2/s/um]) + 16.852 - 2.5 * log(lambda um) [Y Jy] = 1.0E+23 * [X erg/cm^2/s/Hz]

   Daniel Durand, April 2004

   ---

   Alberto Micol, September 2004

   Photometric systems: zeropoints and formulae

   
   Generic: m = -2.5 * log10( DN/EXPTIME ) + ZEROPOINT
   
   ST sys:  mst = -2.5 * log10( Flambda ) - 21.10
   
   AB sys:  mAB = -2.5 * log10( Fnu ) - 48.60
   
   where Flambda is in erg/cm2/sec/A
     and Fnu        in erg/cm2/sec/Hz
   
   Flambda = countRate * PHOTFLAM
   
   Fnu     = Flambda * lambda2 / c
                      = countRate * PHOTFLAM * PHOTPLAM2 / c
   
   where:
   
    - counteRate is the number of counts diveded by the exptime
   
    - PHOTFLAM is the flux of a source with constant flux per unit wavelength
               (in erg/cm2/sec/A) which produces a count rate of 1DN/sec.
      Note: Obviously a change of GAIN will have riperussions on PHOTFLAM
   
    - c must be given in A/sec, that is: 2.99792E+18 A/sec
        since PHOTPLAM is in A, and PHOTFLAM in erg/cm2/sec/A.
   
   Hence,
   
    mst = -2.5 * log10( countRate * PHOTFLAM ) - 21.10
        = -2.5 * log10( countRate ) -2.5*log10( PHOTFLAM ) - 21.10
   
        = -2.5 * log10( count ) -2.5*log10( PHOTFLAM ) - 21.10 + 2.5 * log10( EXPTIME )
   
   
    mAB = -2.5 * log10( countRate * PHOTFLAM * PHOTPLAM2 / c ) - 48.60
        = -2.5 * log10( countRate ) -2.5*log10(PHOTFLAM*PHOTPLAM2/c) - 48.60
   
        = -2.5 * log10( count ) -2.5*log10(PHOTFLAM*PHOTPLAM2/c) - 48.60 + 2.5*log10( EXPTIME )
   
    
   

   Effective Gain

   The effective gain is to be used within PIPE only when running sextractor, that is, when the noise of the image must be computed.

   
   The effective gain is g for the noise of an single image.
   
   The effective gain is g for the noise of an coadded image (pure sum).
   
   The effective gain is N * g for the noise of an averaged image.
   
      That's because the poissonian noise can be computed only from the Total number of electrons, not from the average number of electrons; and
   the total number of electrons is N * g * the number of counts of an averaged image.
   
   All the formulae to do with gain
   
   

   Following the ZEROPOINTS (from imstack.py to sex_cat.py)

   
   imstack.py sets the following zeropoints:
   
   ZEROSTS: zeropt_stmag_second, comment="CADC: zeropt STMAG for one second" 
   ZEROSTE: zeropt_stmag_exposure, comment="CADC: zeropt STMAG for average exposure time"
   ZEROABS: zeropt_abmag_second, comment="CADC: zeropt ABMAG for one second"
   ZEROABE: zeropt_abmag_exposure, comment="CADC: zeropt ABMAG for average exposure time"
   
   in the following way:
   
   ZEROSTS: zeropt_stmag_second = -2.5 * math.log10( photflam ) + photzpt
   ZEROSTE: zeropt_stmag_exposure = zeropt_stmag_second + 2.5 * math.log10( average_exp_time )
   ZEROABS: zeropt_abmag_second   = -2.5 * math.log10( photplam**2 * photflam / 2.99792458e18 ) -48.594
   ZEROABE: zeropt_abmag_exposure = zeropt_abmag_second + 2.5 * math.log10( average_exp_time )
   
   where PHOTZPT comes from the header (and should be -21.10)
   and the following related CVO parameters:
   
   CVOMAG0:== ZEROABE = zeropt_abmag_exposure
   
   Note: imstack.py uses a slightly different zeropoint for the AB sys: -48.594 instead of 48.60.
   
   
   
   
   

   Following the