Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:
    ABbuilder documentation
    OCA syntax
    createAB 
    AB concepts, events and rules

Association Block: structure and content

The syntax is the same as for the dfos configuration files:

- The first column has the name of the qualifier.
- The other columns have the content.
- Any text behind a # is considered a comment.

Qualifier description
TOOL_VERSION version of createAB (read from createAB)
CONFIG_VERSION version of config.createAB used (read from config file)
 
INSTRUMENT read from $DFO_INSTRUMENT
DATE DFO processing date (from tool call)
 
DPR_CATG value of DPR.CATG of primary file
RAW_TYPE as defined in <instr>_classification.h
RECIPE pipeline recipe, as defined in <instr>_association.h
DRS_TYPE type of DRS (CPL/CON), as defined in config.createAB
PACK_DIR HISTORICAL, OBSOLETE. Was:
name of subdir for packing, as defined in <instr>_classification.h
BATCH_ID name of processing batch to which the AB belongs to ($MODE_$DATE)
 
AB_NAME name of AB (inherited from first raw file, with grouping tag read from <instr>_organisation.h)
AB_EVENT as defined in <instr>_organisation.h
COMPLETENESS completeness status (COMPLETE/INCOMPLETE; derived by createAB)
PROCESS_STATUS CREATED/SUCCESSFUL/FAILED (set by createAB and processAB)
 
OBS_PROG_ID value of OBS.PROG.ID of first raw file
OBS_MODE mode of observing run (SM/VM/NONE); SCIENCE data have SM or VM, CALIB have NONE. To be evaluated by monitor tools
OBS_ID value of OBS.ID of first raw file
 
MJD-OBS value of MJD-OBS of first raw file (start of integration)
MJD-OBS_MOD calculated from MJD-OBS+EXPTIME/2 (MJD-OBS of middle integration; used for time matching; applicable to whole group of input files)
   
PROD_ROOT_NAME root name of products (inherited from first raw file and grouping tag read from <instr>_organisation.h)
   
LOG_NAME name of processing log (inserted by processAB)
# pipeline product path
PROD_PATH full path name to products
# raw match key
RAW_MATCH_KEY set of specific keys (name=value) used to match the raw (input) files;
RAW_MATCH_KEY read from <instr>_organisation.h; e.g.:
#RAW_MATCH_KEY INS_GRAT_NAME=HR
#RAW_MATCH_KEY INS_GRAT_ORDER=ORD12
#RAW_MATCH_KEY INS_GRAT_WLEN=WLN447.1
# input file(s)
RAWFILE column 2: full pathname of input file; column 3: DO_CLASS (read from <instr>_classification.h); e.g.:
#RAWFILE ${DFO_RAW_DIR}/2004-08-31/GIRAF.2004-09-01T14:26:24.151.fits FIBER_FLAT
# associated raw file(s) (only for SCIENCE; taken from same night only!)
RASSOC full pathname of associated raw files (e.g. ACQUISITION frames; SCIENCE only);
RASSOC used for packing; read from <instr>_association.h, qualifier RASSOC
# product file(s) under ${DFO_CAL_DIR}/<DATE> (with status in calib archive)
PRODUCTS column 2: final name of product file(s) of this AB (if any; set by updateAB)
PRODUCTS column 3: name of ingested file (if any; CALIB only; set by updateAB)
# associated mcalib file(s), used for processing
MCALIB associated master calibration data, needed for processing (set by createAB; read from <instr>_association.h)
MCALIB column 2: type (REAL or VIRTUAL); VIRTUAL stands for predicted product names, REAL for existing ones
MCALIB column 3: full pathname to product (VIRTUAL files will have PIPEFILE names, REAL files CALIBFILE names)
MCALIB column 4: PRO_CATG of product
MCALIB column 5: associated AB of that product (pointing to the associated calib raw files)
MCALIB column 6: DELTA_T between MCALIB and group of input files, in days
# addional mcalib file(s), used for packing (SCIENCE only)
MASSOC calibration products required for packing (inserted by createAB; read from <instr>_association.h)
# parameters for processing
PARAM name and value of non-default parameter(s) required to execute the recipe (set by createAB; read from <instr>_association.h)
# defined waitfors (for CONDOR)
WAITFOR name of waitfor(s) for this AB (relevant for DRS=CON only; set by createAB; read from <instr>_association.h)
# further associated information:
FURTHER_PS name(s) of associated 'further' information (graphical etc.) (set by updateAB; read from config.createAB, section PACK_ADD; relevant for packing)
# status of AB
TEXEC execution time, in seconds; inserted by processAB
TQCEXEC same, for QC report; inserted by processQC
AB_STATUS messages about creation, execution, update of this AB
 
# ============== SOF section starts here ============================
 
SOF_CONTENT full SOF (set of frames) file content extracted from this AB (read by processAB)