Common DFOS tools:
= Data Flow Operations System, the common tool set for DFO
Calibration memory and AB creation
This page deals with the operational issue of how to fill the
calibration memory properly, and with the interplay of createAB and that
The calibration memory is the list of all calibration product files currently
available for association. It is used and maintained by the association tool createAB.
There are two flavours of calibration products: virtual calibrations
and real calibrations.
- Real calibration products are the processed and certified products, stored
in $DFO_CAL_DIR. They are either dynamic (updated regularly) or static (updated
never or rarely). The dynamic
calibrations are created by the pipeline from raw files and are stored in $DFO_CAL_DIR/<date>.
The static calibrations normally have no parent raw file and are stored in $GEN_CALDIR
- Virtual calibrations are calibrations which are predicted
and do not yet exist as fits files at creation time. With their predicted name,
they can be used for association in a cascade. They either will become available later,
or will only be used for association of raw files, in which case they will never become
The two flavours of calibration products are collected in two
- $DFO_CAL_DIR/MCAL is the pool of all real calibrations available
for association by createAB. It is filled with symbolic links
to the real files in the storage area ($DFO_CAL_DIR/<date>.
- $DFO_CAL_DIR/VCAL is the pool of virtual calibrations. It
is filled with header files being extracted from the parent raw files.
In either case the calibration memory is only a subset of the available dates
(for performance reasons). The date range is configured in config.createAB.
The calibration memory is managed by DFOS in the following
- MCAL is filled from scratch by createAB each time the tool is
called. The memory depth (= the number of days contained in MCAL) is determined
by its reference date (the latest date in the pool) and the number of dates.
The number is configured as $N_MCAL_LIST in config.createAB. The reference
date is either the latest available date in $DFO_CAL_DIR: this is the default
choice if createAB is called in its standard way (createAB -m <mode>
-d <process_date>); or a specified date (createAB -m <mode>
-d <process_date> -D <reference_date>).
- VCAL is filled and depleted incrementally by createAB/updateAB
in the course of day-to-day operations. It has a depth of $N_VCAL_LIST dates as configured in config.createAB.
Virtual calibrations. In routine operations the virtual
calibrations, once they are certified and become real, are automatically removed
from VCAL and enter MCAL, until they finally become outdated, are replaced by similar, new ones
and disappear completely from the calibration memory. The QC scientist usually does not need
to take care about these directories, except for fine tuning the memory depth parameters
(typically 5-10 days).
There are two exceptional cases when entries in VCAL
are removed, or kept, in a user-controlled way:
- in the course of certifyProduct, a product is rejected
(certify='n'); then the user is asked whether or not its name should be deleted
- moveProducts has the configurable option VCAL_REMOVE
to remove certain products, defined per RAW_TYPE.
Criteria for removing or not are:
- "case 1: bad raw, bad product, or no product" A raw file
has bad quality (e.g. zero exposure level), hence cannot produce a reasonable product.
It is then completely useless and should also be removed from VCAL (to prevent its association
into other ABs). The QC scientist might consider to enter a comment and use hideFrames for
hiding this file in the archive.
- "case 2: good raw, bad product or no product" A raw
file is good, but the pipeline failed to produce a reasonable result: then you may want
to keep associations of this raw file to science files or other calibrations and hence
do not remove it from the VCAL directory.
- "case 3: good raw, no product" The pipeline is not
able to correctly process certain setups or even RAW_TYPEs, independent of their quality;
it never produces a product which could go through certification. Then, again, you would
like to keep the VCAL product for the associations.
Cases 1 and 2 are a case-by-case decision which is done
as part of certifyProducts. Case 3 is rule-based and hence
is configured, as part of moveProducts.
Explicit deletion of virtual calibrations. Only when there is
the need to create ABs outside of the daily workflow (e.g. for a historical
set of June data while DFO date is already in December), some explicit actions are necessary for calibration memory maintenance:
- delete all entries under $DFO_CAL_DIR/VCAL/.
- run createAB -m CALIB -d <june-date> -D <june-date>
Then MCAL is correctly filled with June data, and
no virtual calibs from December can be associated. If you would call createAB without explicitly specifying -D <reference_date>, it would scan and find the December data instead!
If you want
to switch back to December, remove again the files in $DFO_CAL_DIR/VCAL/, and call createAB again in the standard way.
The current content of VCAL and MCAL is displayed in the dfoMonitor output.
| Last update: April 26, 2021 by rhanusch