Common DFOS tools:

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:

- supports complex between/tryBetween timematch rules


- uploads list_static_mcalibs to $DFO_WEB_SERVER for OPSHUB tool

Please call the tool once, without parameters, to make this happen.

tools related to calSelector:
| createCalibMap | verifyAB | writeBreakpoint

general overview of calSelector here

OCA: rules | syntax

- Find all calibration maps here.
- The CALCHECKER and DFOS_OPS calib maps are linked to other pages.


[ used databases ] databases none
[ used dfos tools ] dfos tools
[ output used by ] output html pages under
[ upload/download ] upload/download html pages and RLS files to
topics: description | syntax rules | types of rules | validity | output | consistency checks | how to use | tool configuration | comments | operational hints


[Historical: For the transition from the old to the new version of calSelector, check the migration description here.]

[ top ] Description

For a given instrument, the calibration cascade is the fundamental relationship between the data types (calibrations and science). The cascade is coded in OCA syntax (organisation, classification, association). In the dfos system, this information is structured in three main configuration files, with further files containing macros. These files are compiled by the OCA tools (createAB, calChecker) into a single OCA rules file, named <instr>.RLS.

Several dfos applications use OCA syntax. There is dfos daily operations, using createAB to generate ABs for processing and packing. There is calChecker, creating ABs to analyze their completeness. There is calSelector, an application running behind the archive web interfaces and creating association information for the delivery of calibration data sets for a user-selected science file.

Each of these tools uses a dedicated OCA rule set, coming in different flavours: raw-to-master (dfos and calSelector), raw-to-raw (calChecker; see here for more). Apart from the modes, the rulesets also differ by completeness. For instance, Health Check calibrations are included in dfos but not in calChecker.

All data types are handled together in one single rule set, although every instrument has many sub-cascades: modes, arms, settings. These sub-cascades are sometimes loosely connected, via detector raw types, sometimes not at all. The single ruleset corresponds to the OCA rules file. This file is usually big, machine but not human readable. A key concept in this situation is visualization. Without visualization of the rules file, it is almost impossible to properly understand, check, and modify the calibration cascade. This visualization is the purpose of createCalibMap. Its output is called a calibration map.

The ruleset is structured top-down such that we start with the most fundamental calibrations (detector calibrations) and then add the more complex ones following the calibration cascade (e.g. flat fields, arclamp, standard stars). At the bottom the science data follow. The calibration map displays this arrangement turned by 90 degrees, ordered from left to right, with the simplest detector calibrations at left and the science data at the right end. The map is ordered as a table, where each raw type is sorted in a column, while its properties (raw_type, do_class, products, recipe name etc.) can be found in rows. Further to the cells, the map has also relations between cells, corresponding to the input products from other raw types as required for processing a given raw type.

Find an example of a calibration map here (produced by createCalibMap).

[ top ] Business rules for syntax.

The tool createCalibMap parses the (gcc-compiled) rule set with name <inst>.RLS and works out the individual components of the calibration map and its relations: raw_types, grouping rules, products etc. These components are then turned into graphical elements to constitute the calibration map.

[gcc compilation is the process needed to go from the individual OCA files (<instr>_association.h etc.) to the rules file <instr>.RLS, with macro expansion and removal of comment lines. This is done within createAB.]

Most of, if not all, the parsing can be provided in an automatic way by the utility tool that is called in option -C and also by createAB. Hence the sentence below is likely to be obsolete.

[The parsing cannot be done in a completely automatic way but needs some help in the form of conventions being followed. In most cases they result in some changes to be done in the OCA files. The modifications are not very challenging. These changes need to be done in the respective OCA files, not in the rules file directly (they would be overwritten by createAB or calChecker quickly!). This has also the advantage that these contain the macros which are easier to edit (if necessary at all) than the expanded strings.]

Note that these modifications are transparent to the OCA tools, they are required only for createCalibMap, with the modified OCA files still working fine with the OCA tools.

Here is an overview of the conventions for the OCA files (most if not all conventions are controlled by the utility tool

General rules:
- raw types will appear colour-coded by their CATG (ACQUISITION, CALIB, SCIENCE)
an acquisition should have CATG = "ACQUISITION" (not: CALIB or SCIENCE)



- a classification record is recognized by the starting 'if', it must be the first element [the parser uses (if $1=="if")]
- the classification rule must be one line [everything until 'then']
- the CATG must be the second-last row of each record [the last one being '}'].

An exceptional EXPOSE_TO_CALSELECTOR=T is recognized by the tool, if it follows the line with the RAW.TYPE definition.


select execute(ACTION_DETMON_BIAS) from inputFiles where RAW.TYPE=="DETMON_BIAS"
   group by DET.READ.SPEED,TPL.START as (TPL_A,tpl);

- an organisation record is recognized by the starting 'select execute' [the parser uses this as a marker]
- it must be one line up to RAW.TYPE==... [otherwise only the first line will be recognized by the parser]
- the 'group by' clause must be one line [otherwise only the first line will be recognized by the parser]

  minRet = 1; maxRet = 1;
  select file as MASTER_BIAS from calibFiles where PRO.CATG=="MASTER_BIAS" and inputFile.DET.READ.SPEED==DET.READ.SPEED
  minRet = 1; maxRet = 1;
  select file as NF_LOCWIDTH from calibFiles where PRO.CATG=="NF_LOCWIDTH" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.DET.READ.SPEED==DET.READ.SPEED
  minRet = 1; maxRet = 1;
  select file as FF_LOCWIDTH from calibFiles where PRO.CATG=="FF_LOCWIDTH" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.DET.READ.SPEED==DET.READ.SPEED
  minRet = 1; maxRet = 1;
  minRet = 1; maxRet = 1;
  select file as STD_ARGSPECTRA from calibFiles where PRO.CATG=="STD_ARGSPECTRA" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.INS1.OPTI1.POS==INS1.OPTI1.POS and inputFile.DET.READ.SPEED==DET.READ.SPEED

- an association record is recognized to start with 'action' [the parser uses (if $1=="action")]
- its end is recognized by the last '^}' [avoid having ^} before (e.g. as part of the recipe section!) where ^} means } being the first character in the line]
- select file as: must be one line [otherwise tooltips with match rules get confused; the semicolon belongs to the next line]
- the label "as ... from" must be the same as the one in "PRO.CATG==..."

For CALSELECTOR rules, there is the additional requirement to normally have the validity of the association encoded as tryBetween/between pair:

 select file as DISPERSION_SOLUTION from calibFiles where PRO.CATG=="DISPERSION_SOLUTION" and ... and MJD-OBS between inputFile.MJD-OBS - <ext_val> and inputFile.MJD-OBS + <ext_val> and MJD-OBS tryBetween inputFile.MJD-OBS - <val> and inputFile.MJD-OBS + <val>  ;

where <val> is the calibration plan validity (for migrated rules: taken from the DELTAT_RULE in the DFOS_OPS rules), and <gen_val> an extended (relaxed) validity, used for performance and to deliver a master calibration with a disclaimer if no instance could be found within the <val> range. Both <val> and <gen_val> can be asymmetric.

Find more under 'validity', and here.

If you have applied changes to your RLS file (other than the format changes), call createCalibMap -r <rules file> -C, in order to gcc-compile your RLS file properly.

[ top ] Supported types of OCA rule sets.

There are three supported types of OCA rule sets. These types are configured in the respective tool configuration file (see below). These are the types:

types association DO_CLASS
complete description
of all raw types?
static calibrations used? versioning supported? $RLS_PATH
DFOS_OPS raw to master yes no yes yes no $DFO_CONFIG_DIR/OCA
CALCHECKER raw to raw no yes no no no $DFO_CONFIG_DIR/CALCHECK/OCA
CALSELECTOR raw to master yes yes yes* yes yes $DFO_CONFIG_DIR/CALSELECTOR/<start_date>

* same OCA rule as DFOS_OPS, but display in calibmap is restricted by configuration

[ top ] Comments

You have the option to provide comments to the calib map. Comments are collected in a comment file called info.createCalibMap (in the same path as the config file). There are the following types of comments:
comment type displayed where?
whole calib map top table, "General comment"
sub-cascade same
raw_type bottom part of calib map, in raw_type column

Find the description of the comment file here.

[ top ] Validity

Validity is the time range in which a calibration can (likely) be applied. It describes the empirical timescale of changes in the parameters to be measured by the calibration. This concept makes sense only in the framework of associations, i.e. in the relation between two files (science and calibration). An individual calibration is always valid if it has measured the target property in a valid way.

The concept of validity is implicit in the DFOS_OPS rules: there is the DELTAT_RULE part in the <instr>_associations.h files which however is not evaluated by OCA itself but is used by createAB to flag the ABs in case of violation. For calChecker, validity is evaluated by the tool itself. For calSelector v2.0, the same validity information as for the DFOS_OPS rules is required, but it must be coded within OCA. This is done in the automatic migration by parsing the DELTAT_RULEs and translating them into a combined between and tryBetween statement.

The between statement (-30...30) gets all candidates matching between -30 and 30 days; from those candidates, the closest-in-time is taken. Depending on whether the match is within the inner boundaries (defined by tryBetween) or only the outer boundaries, calSelector then returns a flag "calib_plan" (for the tryBetween match being satisfied) or "extended" (if not). In that way we can convey the QC concept of 'OK' or 'NOK' for time matches and avoid delivering no return data sets at all if there is only a mild violation of the formal validity. The logical justification of the two statements is that in DFOS operations, we have the calibration plan matches (corresponding to tryBetween) but we also accept (at least in many cases) a mild violation if no closer match is possible. With calSelector v2 we cannot convey the message that this particular exception is acceptable, but we can generally offer those candidates, with a disclaimer.

For static calibrations, usually the PREVIOUS time match should be applied.

createCalibMap has functions to check and display the validity (time-match) information in the calibration map, see here.

[ top ] Output

The tool creates the following output:

Here, <name> is the configured type of OCA rule set (one of DFOS_OPS, CALCHECKER, CALSELECTOR).

Find all calibration maps from .

[ top ] Consistency checks

There are some consistency checks performed by the tool. They are marked in the product match section of the HTML output file:
in the product match rule (meaning minRet=0; maxRet=0): obsolete match, could be removed from <instr>_association.h
M 0,1
this cell has multiple match rules between a given raw_type and a given product pro_catg. This could indicate a logical problem although it does not fire an OCA error. It creates a display issue for the calibration map, which is why only the first match is displayed.
unconstrained minRet, maxRet undefined in <instr>_association.h: could be risky
  as background of a cell with a match rule (and tooltip "none"): no match rule found, could be a violation of the "one-line rule" (see above under association.h) or just a very simple match by PRO.CATG only .
GT or T
CALSELECTOR: a time-match rule is missing for a general calibrations (GT) or a normal calibration (T)

The tool also checks the completeness of the RLS file:

- raw_types without products are flagged in the ANALYSIS section, first line, for OCA ruleset type DFOS_OPS and CALCHECKER (CALIB only); this indicates that these data never produce virtual products and cannot be associated to other data types, which is ok for Health Check calibrations but worth checking;

- products listed for matching but not listed as products of raw_type (type DFOS_OPS; ANALYSIS section, second line): these could either be static calibrations, or configuration errors;

- unconstrained matches (minRet, maxRet): these could be potentially dangerous if more than one instance could exist.

The tool assumes that there is exactly one action per raw_type. In those cases where more than one action is defined, the tool raises a warning and takes the first defined action.

There are legal cases with multiple raw_type definitions (most likely when there is the need for multiple DO_CLASSes, as e.g. for the IR instruments with ON/OFF definitions). That raw_type is then marked as M, the multiple definitions are connected by 'OR', and the DO_CLASSes listed comma-separated.

Additional markings

Certain properties are highlighted in the calibration maps:

- type DFOS_OPS: those products which are listed but not used further are marked in italics

- type DFOS_OPS: MASSOCs are marked

- types CALCHECKER and CALSELECTOR: validity in days

- type CALSELECTOR: proper time-match rule for gencalibs (<); valdity for between and tryBetween (see the codes on the result page)

- type CALSELECTOR: the property EXPOSE_TO_CALSELECTOR=T is marked.

[ top ] How to use

Type createCalibMap -h | -v for on-line help and version.


createCalibMap -r <path>/<your_instr>.RLS [-m ALL]

(complete pathname!) to create the calibration map for all <your_instr> modes, and

createCalibMap -r <path>/<your_instr>.RLS -m IMG

to create the calibration map for <your_instr> mode IMG. The default is $DFO_CONFIG_DIR/OCA/<your_instr>.RLS.

The type of rules is configured in the corresponding tool configuration file config.createCalibMap under the same path as the OCA rule set. For the operational DFOS rules, the type is DFOS_OPS, and the path is $DFO_CONFIG_DIR/OCA.


createCalibMap -r <path>/<your_instr>.RLS -D [-m <mode>]

to check all raw_types and all pro_catgs defined (or for a specified mode) in the rules file.


createCalibMap -r <path>/<your_instr>.RLS -A

to create the calibration maps for all modes configured in the tool config file, and

createCalibMap -r <path>/<your_instr>.RLS -E <selected science raw_type >

to get a listing of the dependencies for the specified pro_catg.

Replace <path>/<your_instr>.RLS in the above examples by the pathname of your operational rules file (e.g. for the calChecker calibmap of HAWKI: -r $DFO_CONFIG_DIR/CALCHECK/OCA/HAWKI.RLS).


createCalibMap -X

to enter a dialog for bulk creation of calib maps. The tool offers a menue. You select an option (one of DFOS_OPS, CALCHECKER, CALSELECTOR) and then the tool creates all calib maps for that option and the complete overview, without further asking.

[ top ] Configuration files

The tool has its configuration file config.createCalibMap which comes in one version per rules file, and resides in the same path. For instance, the configuration file for the calChecker calibration map resides under $DFO_CONFIG_DIR/CALCHECK/OCA.

For the CALSELECTOR configuration, there is the need for versioning. This is solved by having one subdirectory for each version, with the start-date (first date of validity) being the label for the version and the subfolder: $DFO_CONFIG_DIR/CALSELECTOR/<start_date>. To manage the various versions, there is the configuration file config.createCalibMap.calsel in $DFO_CONFIG_DIR/CALSELECTOR.

config.createCalibMap config.createCalibMap config.createCalibMap.calsel
    <date1> (version) <date2> <date3>
    config.createCalibMap config.createCalibMap config.createCalibMap

The configuration file config.createCalibMap has the following structure:

1. General parameters


DFOS_OPS type of this cascade
KEEP_SHORT YES YES|NO; YES: tags will be shortened to the value of SHORT_CHAR
default: YES; if you mouse over, the full value will display
SHORT_CHAR 15 value to which tags will be shortened if $KEEP_SHORT=YES; default: 15

2. Sub-cascades
2.1 define here your sub-cascades per mode
one line per mode; the sequence is evaluated for the navigation bar

DET (=detector) is a reserved flag for calibrations assumed to be shared among all modes. (You could also include ACQ here.)
ALL_MODES etc.  
2.2 All RAW_TYPEs per mode
The ones listed here will appear on the output.
The ones not listed here will show up only for mode ALL.

one block per mode; list all raw_types which constitute the sub-cascade: ACQ, CAL, SCI

The configuration file config.createCalibMap.calsel has the following structure:

1. Versions for CALSELECTOR configuration

There is one row per version. A version is defined by the following keys:

START_DATE: start-dates (--> versions) of calSelector RLS files
COMMENT: description of this version (include in &&...&&)

This is a typical record:

START_DATE 2008-05-01 &&applicable after CCD upgrade&&

[ top ] Comment file

The comment file is optional. It is called info.createCalibMap (in the same path as the config file).

- are all optional
- come in flavours MODE_COMMENT for the full cascade or a mode, and RT_COMMENT for raw-type specific comments, displayed for that column only
- start with && and finish with && (one line only, && to be the last characters on the line)

1. MODE_COMMENT: their qualifier is either 'ALL' or the mode


Qualifier Content



ALL &&A comment applicable to the full cascade&&
MODE_COMMENT MEDUSA &&A Medusa commment&&

: their qualifier is the RAW_TYPE

&&HC calibs; used for cascade with old CCD&&
RT_COMMENT FFLAT &&normal fibre flats&&

[ top ] Operational hints.



createCalibMap supports sub-cascades. The concept of sub-cascades is not part of OCA but very useful for calib maps. It helps reducing complexity and makes dependencies much more obvious.

Configuration of raw types.


Further hints


Use the already existing calibration maps under for reference.

Concluding remarks

Yes, it's kind of complicated but certainly pays off.

Last update: April 26, 2021 by rhanusch