Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:
  EXPOSE_TO_CALSELECTOR = T
createAB documentation, ABbuilder documentation
filterRaw
  RAW_MINNUM: supported by filterRaw AB concepts

OCA configuration and syntax

Find the current OCA manual here (from 2013, does not contain some calSelector features like tryBetween).

OCA stands for Organisation, Classification, and Association. It is the classification and association framework used by various dfos tools (and also non-dfos tools like calSelector or the Data Transfer System).

Configuration. The association configuration (used by createAB, calChecker, and filterRaw) is somewhat more complex than for most other DFOS tools. All relevant configuration is kept under $DFO_CONFIG_DIR/OCA (where $DFO_CONFIG_DIR could be the classical one as defined in .dfosrc or a separate workspace like for calChecker or calSelector):

AND


OCA syntax. The OCA syntax is SQL-like and consists of a set of

OCA syntax is very flexible. For instance:

The same syntax is used for other applications like filterRaw or calChecker.

The main configuration files are <instr>_<oca>.h (where <oca> is short for organization, classification, association). The usage of macros is supported. These are defined in two macro files:

The macro files need to be defined (# include) in the <instr>_<oca>.h files. All <instr>_<oca>.h files need to be defined (# include) in a <instr>.h file. All these files are then pre-compiled using gcc into a single <instr>.RLS file which is dynamic and is read by ABbuilder. All these files have a special syntax:


<instr>_classification.h:

consists of multiple entries (one per DO.CLASS) like the following:

if DPR.CATG=="CALIB" and DPR.TYPE=="FLAT,LAMP" then
{
RAW.TYPE = "FLAT";
DO.CLASS = "FIBER_FLAT";
PACK.DIR = "FLAT";
CATG = "CALIB";

}

NOTE: do not forget the blanks before and after the equal sign within the curly brackets (red part). Although ABbuilder does not require that syntax, createAB expects it.

For exceptional cases, non-SCIENCE data can be 'upgraded' to become possible input data for calSelector. A typical example are standard stars which a user might want to enter on the calSelector interface in order to get all required clibration data for a follow-up data reduction. To mark those data, just enter EXPOSE_TO_CALSELECTOR=T (as a boolean, no quotes):

if DPR.CATG=="CALIB" and DPR.TYPE=="FLAT,LAMP" then
{
RAW.TYPE = "FLAT";
EXPOSE_TO_CALSELECTOR=T;

DO.CLASS = "FIBER_FLAT";
PACK.DIR = "FLAT";
CATG = "CALIB";

}

 


<instr>_organisation.h:

consists of multiple entries (one per ACTION) like the following:

select execute(ACTION_FLAT) from inputFiles where RAW.TYPE=="FLAT"
group by RAW_MATCH_RULE2,GRP_RULE_TPL_A

The match and grouping rules RAW_MATCH_RULE2,GRP_RULE_TPL_A can be macros which then need to be defined in <instr>_macro.h.

You can define multiple ACTIONs per RAW.TYPE in <instr>_organisation.h.

The where clause can contain complex expressions, like

where RAW.TYPE=="SCI_IMG" and TPL.NEXP>3

This may become useful when you want to trigger more than one action on a given RAW_TYPE, like in the following example:


select execute(ACTION_SCI_IMG) from inputFiles where RAW.TYPE=="SCI_IMG"
  group by IMA_MATCH_RULE,GRP_RULE_SINGLE;

select execute(ACTION_FLAT_NIGHT) from inputFiles where RAW.TYPE=="SCI_IMG" and TPL.NEXP>3
  group by FLAT_NIGHT_MATCH_RULE,GRP_RULE_TPL_A;

First, ACTION_SCI_IMG is triggered, applying the match rule IMA_MATCH_RULE and the grouping rule GRP_RULE_SINGLE on all input files of RAW.TYPE "SCI_IMG". Then, ACTION_FLAT_NIGHT is called on those SCI_IMG raw files which also satisfy the condition TPL.NEXP>3. These are then grouped by GRP_RULE_TPL_A.

RAW_MINNUM: there is an optional, non-OCA configuration possible in <instr>_organisation.h: a minimum number of input raw files for an AB to be built. This information is read by the filter tool filterRaw. There is one line per condition. The exact location within the OCA classification does not matter. Traditionally, it is found in the same place as the other configuration for <raw_type>:

//RAW_MINNUM <raw_type> <value>

This entry is interpreted by filterRaw to create ABs for <raw_type> only if at least <value> input raw files have been found, otherwise no AB is created.


<instr>_association.h:

consists of multiple entries (one per ACTION) like the following (the string ACTION_FLAT MASTER_BIAS DELTAT_RULE=2.5 is not part of OCA but is evaluated by createAB to establish the rule that for FLAT raw types a MASTER_BIAS is needed within a validity range of 2.5 days; the string FLAT needs to be identical to the RAW.TYPE in <instr>_classification.h and <instr>_organisation.h):

MCALIB MASSOC RASSOC

 

action ACTION_FLAT


//
ACTION_FLAT MASTER_BIAS DELTAT_RULE=2.5
{ minRet = 1; maxRet = 1;
select file as
MASTER_BIAS from calibFiles where
PRO.CATG=="
MASTER_BIAS"
and
MATCH_RULE1
and
NEXT
;


<more entries, one per mcalib>


recipe
masterflat {
"--stack-method=median"; **
}
product
MASTER_FLAT {PRO.CATG="MASTER_FLAT"; PRO.EXT="tpl_0002.fits"}
}

...
{ minRet =
0; maxRet = 1;
...
select file as
BP_MASK_MASSOC from calibFiles where
PRO.CATG=="
MASTER_BIAS"
and
MATCH_RULE3
and
ALL
;
...

...
{ minRet =
0; maxRet = 10;
...
select file as
FLAT_RASSOC from rawFiles* where
RAW.TYPE=="
FLAT"
and
MATCH_RULE1
and
NEXT
;
...

*: note to use rawFiles here instead of the usual inputFiles (see below for an explanation).
**: any content between "..." is interpreted as valid; you can also enter blanks etc.

MATCH_RULE1 and NEXT can be macros to be defined in <instr>_macro.h.

Note that lines starting with '// ACTION_' will be interpreted by tools like createAB. Any other comment is permitted.


Syntax rules

Possible operators in the condition section (if...then or where...) are:

== equal (UNDEFINED would not match)
?= possibly equal (UNDEFINED would match)
!= not equal
between used by calSelector to flag a match as 'extended validity'
tryBetween for calSelector: if found, the match gets flagged as 'calib_plan'
like pattern match (use % as wildcard)
+ - * / < <= > >= mathematical operators
and / or operators to connect conditions

Key arithmetics. With mathematical operators, you can do e.g. things like

if ... then {
  KEY = KEY + 1;
 }
and then use the modified KEY in a grouping rule.

Meta keywords. Another powerful mechanism is meta keywords. These are "artificial" keywords created and handled by OCA in the same way as fits header keywords. One example is MJD-OBS_MOD being created by ABbuilder and written into ABs and headers of virtual calibration files. Another way of using this mechanism is to define a meta key, e.g. by saying

if KEY >= 1 and KEY < 2 then M.KEY = 1;

and then use M.KEY within a grouping rule.

Data sources. Currently ABbuilder knows three data sources:

Step2 ABs. There are sometimes cases when a certain raw_type triggers a recipe, and the products of that recipe need to be processed in a second step, without having raw input files. The concept of step1-step2, or parent-child, ABs is supporting this scenario. In OCA, there is no specific syntax required to support this scenario.

Macros. A very useful feature is the availability of macros. Rather than having many matchkey definition lines, which repeat in many rule files, one can use one unique string and define it in the macro files once. There is the OCA_macro.h file which has the OCA system-wide definitions (e.g. grouping rules, time match rules), and the <instr>_macro.h collecting the match keys.

macro file type of macro cases examples
OCA_macro.h grouping rules SINGLE
TPL_A
TPL_D
TPL_I
ALL
COMB

TPL_START as (TPL_A,tpl):

defining key(s), name of grouping rule (written into AB), tag in PIPEFILE name

NOTE: these settings must not be changed unless authorized, to keep the naming scheme uniform. The PIPEFILE tag propagates into product names.

OCA_macro.h time match rules CLOSEST
NEXT
PREVIOUS
ATTACHED
ALL
MJD-OBS < inputFile.MJD-OBS
OCA_macro.h parameter match macro _m(A) --> inputFileA==A macro for 'same parameter value' rule
<instr>_macro.h short names for FITS keys e.g. SLIT_NAME for INS.SLIT.NAME NOTE: you need to define only the short names you are using, if any. You don't need to define all fits keys!
<instr>_macro.h raw match keys multiple entries separated by comma RAW_MATCH_RULE2 --> SLIT_NAME,EXP_MODE
<instr>_macro.h product match keys use _m(<short name>); use 'and' for multiple entries MATCH_RULE6 --> _m(SLIT_NAME) and _m(EXP_MODE)