Common DFOS tools:

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:

- new optional key CHECK_SOF_EXIST (for PHOENIX)

createJob; AB monitor: getStatusAB;
batch queue system: CONDOR

While processAB calls the pipeline, processQC calls QC reports.


- enabled for OPSHUB environment

[ used databases ] databases
[ used dfos tools ] dfos tools esorex; ngasClient
[ output used by ] output used by pipeline products: to be QC-processed and certified
[ upload/download ] upload/download download fits files with ngasClient



enabled for parallel execution
enabled for condor execution

This tool is the standard interface between an AB and the pipeline/DRS. It extracts from the AB the information which is relevant for pipeline processing: the type of DRS (CON, CPL or INT), the recipe name, the recipe parameters, the set of input frames and associated calibration product names, the location and predicted names of products. This information is written into a set of frames file (sof). The DRS system is invoked to process the AB. Finally processAB checks for success/failure of the recipe, and (if configured) renames and moves the products from the workspace to the final certification area.

The tool checks first for availability of the required raw files and the calibration products, and downloads them if not found. Thereby a valid, historical AB stored in the DFOS system can be executed anytime later. Only exception is if the recipe, as coded in the OCA rules and listed in the AB, is "none" or "NONE": then data download is suppressed. If a download error occurs (e.g. because a file is not yet available in NGAS, or NGAS is temporarily unavailable), the tool reports this just like a processing error, i.e. the AB is flagged as 'FAILED'.

The tool supports the following types of DRS (see also the help page for createJob):

The DRS_TYPE is read from config.createAB.

How to avoid data downloads for unsupported modes. Per default, processAB always first downloads the raw and mcalib files listed in the SOF part if they are not already available in the system. Then it attempts pipeline processing. This is usually what you want, but in case of pipeline-unsupported modes, this does not make sense. If you use the reserved tags "none" or "NONE" as recipe name in the OCA association file, processAB interprets this as a signal to not download the files listed in the SOF part.

Other options. In addition to these standard features, the tool offers extended functionalities:

esorex parameter configuration file. In section 2, you can enter the name of an esorex pipeline configuration file. This is currently useful for the generic detmon pipeline.


To protect against processing "unusual" ABs having much more than the usual number of raw frames, thereby potentially slowing down the incremental processing system to an unacceptable degree, there is a parameter called MAXN_RAW which is by default 10. Any AB having more than MAXN_RAW input files is not processed and fires an email alert. Once you are out of the autoDaily regime, you can process that AB by calling the tool on the command line with option -F (force), see below. Or you can fine-tune the default, global MAXN_RAW for a specific RAW_TYPE to a more appropriate number, by using the section 3.3 in the configuration file.



processAB -a FORS1.2024-07-12T06:59:13.223_tpl.ab

How to use

Type processAB -h for on-line help, processAB -v for the version number,

processAB -x

for help about esorex and your pipeline (see below);

processAB -a FORS2.2024-07-12T06:59:13.223_tpl.ab

to process a particular AB;

processAB -a FORS2.2024-07-12T06:59:13.223_tpl.ab -D

to do the same in DEBUG mode (prepare the environment, but leave the execution to a manual call);

processAB -a HAWKI.2013-01-06T23:55:39.189_tpl.ab -F

force the execution of an AB which was refused by the automatic system (because of NRAW > NRAW_MAX).

Note that the tool logs per AB into <AB>.rblog which is created under $DFS_LOG. (The name rblog is used for historical reasons only, this was once the "reduction block log".)

For execution by condor (the standard way), the option -u <user> has to be added, like e.g. -u vircam, in order to make condor find the home directory and source the environment properly. This is done automatically by createJob.

For execution by DRS_TYPE=INT, the option -l (no argument) has to be added (l stands for LIKWID), to mark the AB as being executed by autoDaily. This is done automatically by createJob.

For execution by CONDOR in the OPSHUB environment, the option -i <instr> has tobe added. This is done automatically by createJob.

processAB and esorex

The tool calls the DFS tool esorex as the standard interface to the CPL-based pipelines:

esorex --time --output-dir=$WORK_AREA --suppress-prefix $ESOREX_PARAMS $RECIPE $PARAMS sof

No other esorex parameters are specified explicitly. If wanted, the user can configure those in $HOME/.esorex/esorex.rc (e.g. modify the log level). Also, the processAB config key NO_CHECKSUM has direct impact on the esorex parameters (see below).

For generic pipelines (e.g. the detmon pipeline), you can configure a set of parameters in a configuration file $HOME/.esorex/detmon_{ir|opt}_lg_{instr}.rc. You need then to define these in section 2 of the configuration file. These parameters are extracted by the tool and called as $ESOREX_PARAMS. If no such configuration file is defined in section 2, the variable $ESOREX_PARAMS remains empty.

For DRS_TYPE=INT, the esorex call is embedded in a call of the tool likwid-pin, to enable internal parallelization:

$LIKWID_PATH/likwid-pin -c $LIKWID_CALL esorex --time --output-dir=$WORK_AREA --suppress-prefix $ESOREX_PARAMS $RECIPE $PARAMS sof

The parameters $LIKWID_PATH and $LIKWID_CALL need to be configured in config.processAB.

For DRS_TYPE=INT, you may also need to define OMP_NUM_THREADS=24 in your .qcrc (corresponding to the value 0-23 for LIKWID_CALL).

Because of the close logical connection between processAB, esorex and the pipeline, the tool provides esorex help and pipeline recipe help, invoked as processAB -x.

DEBUG: The DEBUG mode can be invoked as -D. It sets up the $WORK_AREA, extracts the sof etc., so that you can start the esorex command from the command line. This may be useful if you want to check esorex parameters, products before renaming etc. Products will not be renamed or shifted. Don't forget to delete the $WORK_AREA after your tests.


Use dfosExplorer or dfosInstall.

Since the tool is designed for parallel processing of ABs, it uses $WORK_AREA/$AB as recipe work area. These subdirs are deleted after execution.

Configuration file

config.processAB defines:

Section 1: general
WORK_AREA /data28/visir/CONDOR_WORK root directory where pipeline recipes are executed; make sure it exists!
RENAME_SHIFT YES | NO enable/disable renaming and shifting of pipeline products; NO makes sense for debugging only.
ACCEPT_FZ YES | NO YES: fits file names are replaced by fits.fz in the sof; if found, fits.fz files are not downloaded by ngasClient; default: NO
# for DRS_TYPE=INT only:
LIKWID_PATH /opt/likwid/bin path to likwid tool (since this is not a standard dfs tool)
LIKWID_CALL 0-23 translated into '-c 0-23': split by 24 detectors
# for PHOENIX only:
DRS_FROM_AB YES|NO for PHOENIX only: read DRS_TYPE from the AB (instead of config.createAB); this to support special situation for MUSE IDPs that some ABs need a different DRS_TYPE than others; default: NO
NO_CHECKSUM YES|NO for PHOENIX only: adds --no-checksum and --no-datamd5 in esorex call; for better performance in well-defined exceptional cases (like MUSE IDPs); default: NO
CHECK_SOF_EXIST YES|NO for PHOENIX only: esorex checks all input files for existence before starting the recipe; default: NO
Section 2: esorex configuration file name(s) for generic pipeline(s)
List here the name(s) of esorex configuration file(s) of general pipeline(s) (e.g. detmon)
RAW_TYPE DETMON_FLAT usually just one, but could be more
DET_ID ANY required only if you need to specify config files per detector (avoid if possible), one line per detector;
if not needed, just enter ANY
  det1 etc.
CONFIG detmon_ir_lg_vircam.rc name of config file for generic pipeline, under $HOME/.esorex

Section 3: plugins
3.1 GENERAL plugins

There are three such plugins. They are optional and of type 'post-processing'.
COMMON_PLUGIN my_plugin optional post-processing plugin to be applied to all raw_types
ERROR_PLUGIN my_error_plugin optional post-processing plugin to be applied if a recipe fails
FINAL_PLUGIN &&my_final_plugin $AB&& optional plugin to be called at the end of RENAME_SHIFT, accepts $AB as argument, to be enclosed as &&...&&

3.2: PRE/POST plugin
names of optional plugins specific to a particular RAW_TYPE. They have to reside under $DFO_BIN_DIR; PRE and POST means before and after recipe execution. Syntax:

- RAW_TYPE: e.g. BIAS, as defined in the OCA files
- PRE_PLUGIN: name of plugin (can be NONE)
- POST_PLUGIN: name of plugin (can be NONE)

SPEC_PLUGIN DARK pgi_preDARK pgi_postDARK only one plugin per RAW_TYPE!

3.3: MAXN_RAW section
optional configuration for maximum number of accepted raw frames in the AB, this to override the tool default (10). Syntax:

- RAW_TYPE: e.g. BIAS, as defined in the OCA files
- MAXN_RAW: e.g. 15

Section 4: Define numbering scheme for products
That section defines the final product names. The pipelines deliver internal names. The DRS may or may not rename them into the PIPEFILE scheme. Hence it is better to have the final names created by the user. While the root name is defined in the AB, the index and the extension need to be defined here to make the final name complete. Index and extension are defined per PRO.CATG.

- RAW_TYPE is needed since some recipes may work on multiple RAW_TYPEs
- Recipe: pipeline recipe name
- internal: internal name of product
- index_ext: user-specified product index and extension
- PRO.CATG: pro.catg key
- RAW_TYPE: as read from the AB

PRO_CATG gimasterbias | master_bias.fits | 0000.fits | MASTER_BIAS | BIAS  

Status information

The tool writes status information into the AB (PROCESS_STATUS = SUCCESSFUL or FAILED), based on the availability of products and on the recipe return code. It also writes the total execution time in seconds into the AB (under TEXEC), to be evaluated later by the statistics tool extractStat. It writes the corresponding updates into the .tab file.

Operational aspects

Workflow description

1. Create a unique work directory as $WORK_AREA/$AB

2. Preparation

2.1 parse the AB for pipeline processing information (recipe, sof, parameters etc.); ACCEPT_FZ=YES: replace in the sof fits by fits.fz for those files which exist already as fits.fz files
2.2 check for MAXN_RAW (either as configured, or default); if actual number is higher, do not execute the AB but send an information mail to $OP_ADDRESS
2.3 read DRS_TYPE (from config.createAB
2.4 call PRE_PLUGIN
2.5 Check for existence of MCALIBs, download with ngasClient if necessary (downloaded mcalibs are listed in $DFO_MON_DIR/MCAL_DOWNLOAD, for later deletion with cleanupProducts); if download impossible, send out warning, set AB status to "FAILED", finish
2.6 Check for existence of target directory ($DFS_PRODUCT/$RAW_TYPE/$DATE); create if not existing

3. Launch recipe

4. Verify results

4.1 no products in $WORK_AREA: send out warning, set AB status to "FAILED", apply ERROR_PLUGIN, finish
4.2 products found: set AB status to "SUCCESSFUL"
4.3 insert the correct value for PIPEFILE

4.4 apply configured common plugin
4.5 apply POST_PLUGIN
4.6 move all products to final certification area, rename them to pipefile name (if RENAME_SHIFT is enabled)
4.7 if not: leave them in $WORK_AREA for inspection (note: this means keep the WORK_AREA subdirectories, which will accumulate quickly!)
4.8 update TEXEC, T_QCEXEC and AB_status in the AB
4.9 apply FINAL_PLUGIN.

Last update: April 26, 2021 by rhanusch