Common DFOS tools:

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:

version 1.4: called with '<instr>' or '-i <instr>'

The preimg account is a frozen installation. Never update dfos tools on preimg@muc05 unless authorized!

How to login:
login as qc_shift_muc on muc02; then type 'preimg' and you are on preimg@muc05

how to maintain preImg monitor
topics: process description | tool description | output | README file | usage | installation | configuration files |


[ top ] Description of pre-imaging process

This tool is a workflow tool to support the automatic processing of pre-imaging (pre-IMG) data. It is a wrapper around some dfos tools from the daily workflow. The pre-IMG process is implemented for some instruments but not for all. Any account without a pre-IMG process can ignore this tool.

The pre-IMG process has the following components:
Step description controlled by monitor
Step 1 Paranal measures a set of raw files with DPR keys indicating that this is pre-IMG data (e.g. DPR.TECH = IMAGING,PRE). PSO  
Step 2 Data are transferred via the Data Transfer System into NGAS DHA  
Step 3 AOG discovers the new pre-IMG data in NGAS, retrieves and pushes them to the QC machine into $DFO_RAW_DIR/PREIMAGING, launches processPreImg there; waits for EXECUTED file AOG  
Step 4 processPreImg is executed, writes a file EXECUTED into $DFO_SCI_DIR/PREIMAGING (used as a flag to AOG) QC preimgMonitor
Step 5 the EXECUTED file is detected by the AOG process; all product data (fits files, logs, README, USERREPORT) are sent to AOG AOG  
Step 6 an ftp account is created, all data are offered there for a limited period, the PI is notified AOG  

In the following, step 4 is documented.

Expanded scheme for step 4:
pre-IMG data detected and delivered by AOG to $DFO_RAW_DIR/PREIMAGING; their script calls processPreImg

processPreImg ...

workflow step AB creation job creation job execution product distribution report creation
processPreImg calls ... createAB -m SCIENCE -d 2099-99-99 createJob -m SCIENCE -d 2099-99-99 execAB_SCIENCE_2099-99-99 moveProduct -m SCIENCE -d 2099-99-99 [OB report, frame report; list_mcalib_DATE]
calling level 2 tools ...
processAB -a <ab> ...
renameProducts -m SCIENCE -d 2099-99-99
updateAB -m SCIENCE -d 2099-99-99

Note: AOG delivers raw files into $DFO_RAW_DIR/PREIMAGING, and expects products under $DFO_SCI_DIR/PREIMAGING. Because ABbuilder does not understand 'PREIMAGING', this tag is renamed to '2099-99-99' for the internal dfos part of the processing. At the end, the product directory is renamed to $DFO_SCI_DIR/PREIMAGING, as expected by the AOG script.

[ top ] Description of tool

The tool scans the raw data directory $DFO_RAW_DIR/PREIMAGING. If there are data, it calls:

Log files and a report file are created; the report is also mailed and copied to qc@stargate1 where it becomes visible under the URL<instr>. Under the same URL, there is also a preimgMonitor.html file which tabulates the most recent executions of the tool.

It is assumed that after execution the product data are collected by an AOG process (step 5).

By using dfos tools, processPreImg offers the same kind of functionalities as these tools. It is built around ABs and supports all kinds of grouping, time match, and association rules available for these (no matter whether there is pipeline support or not). At the end, it always creates a zero-size file EXECUTED under $DFO_SCI_DIR/PREIMAGING which is interpreted by the AOG master process as end of processPreImg.

[ top ] Output

[ top ] User README file

There is a README file which details some general aspects of the pre-imaging process and some instrument specific aspects. This README file must be provided to each package. It is maintained by the QC scientist under<instr>/<instr>_PRE_README.txt . The file is downloaded by the tool and copied to $DFO_LOG_DIR, from where it is included by the AOG process in the package.

[ top ] How to use

Type $DFO_BIN_DIR/processPreImg -h for on-line help, and $DFO_BIN_DIR/processPreImg -v for the version number. You can test the tool anytime (with dummy raw data in $DFO_RAW_DIR/<instr>):

processPreImg -i <instr> OR processPreImg <instr> (as done by the AOG script)

Caution: You are not supposed to run the tool operationally. It is launched by an AOG process. You can of course test it but make sure it does not collide with an external call. This is also true if you want to update it. Best is to check with Any successful tool call will produce an entry in the<instr>/preimgMonitor.html monitor file. Check here how to maintain it.

[ top ] Installation

Do not use dfosExplorer or dfosInstall. Download the tar file from the main tool page into your $HOME directory. Untar it and follow the instructions in the README file.

The process assumes that all raw data are delivered into $DFO_RAW_DIR/PREIMAGING which needs to be created by hand if not pre-existing. It is then renamed to '2099-99-99' by the tool. You will also need $DFO_LOG_DIR/PREIMAGING, $DFO_SCI_DIR/PREIMAGING, and $DFO_CAL_DIR/2099-99-99 and $DFO_CAL_DIR/gen.


If processing of pre-IMG SCIENCE data requires product calibration data, you need to feed them into your pre-defined $DFO_CAL_DIR (the one defined in.dfosrc_<instr>, not the standard $DFO_CAL_DIR!). A complete selection of dynamic calibration files goes to $DFO_CAL_DIR/2099-99-99. 'Complete' means covering all setups relevant for pre-imaging. One representative file per setup and pro.catg is sufficient since the pre-imaging process is designed to work with non-dynamic calibrations (like the Paranal quick-look processing). In that sense this directory also becomes static, although you may want to update it from time to time. The static calibrations go to $DFO_CAL_DIR/gen.

The MCAL directory managed by createAB will be created under $DFO_CAL_DIR. It will contain the calibrations under '2099-99-99' and 'gen' only. The VCAL directory will be empty.

The OCA configuration, including config.createAB, goes to the tool-specific $DFO_CONFIG_DIR/OCA directory. It needs only the part dealing with SCIENCE data.


This is required to modify some standard dfos variables to their PREIMG equivalents. There is one such file for each supported instrument. We assume that processPreImg will be started by a cronjob. Therefore it explicitly sources .dfosrc first, then .dfosrc_<instr>.

[ top ] Configuration files

The configuration files for the dfos tools involved need to be copied into $HOME/<instr>/config and edited, to support automatic processing:

config.createAB with the set of OCA config files goes to $HOME/<instr>/config/OCA.

Within processPreImg, the automatic tool mode of createAB is enforced. Edit config.createAB to enforce DRS_TYPE=CPL.

fitsreport_preimg.cfg is a fitsreport config file specific for the preimg product directory.

All other OCA configuration files can just be linked by symbolic links to the dfos-operational set, thereby enforcing consistency. You would then maintain only that standard set.

Here is a complete list of required config files and their need to be edited (if any):

config file keys or sections to edit
config.createAB (under $HOME/<instr>/config/OCA): CONF_VERSION; DRS_TYPE (must be CPL)
all OCA files (under $HOME/<instr>/config/OCA): have symbolic links to standard dfos versions, or create copies of dfos versions. You may need to edit them to include pre_IMG configuration.
config.moveProducts TOOL_MODE=AUTO; CDB_INGEST (SCIENCE)=NO; here you may want to fine-tune your delivery pattern, e.g. suppress some products for pre-IMG delivery which in standard dfos would go to $DFO_SCI_DIR.
config.processAB WORK_AREA=<different from dfos operational work area>; START_MONITOR_PLUGIN and END_MONITOR_PLUGIN: empty, or like in the delivered template); all other configuration can be left, read and evaluated is only the SCIENCE part
fitsreport_preimg.cfg new, does not exist for standard processing. Should be a copy of fitsreport_sciProc.cfg .

After the tar file has been unpacked, all the above configuration files are found in $HOME/preimg/config/template and .../template/OCA. (The OCA set is not complete, here you need to copy or link your operational instrument-specific files). As usual, these templates can be used for adaption, but it may be wiser to understand first what has been modified or removed in comparison to the standard config files, and then do this with your own versions under $DFO_CONFIG_DIR.

The tool confguration file (config.processPreImg) also goes to $HOME/config/ (!). It defines:

Section 1: general
PIPE_SUPPORT YES | NO controls the call of createJob and processAB; if NO, these are not called at all.
EMAIL_NOTE YES | NO enable/disable email notification to $OP_ADDRESS
QC_SERVER stargate1 server for internal web site
README_NAME PRE_README.txt rootname of README file to be given to user (to be maintained under<instr> !)

Section 2: Required dfos tools
This is section is used for an internal consistency check (processPreImg cannot do much for you if any of these is missing).

TOOLS createAB


Workflow description

processPreImg -i <instr>:

0. check for, and source, .dfosrc and .dfosrc_<instr>; check for required tools

1. mv $DFO_RAW_DIR/PREIMAGING --> $DFO_RAW_DIR/2099-99-99

2. call createAB -m SCIENCE -d 2099-99-99 -a; ABs go to $DFO_AB_DIR, job file created execAB_SCIENCE_2099-99-99

3. createJob -m SCIENCE -d 2099-99-99 (if PIPE_SUPPORT = YES)

4. call $DFO_JOB_DIR/execAB_SCIENCE_2099-99-99 (using processAB)

4.1 products go to $DFS_PRODUCT/<RAW_TYPE>/2099-99-99
4.2 rblogs go to $DFS_LOG_DIR

5. call moveProducts -m SCIENCE -d 2099-99-99

5.1 products go to $DFO_SCI_DIR/2099-99-99
5.2 ABs and rblogs go to $DFO_LOG_DIR/2099-99-99

6. create a report REPORT_<instr>_<date>_<runid>.log, to go to <instr>/LOGS

7. create OB and frame reports (using fitsreport) and merge into USERREPORT_<instr>_<date>_<runid>.log, to go to <instr>/LOGS

8. create list_mcalib_DATE (all mcalib files with pro.catg and date information)

9. update the preimgMonitor.html page (wget, edit, scp)

10. scp the REPORT to $QC_SERVER

11. write EXECUTED into $DFO_SCI_DIR/2099-99-99 and rename to $DFO_SCI_DIR/PREIMAGING

How to maintain the preImg monitor site

Under<instr>, you will find:

If you do a test run, you may want to clean up the directory and the html monitor file. This needs to be done manually:

If you delete the entire preimgMonitor.html file, it will be created from scratch the next time processPreImg is run.

Last update: April 26, 2021 by rhanusch