|dfos = Data Flow Operations System, the common tool set for DFO|
|make printable||new:||see also:|
version 1.4: called with '<instr>' or '-i <instr>'
The preimg account is a frozen installation. Never update dfos tools on preimg@muc05 unless authorized!
How to login:
|how to maintain preImg monitor|
|topics: process description | tool description | output | README file | usage | installation | configuration files ||
This tool is a workflow tool to support the automatic processing of pre-imaging (pre-IMG) data. It is a wrapper around some dfos tools from the daily workflow. The pre-IMG process is implemented for some instruments but not for all. Any account without a pre-IMG process can ignore this tool.
The pre-IMG process has the following components:
|Step 1||Paranal measures a set of raw files with DPR keys indicating that this is pre-IMG data (e.g. DPR.TECH = IMAGING,PRE).||PSO|
|Step 2||Data are transferred via the Data Transfer System into NGAS||DHA|
|Step 3||AOG discovers the new pre-IMG data in NGAS, retrieves and pushes them to the QC machine into $DFO_RAW_DIR/PREIMAGING, launches processPreImg there; waits for EXECUTED file||AOG|
|Step 4||processPreImg is executed, writes a file EXECUTED into $DFO_SCI_DIR/PREIMAGING (used as a flag to AOG)||QC||preimgMonitor|
|Step 5||the EXECUTED file is detected by the AOG process; all product data (fits files, logs, README, USERREPORT) are sent to AOG||AOG|
|Step 6||an ftp account is created, all data are offered there for a limited period, the PI is notified||AOG|
In the following, step 4 is documented.
Expanded scheme for step 4:
|pre-IMG data detected and delivered by AOG to $DFO_RAW_DIR/PREIMAGING; their script calls processPreImg||
|workflow step||AB creation||job creation||job execution||product distribution||report creation|
|processPreImg calls ...||createAB -m SCIENCE -d 2099-99-99||createJob -m SCIENCE -d 2099-99-99||execAB_SCIENCE_2099-99-99||moveProduct -m SCIENCE -d 2099-99-99||[OB report, frame report; list_mcalib_DATE]|
|calling level 2 tools ...||
processAB -a <ab> ...
renameProducts -m SCIENCE -d 2099-99-99
updateAB -m SCIENCE -d 2099-99-99
Note: AOG delivers raw files into $DFO_RAW_DIR/PREIMAGING, and expects products under $DFO_SCI_DIR/PREIMAGING. Because ABbuilder does not understand 'PREIMAGING', this tag is renamed to '2099-99-99' for the internal dfos part of the processing. At the end, the product directory is renamed to $DFO_SCI_DIR/PREIMAGING, as expected by the AOG script.
Description of tool
The tool scans the raw data directory $DFO_RAW_DIR/PREIMAGING. If there are data, it calls:
Log files and a report file are created; the report is also mailed and copied to qc@stargate1 where it becomes visible under the URL http://www.eso.org/~qc/monitor/PREIMG/<instr>. Under the same URL, there is also a preimgMonitor.html file which tabulates the most recent executions of the tool.
It is assumed that after execution the product data are collected by an AOG process (step 5).
By using dfos tools, processPreImg offers the same kind of functionalities as these tools. It is built around ABs and supports all kinds of grouping, time match, and association rules available for these (no matter whether there is pipeline support or not). At the end, it always creates a zero-size file EXECUTED under $DFO_SCI_DIR/PREIMAGING which is interpreted by the AOG master process as end of processPreImg.
User README file
There is a README file which details some general aspects of the pre-imaging process and some instrument specific aspects. This README file must be provided to each package. It is maintained by the QC scientist under http://www.eso.org/~qc/monitor/PREIMG/<instr>/<instr>_PRE_README.txt . The file is downloaded by the tool and copied to $DFO_LOG_DIR, from where it is included by the AOG process in the package.
Type $DFO_BIN_DIR/processPreImg -h for on-line help, and $DFO_BIN_DIR/processPreImg -v for the version number. You can test the tool anytime (with dummy raw data in $DFO_RAW_DIR/<instr>):
processPreImg -i <instr> OR processPreImg <instr> (as done by the AOG script)
Caution: You are not supposed to run the tool operationally. It is launched by an AOG process. You can of course test it but make sure it does not collide with an external call. This is also true if you want to update it. Best is to check with firstname.lastname@example.org. Any successful tool call will produce an entry in the http://www.eso.org/~qc/monitor/PREIMG/<instr>/preimgMonitor.html monitor file. Check here how to maintain it.
Do not use dfosExplorer or dfosInstall. Download the tar file from the main tool page into your $HOME directory. Untar it and follow the instructions in the README file.
The process assumes that all raw data are delivered into $DFO_RAW_DIR/PREIMAGING which needs to be created by hand if not pre-existing. It is then renamed to '2099-99-99' by the tool. You will also need $DFO_LOG_DIR/PREIMAGING, $DFO_SCI_DIR/PREIMAGING, and $DFO_CAL_DIR/2099-99-99 and $DFO_CAL_DIR/gen.
If processing of pre-IMG SCIENCE data requires product calibration data, you need to feed them into your pre-defined $DFO_CAL_DIR (the one defined in.dfosrc_<instr>, not the standard $DFO_CAL_DIR!). A complete selection of dynamic calibration files goes to $DFO_CAL_DIR/2099-99-99. 'Complete' means covering all setups relevant for pre-imaging. One representative file per setup and pro.catg is sufficient since the pre-imaging process is designed to work with non-dynamic calibrations (like the Paranal quick-look processing). In that sense this directory also becomes static, although you may want to update it from time to time. The static calibrations go to $DFO_CAL_DIR/gen.
The MCAL directory managed by createAB will be created under $DFO_CAL_DIR. It will contain the calibrations under '2099-99-99' and 'gen' only. The VCAL directory will be empty.
The OCA configuration, including config.createAB, goes to the tool-specific $DFO_CONFIG_DIR/OCA directory. It needs only the part dealing with SCIENCE data.
This is required to modify some standard dfos variables to their PREIMG equivalents. There is one such file for each supported instrument. We assume that processPreImg will be started by a cronjob. Therefore it explicitly sources .dfosrc first, then .dfosrc_<instr>.
The configuration files for the dfos tools involved need to be copied into $HOME/<instr>/config and edited, to support automatic processing:
config.createAB with the set of OCA config files goes to $HOME/<instr>/config/OCA.
Within processPreImg, the automatic tool mode of createAB is enforced. Edit config.createAB to enforce DRS_TYPE=CPL.
fitsreport_preimg.cfg is a fitsreport config file specific for the preimg product directory.
All other OCA configuration files can just be linked by symbolic links to the dfos-operational set, thereby enforcing consistency. You would then maintain only that standard set.
Here is a complete list of required config files and their need to be edited (if any):
|config file||keys or sections to edit|
|config.createAB||(under $HOME/<instr>/config/OCA): CONF_VERSION; DRS_TYPE (must be CPL)|
|all OCA files||(under $HOME/<instr>/config/OCA): have symbolic links to standard dfos versions, or create copies of dfos versions. You may need to edit them to include pre_IMG configuration.|
|config.moveProducts||TOOL_MODE=AUTO; CDB_INGEST (SCIENCE)=NO; here you may want to fine-tune your delivery pattern, e.g. suppress some products for pre-IMG delivery which in standard dfos would go to $DFO_SCI_DIR.|
|config.processAB||WORK_AREA=<different from dfos operational work area>; START_MONITOR_PLUGIN and END_MONITOR_PLUGIN: empty, or like in the delivered template); all other configuration can be left, read and evaluated is only the SCIENCE part|
|fitsreport_preimg.cfg||new, does not exist for standard processing. Should be a copy of fitsreport_sciProc.cfg .|
After the tar file has been unpacked, all the above configuration files are found in $HOME/preimg/config/template and .../template/OCA. (The OCA set is not complete, here you need to copy or link your operational instrument-specific files). As usual, these templates can be used for adaption, but it may be wiser to understand first what has been modified or removed in comparison to the standard config files, and then do this with your own versions under $DFO_CONFIG_DIR.
The tool confguration file (config.processPreImg) also goes to $HOME/config/ (!). It defines:
|Section 1: general|
|PIPE_SUPPORT||YES | NO||controls the call of createJob and processAB; if NO, these are not called at all.|
|EMAIL_NOTE||YES | NO||enable/disable email notification to $OP_ADDRESS|
|QC_SERVER||stargate1||server for internal web site|
|README_NAME||PRE_README.txt||rootname of README file to be given to user (to be maintained under http://www.eso.org/~qc/monitor/PREIMG/<instr> !)|
Section 2: Required dfos tools
processPreImg -i <instr>:
0. check for, and source, .dfosrc and .dfosrc_<instr>; check for required tools
1. mv $DFO_RAW_DIR/PREIMAGING --> $DFO_RAW_DIR/2099-99-99
2. call createAB -m SCIENCE -d 2099-99-99 -a; ABs go to $DFO_AB_DIR, job file created execAB_SCIENCE_2099-99-99
3. createJob -m SCIENCE -d 2099-99-99 (if PIPE_SUPPORT = YES)
4. call $DFO_JOB_DIR/execAB_SCIENCE_2099-99-99 (using processAB)
4.1 products go to $DFS_PRODUCT/<RAW_TYPE>/2099-99-99
4.2 rblogs go to $DFS_LOG_DIR
5. call moveProducts -m SCIENCE -d 2099-99-99
5.1 products go to $DFO_SCI_DIR/2099-99-99
5.2 ABs and rblogs go to $DFO_LOG_DIR/2099-99-99
6. create a report REPORT_<instr>_<date>_<runid>.log, to go to <instr>/LOGS
7. create OB and frame reports (using fitsreport) and merge into USERREPORT_<instr>_<date>_<runid>.log, to go to <instr>/LOGS
8. create list_mcalib_DATE (all mcalib files with pro.catg and date information)
9. update the preimgMonitor.html page (wget, edit, scp)
10. scp the REPORT to $QC_SERVER
11. write EXECUTED into $DFO_SCI_DIR/2099-99-99 and rename to $DFO_SCI_DIR/PREIMAGING
How to maintain the preImg monitor site
Under http://www.eso.org/~qc/monitor/PREIMG/<instr>, you will find:
If you do a test run, you may want to clean up the directory and the html monitor file. This needs to be done manually:
If you delete the entire preimgMonitor.html file, it will be created from scratch the next time processPreImg is run.
|Last update: April 26, 2021 by rhanusch|