make printable	new:	see also:
	version 1.4: called with '<instr>' or '-i <instr>' The preimg account is a frozen installation. Never update dfos tools on preimg@muc05 unless authorized! How to login: login as qc_shift_muc on muc02; then type 'preimg' and you are on preimg@muc05	how to maintain preImg monitor
topics: process description | tool description | output | README file | usage | installation | configuration files |

processPreImg

Description of pre-imaging process

This tool is a workflow tool to support the automatic processing of pre-imaging (pre-IMG) data. It is a wrapper around some dfos tools from the daily workflow. The pre-IMG process is implemented for some instruments but not for all. Any account without a pre-IMG process can ignore this tool.

The pre-IMG process has the following components:

Step	description	controlled by	monitor
Step 1	Paranal measures a set of raw files with DPR keys indicating that this is pre-IMG data (e.g. DPR.TECH = IMAGING,PRE).	PSO
Step 2	Data are transferred via the Data Transfer System into NGAS	DHA
Step 3	AOG discovers the new pre-IMG data in NGAS, retrieves and pushes them to the QC machine into $DFO_RAW_DIR/PREIMAGING, launches processPreImg there; waits for EXECUTED file	AOG
Step 4	processPreImg is executed, writes a file EXECUTED into $DFO_SCI_DIR/PREIMAGING (used as a flag to AOG)	QC	preimgMonitor
Step 5	the EXECUTED file is detected by the AOG process; all product data (fits files, logs, README, USERREPORT) are sent to AOG	AOG
Step 6	an ftp account is created, all data are offered there for a limited period, the PI is notified	AOG

In the following, step 4 is documented.

Expanded scheme for step 4:

pre-IMG data detected and delivered by AOG to $DFO_RAW_DIR/PREIMAGING; their script calls processPreImg	processPreImg ...
workflow step	AB creation	job creation	job execution	product distribution		report creation
processPreImg calls ...	createAB -m SCIENCE -d 2099-99-99	createJob -m SCIENCE -d 2099-99-99	execAB_SCIENCE_2099-99-99	moveProduct -m SCIENCE -d 2099-99-99		[OB report, frame report; list_mcalib_DATE]
calling level 2 tools ...	ABbuilder		processAB -a <ab> ...	renameProducts -m SCIENCE -d 2099-99-99	updateAB -m SCIENCE -d 2099-99-99	fitsreport

Note: AOG delivers raw files into $DFO_RAW_DIR/PREIMAGING, and expects products under $DFO_SCI_DIR/PREIMAGING. Because ABbuilder does not understand 'PREIMAGING', this tag is renamed to '2099-99-99' for the internal dfos part of the processing. At the end, the product directory is renamed to $DFO_SCI_DIR/PREIMAGING, as expected by the AOG script.

Description of tool

The tool scans the raw data directory $DFO_RAW_DIR/PREIMAGING. If there are data, it calls:

• createAB -m SCIENCE -d 2099-99-99
• createJob -m SCIENCE -d 2099-99-99 (with configuration taken from $DFO_CONFIG_DIR)
• $DFO_JOB_DIR/execAB_SCIENCE_2099-99-99, calling processAB for each AB created
• moveProducts -m SCIENCE -d 2099-99-99

Log files and a report file are created; the report is also mailed and copied to qc@stargate1 where it becomes visible under the URL http://www.eso.org/~qc/monitor/PREIMG/<instr>. Under the same URL, there is also a preimgMonitor.html file which tabulates the most recent executions of the tool.

It is assumed that after execution the product data are collected by an AOG process (step 5).

By using dfos tools, processPreImg offers the same kind of functionalities as these tools. It is built around ABs and supports all kinds of grouping, time match, and association rules available for these (no matter whether there is pipeline support or not). At the end, it always creates a zero-size file EXECUTED under $DFO_SCI_DIR/PREIMAGING which is interpreted by the AOG master process as end of processPreImg.

Output

• pipeline-processed files (renamed) under $DFO_SCI_DIR/PREIMAGING (if the AB execution was successful)
• log files and ABs under $DFO_LOG_DIR
• process log files under $HOME/<instr>/LOGS:
  • PREIMG_<instr>_<date>_<runid>.log containing the main process log
  • DETAILS_<instr>_<date>_<runid>.log containing the more detailed logs (incl. the pipeline processing logs)
  • REPORT_<instr>_<date>_<runid>.log containing a summary report (also copied to qc@stargate1)
  • USERREPORT_<instr>_<runid>.txt containing the OB report and the frame report (also copied to qc@stargate1)
  • a general README file created and maintained by the QC scientist (<instr>_PRE_README.txt)
• an updated entry in http://www.eso.org/~qc/monitor/PREIMG/<instr>/preimgMonitor.html.

User README file

There is a README file which details some general aspects of the pre-imaging process and some instrument specific aspects. This README file must be provided to each package. It is maintained by the QC scientist under http://www.eso.org/~qc/monitor/PREIMG/<instr>/<instr>_PRE_README.txt . The file is downloaded by the tool and copied to $DFO_LOG_DIR, from where it is included by the AOG process in the package.

How to use

Type $DFO_BIN_DIR/processPreImg -h for on-line help, and $DFO_BIN_DIR/processPreImg -v for the version number. You can test the tool anytime (with dummy raw data in $DFO_RAW_DIR/<instr>):

processPreImg -i <instr> OR processPreImg <instr> (as done by the AOG script)

Caution: You are not supposed to run the tool operationally. It is launched by an AOG process. You can of course test it but make sure it does not collide with an external call. This is also true if you want to update it. Best is to check with aog@eso.org. Any successful tool call will produce an entry in the http://www.eso.org/~qc/monitor/PREIMG/<instr>/preimgMonitor.html monitor file. Check here how to maintain it.

Installation

Do not use dfosExplorer or dfosInstall. Download the tar file from the main tool page into your $HOME directory. Untar it and follow the instructions in the README file.

The process assumes that all raw data are delivered into $DFO_RAW_DIR/PREIMAGING which needs to be created by hand if not pre-existing. It is then renamed to '2099-99-99' by the tool. You will also need $DFO_LOG_DIR/PREIMAGING, $DFO_SCI_DIR/PREIMAGING, and $DFO_CAL_DIR/2099-99-99 and $DFO_CAL_DIR/gen.

Preparation

If processing of pre-IMG SCIENCE data requires product calibration data, you need to feed them into your pre-defined $DFO_CAL_DIR (the one defined in.dfosrc_<instr>, not the standard $DFO_CAL_DIR!). A complete selection of dynamic calibration files goes to $DFO_CAL_DIR/2099-99-99. 'Complete' means covering all setups relevant for pre-imaging. One representative file per setup and pro.catg is sufficient since the pre-imaging process is designed to work with non-dynamic calibrations (like the Paranal quick-look processing). In that sense this directory also becomes static, although you may want to update it from time to time. The static calibrations go to $DFO_CAL_DIR/gen.

The MCAL directory managed by createAB will be created under $DFO_CAL_DIR. It will contain the calibrations under '2099-99-99' and 'gen' only. The VCAL directory will be empty.

The OCA configuration, including config.createAB, goes to the tool-specific $DFO_CONFIG_DIR/OCA directory. It needs only the part dealing with SCIENCE data.

.dfosrc_<instr>

This is required to modify some standard dfos variables to their PREIMG equivalents. There is one such file for each supported instrument. We assume that processPreImg will be started by a cronjob. Therefore it explicitly sources .dfosrc first, then .dfosrc_<instr>.

Configuration files

The configuration files for the dfos tools involved need to be copied into $HOME/<instr>/config and edited, to support automatic processing:

• OCA/config.createAB (note: FILE_MODE must be FITS; FILTER_RAW recommended NO)
• config.moveProducts
• config.processAB

config.createAB with the set of OCA config files goes to $HOME/<instr>/config/OCA.

Within processPreImg, the automatic tool mode of createAB is enforced. Edit config.createAB to enforce DRS_TYPE=CPL.

fitsreport_preimg.cfg is a fitsreport config file specific for the preimg product directory.

All other OCA configuration files can just be linked by symbolic links to the dfos-operational set, thereby enforcing consistency. You would then maintain only that standard set.

Here is a complete list of required config files and their need to be edited (if any):

After the tar file has been unpacked, all the above configuration files are found in $HOME/preimg/config/template and .../template/OCA. (The OCA set is not complete, here you need to copy or link your operational instrument-specific files). As usual, these templates can be used for adaption, but it may be wiser to understand first what has been modified or removed in comparison to the standard config files, and then do this with your own versions under $DFO_CONFIG_DIR.

The tool confguration file (config.processPreImg) also goes to $HOME/config/ (!). It defines:

Workflow description

processPreImg -i <instr>:

0. check for, and source, .dfosrc and .dfosrc_<instr>; check for required tools

1. mv $DFO_RAW_DIR/PREIMAGING --> $DFO_RAW_DIR/2099-99-99

2. call createAB -m SCIENCE -d 2099-99-99 -a; ABs go to $DFO_AB_DIR, job file created execAB_SCIENCE_2099-99-99

3. createJob -m SCIENCE -d 2099-99-99 (if PIPE_SUPPORT = YES)

4. call $DFO_JOB_DIR/execAB_SCIENCE_2099-99-99 (using processAB)

4.1 products go to $DFS_PRODUCT/<RAW_TYPE>/2099-99-99
4.2 rblogs go to $DFS_LOG_DIR

5. call moveProducts -m SCIENCE -d 2099-99-99

5.1 products go to $DFO_SCI_DIR/2099-99-99
5.2 ABs and rblogs go to $DFO_LOG_DIR/2099-99-99

6. create a report REPORT_<instr>_<date>_<runid>.log, to go to <instr>/LOGS

7. create OB and frame reports (using fitsreport) and merge into USERREPORT_<instr>_<date>_<runid>.log, to go to <instr>/LOGS

8. create list_mcalib_DATE (all mcalib files with pro.catg and date information)

9. update the preimgMonitor.html page (wget, edit, scp)

10. scp the REPORT to $QC_SERVER

11. write EXECUTED into $DFO_SCI_DIR/2099-99-99 and rename to $DFO_SCI_DIR/PREIMAGING

How to maintain the preImg monitor site

Under http://www.eso.org/~qc/monitor/PREIMG/<instr>, you will find:

• processing reports (1 per call)
• the user reports (1 per call)
• preimgMonitor.html (one line added per call).

If you do a test run, you may want to clean up the directory and the html monitor file. This needs to be done manually:

• login as 'qc' to the internal web site
• go to the preimg web site
• delete the corresponding reports
• open preimgMonitor.html with an editor and remove the corresponding line

If you delete the entire preimgMonitor.html file, it will be created from scratch the next time processPreImg is run.