Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:
  v2.1:
- supports DFOS and PHOENIX, no functional change

off-line calls of ingestProducts are collected in JOBS_INGEST and managed by dfoMonitor

 

v2.2:
- VCAL mechanism simpler and safer; VCAL_REMOVE obsolete


[ used databases ] databases none
[ used dfos tools ] dfos tools ingestProducts, renameProducts, updateAB
[ output used by ] output distributed/updated files; VCAL directory managed
[ upload/download ] upload/download export to qcweb: all files (ex hdr/fits) in $DFO_LOG_DIR/<date>; all files (ex hdr/fits, ex ps) in $DFO_PLT_DIR/<date>

moveProducts

Description

This tool is called to distribute, after certification, all or selected pipeline products and graphical information to the final directories. The final directories on the DFOS systems are: $DFO_CAL_DIR/$DATE for calib data, $DFO_SCI_DIR/$DATE for science data, $DFO_PLT_DIR/$DATE for graphical information, and $DFO_LOG_DIR/$DATA for pipeline logs, association logs, trending logs, and ABs. Some data are also exported to the QC server and to the ESO web server.

The tool also calls the DFOS tools renameProducts and updateAB. Calls for product ingestion are written into the JOBS_INGEST file under $DFO_JOB_DIR. The tool also manages the content of $DFO_CAL_DIR/VCAL (the virtual calibrations container), i.e. removes those virtual calibs which became certified (REAL) in the end. All logs and plots (except postscript files) are exported to the QC web server qcweb for on-line reference.

An optional, user-provided plugin can be used e.g. to store certain products in a temporary workspace for detailed analysis. The package has an example plugin.

Find its place in the daily workflow here.

Output

Updated ABs.

How to use

Type moveProducts -h for on-line help, moveProducts -v for the version number, and

moveProducts -m CALIB -d 2014-07-12

to move all pipeline products for mode CALIB on that date to their final place (if they are configured to be moved).

Installation

Use dfosExplorer or dfosInstall . There is an optional plugin, pgi_moveProducts, which remains in $DFO_INSTALL_DIR/moveProducts. Check its function and content and decide if you want to use it.

Configuration file

config.moveProducts defines:

Section 1: general
TOOL_MODE INTER execute mode for tool: INTER or AUTO; INTER for interactive execution
SPECIAL_PLUGIN pgi_moveProducts optional plugin for special tasks (can be NONE)

Section 2: Supported RAW_TYPEs
directories under $DFS_PRODUCT; they are defined per MODE (CALIB or SCIENCE).
- SAVE_MODE describes whether ALL products of that RAW_TYPE should be saved, or NONE, or SELECTed only.
SELECT joins to DELETE table.
Strategy to save files:

  • all products to be saved which are needed for science reduction,
  • and/or input for other mcalib files; others can be safely removed; all saved ones go into the mcalib archive eventually
RAW_TYPE BIAS | CALIB | ALL  
RAW_TYPE FFLAT | CALIB | SELECT if there is a SELECT entry, it joins to the DELETE entries.
Section 3: Selected product types *NOT* to be saved in DFO_CAL_DIR or DFO_SCI_DIR
NOTE: if SAVE_MODE = NONE, all products are deleted and need not be listed here. All PRO_CATGs listed here are deleted.
DELETE FFLAT | FF_EXTPIXELS only EXTPIXELS products are deleted, all others are saved

Section 4: treatment of additional info (graphics)
Here you configure whether additional information (the stuff listed under FURTHER in the AB) should be saved. This could be gif, png or jpg files. FORMAT specifies the extension, SAVE_FLAG is YES or NO.

4.1 Graphical information

PLOT GIF YES
PNG YES
JPG YES
Supported formats: gif/png/jpg. They go to $DFO_PLT_DIR and are exported to qcweb. 'ps' files are not supported.

Section 5: Product renaming
Switch on/off product renaming (done by renameProducts) per MODE (CALIB and SCIENCE)

RENAME CALIB | YES rename CALIB products
RENAME SCIENCE | YES rename SCIENCE products

Status information

The tool writes the DFO status cal_Finished and sci_Finished.

Operational aspects

Workflow description

1. read list of RAW_TYPEs for selected mode in config file

2. loop on RAW_TYPEs:

2.1 go to $DFS_PRODUCT/<RAW_TYPE>/<DATE>

2.2 Check for existence of $DFO_CAL_DIR/<DATE> or $DFO_SCI_DIR/<DATE>. If it already exists, ask the user (in mode INTER) if it should be recreated (hence all existing files there are deleted) or not. If no, all existing files are named back to their PIPEFILE name. Thus, it is guaranteed that later (in step 2.12) all new files are either added, or overwrite pre-existing solutions, and in any case the versioning scheme remains valid (remember that versioning can only work on the whole data pool and not file by file).

2.3 read, under DELETE, list of PRO.CATGs to move for $RAW_TYPE in config file (or read entry ALL under RAW_TYPE);
if no PRO.CATG is listed for $RAW_TYPE (and no entry ALL exists), continue with next RAW_TYPE

2.4 create list of fits file names having movable PRO.CATG ('file_list'). If none found, continue with next RAW_TYPE
2.5 create list of associated ABs, association/trending/processing/scoring logs (evaluate ROOT_NAMEs in file_list; 'ab_list')
2.6 create list of associated further information (evaluate ROOT_NAMEs in file_list; extensions configured per RAW_TYPE in config file, e.g. gif files; 'graph_list'); if none found, or no extensions specified, skip this step
2.7 Call SPECIAL_PLUGIN, if configured; refresh file_list afterwards (since the plugin may have removed files)
2.8 move the fits files in file_list to $DFO_CAL_DIR/<DATE> or $DFO_SCI_DIR/<DATE>
2.9 move the graphics files in graph_list and the coversheet to $DFO_PLT_DIR/<DATE> and other FURTHER information to $DFO_LOG_DIR/<DATE>
2.10 delete $DFS_PRODUCT/<RAW_TYPE>/<DATE>
2.11 remove all $MODE files for $DATE from $DFO_CAL_DIR/VCAL (the container for virtual calibrations created by createAB) unless they have no MCAL counterpart
2.12 call renameProducts for MODE and DATE (
if configured)
2.13 call updateAB to update the ABs with the final file names and locations
2.14 move the ABs, AB logs and processing logs to $DFO_LOG_DIR/<DATE>
2.15 update links in scoring logs (HTML), move to $DFO_LOG_DIR/<DATE>

2.16 write call of ingestProducts into $ING_FILE_NAME

2.18 CALIB: export content of $DFO_LOG_DIR/<DATE> and $DFO_PLT_DIR/<DATE> to qcweb; all links in HTML files are edited properly
2.18 SCIENCE: export content of $DFO_LOG_DIR/<DATE> to qcweb; all links in HTML files are edited properly

3. Set process status to cal_Finished/sci_Finished.