Common DFOS tools:
|dfos = Data Flow Operations System, the common tool set for DFO|
|make printable||new:||see also:|
- supports DFOS and PHOENIX, no functional change
off-line calls of ingestProducts are collected in JOBS_INGEST and managed by dfoMonitor
This tool is called to distribute, after certification, all or selected pipeline products and graphical information to the final directories. The final directories on the DFOS systems are: $DFO_CAL_DIR/$DATE for calib data, $DFO_SCI_DIR/$DATE for science data, $DFO_PLT_DIR/$DATE for graphical information, and $DFO_LOG_DIR/$DATA for pipeline logs, association logs, trending logs, and ABs. Some data are also exported to the QC server and to the ESO web server.
The tool also calls the DFOS tools renameProducts and updateAB. Calls for product ingestion are written into the JOBS_INGEST file under $DFO_JOB_DIR. The tool also manages the content of $DFO_CAL_DIR/VCAL (the virtual calibrations container), i.e. removes those virtual calibs which became certified (REAL) in the end. All logs and plots (except postscript files) are exported to the QC web server qcweb for on-line reference.
An optional, user-provided plugin can be used e.g. to store certain products in a temporary workspace for detailed analysis. The package has an example plugin.
Find its place in the daily workflow here.
Type moveProducts -h for on-line help, moveProducts -v for the version number, and
moveProducts -m CALIB -d 2014-07-12
to move all pipeline products for mode CALIB on that date to their final place (if they are configured to be moved).
Use dfosExplorer or dfosInstall . There is an optional plugin, pgi_moveProducts, which remains in $DFO_INSTALL_DIR/moveProducts. Check its function and content and decide if you want to use it.
|Section 1: general|
|TOOL_MODE||INTER||execute mode for tool: INTER or AUTO; INTER for interactive execution|
|SPECIAL_PLUGIN||pgi_moveProducts||optional plugin for special tasks (can be NONE)|
Section 2: Supported RAW_TYPEs
|RAW_TYPE||BIAS | CALIB | ALL|
|RAW_TYPE||FFLAT | CALIB | SELECT||if there is a SELECT entry, it joins to the DELETE entries.|
| Section 3: Selected product types *NOT* to be saved in
DFO_CAL_DIR or DFO_SCI_DIR
NOTE: if SAVE_MODE = NONE, all products are deleted and need not be listed here. All PRO_CATGs listed here are deleted.
|DELETE||FFLAT | FF_EXTPIXELS||only EXTPIXELS products are deleted, all others are saved|
Section 4: treatment of additional info (graphics)
4.1 Graphical information
|PLOT|| GIF YES
|Supported formats: gif/png/jpg. They go to $DFO_PLT_DIR and are exported to qcweb. 'ps' files are not supported.|
Section 5: Product renaming
|RENAME||CALIB | YES||rename CALIB products|
|RENAME||SCIENCE | YES||rename SCIENCE products|
The tool writes the DFO status cal_Finished and sci_Finished.
1. read list of RAW_TYPEs for selected mode in config file
2. loop on RAW_TYPEs:
2.1 go to $DFS_PRODUCT/<RAW_TYPE>/<DATE>
2.2 Check for existence of $DFO_CAL_DIR/<DATE> or $DFO_SCI_DIR/<DATE>. If it already exists, ask the user (in mode INTER) if it should be recreated (hence all existing files there are deleted) or not. If no, all existing files are named back to their PIPEFILE name. Thus, it is guaranteed that later (in step 2.12) all new files are either added, or overwrite pre-existing solutions, and in any case the versioning scheme remains valid (remember that versioning can only work on the whole data pool and not file by file).
2.3 read, under DELETE, list of PRO.CATGs to move for $RAW_TYPE in config file (or read entry ALL under RAW_TYPE);
if no PRO.CATG is listed for $RAW_TYPE (and no entry ALL exists), continue with next RAW_TYPE
2.4 create list of fits file names having movable PRO.CATG ('file_list'). If none found, continue with next RAW_TYPE
2.5 create list of associated ABs, association/trending/processing/scoring logs (evaluate ROOT_NAMEs in file_list; 'ab_list')
2.6 create list of associated further information (evaluate ROOT_NAMEs in file_list; extensions configured per RAW_TYPE in config file, e.g. gif files; 'graph_list'); if none found, or no extensions specified, skip this step
2.7 Call SPECIAL_PLUGIN, if configured; refresh file_list afterwards (since the plugin may have removed files)
2.8 move the fits files in file_list to $DFO_CAL_DIR/<DATE> or $DFO_SCI_DIR/<DATE>
2.9 move the graphics files in graph_list and the coversheet to $DFO_PLT_DIR/<DATE> and other FURTHER information to $DFO_LOG_DIR/<DATE>
2.10 delete $DFS_PRODUCT/<RAW_TYPE>/<DATE>
2.11 remove all $MODE files for $DATE from $DFO_CAL_DIR/VCAL (the container for virtual calibrations created by createAB) unless they have no MCAL counterpart
2.12 call renameProducts for MODE and DATE (if configured)
2.13 call updateAB to update the ABs with the final file names and locations
2.14 move the ABs, AB logs and processing logs to $DFO_LOG_DIR/<DATE>
2.15 update links in scoring logs (HTML), move to $DFO_LOG_DIR/<DATE>
2.16 write call of ingestProducts into $ING_FILE_NAME
2.18 CALIB: export content of $DFO_LOG_DIR/<DATE> and $DFO_PLT_DIR/<DATE> to qcweb; all links in HTML files are edited properly
2.18 SCIENCE: export content of $DFO_LOG_DIR/<DATE> to qcweb; all links in HTML files are edited properly
3. Set process status to cal_Finished/sci_Finished.
|Last update: April 26, 2021 by rhanusch|