Common DFOS tools:
|dfos = Data Flow Operations System, the common tool set for DFO|
|make printable||new:||see also:|
- createJob writes the calls of processQC into the configured JOB_FILE.
- While processQC calls QC reporting, processAB calls the pipeline.
- rawdisp.py is the tool to create RAW screenshots.
|enabled for parallel execution|
|enabled for condor execution|
The tool processQC is called for the specified AB. It does the following:
The QC procedures are instrument-specific and user-provided. They are documented here. In general they do the following:
processQC is enabled to execute parallel jobs (N processQC calls executed simultaneously). Of course, that option requires the corresponding QC procedure to support parallel processing:
The processQC tool is usually called as part of the process job file created by createJob (execQC), but can also be called from the command line.
Raw file screenshots. As the first step, processQC calls the tool rawdisp.py which is a python tool to create screenshots. The tool creates screenshots of all RAWFILEs in the AB (or, as configured, of a subsample of those). It does so only for the 7 latest dates (counted from TODAY), since the screenshots are intended to support the daytime QC process on Paranal only.
They are organized in a web page per RAW_TYPE, collecting thumbnail plots that can be inspected at larger detail which then also includes histograms, cuts etc. Existing plots are not created again. Find more here. These web pages are created by the tool getStatusAB.
In exceptional cases (instruments with raw files as bintables), you can configure RAWDISP_SUPPORT=NO which then prevents calling qc_rawdisp.py.
Coversheet. As the last step, processQC can create a coversheet page as entry point to the QC reports. In complex situations there is more than one QC report per AB. The coversheet mechanism has been developed to support those cases in a convenient way. The coversheet is an html file which collects all QC reports for a given AB in a convenient way. It provides easy access to the QC reports in case of complicated situations, in particular MEF instruments. There are two options for its creation:
A coversheet is always automatically created by processQC if more than one QC report with the $PROD_ROOT_NAME and extension gif|png|jpg is found. The coversheet name is constructed from the AB name, with extension .ab replaced by _cs.html. It is created under the same $DFS_PRODUCT directory as all other products, and later moved to $DFO_PLT_DIR (not to $DFO_LOG_DIR!).
The following rules apply (this is relevant only if you plan to use your own PGI_COVER procedure!):
[Note that the pgi tool pgi_coversheet is delivered by dfosInstall into $DFO_GUI_DIR but not automatically moved to $DFO_BIN_DIR (to avoid overwriting your customized version the next time).]
Note that there is an "advanced" feature in scoreQC which you could use in your custom-built pgi_coversheet: those of your links which you mark with <!--SCORE_REPORT--> are extracted into the "QC REPORT" section of your score report. See more in the scoreQC documentation.
Depending on the QC procedures; typically a set of graphical information will be created (png, ima, plt files etc.), as well as QC1 parameters which are ingested by the QC procedures, and scores created by scoreQC.
Type processQC -h for on-line help, processQC -v for the version number,
processQC -a <ab_name>
to launch the QC report for a specific AB.
For execution by condor you have to specify the option -u <user>, e.g. -u vircam, in order to make condor find the home directory and source the environment properly.
All configured QC procedures need to exist and need to be called in the proper syntax. That syntax is defined in the config file. The example MIDAS procedure bias.prg remains under $DFO_INSTALL_DIR/processQC/.
|Section 1: general|
|QCBQS_TYPE||SER | PAR | IMPLICIT||SER: series processing (ABs one by one)(default)
PAR: parallel processing with CONDOR
IMPLICIT: called within processAB (this value is read by createJob)
|PGI_COVER||pgi_coversheet||name of user-provided pgi to produce a custom coversheet, to be provided in $DFO_BIN_DIR (optional)|
|RAWDISP_SUPPORT||YES | NO||NO: qc_rawdisp.py is not called; default: YES|
|FINAL_PLUGIN||&&pgi_GIRAFFE_postQC $AB&&||optional plugin to be called at the end; to be enclosed by &&
[note: this was implicitly called as FINAL_PLUGIN of processAB]
|Section 2: QC procedure calls
Defines the calls of the QC procedures. There must be at most one line per RAW_TYPE. More documentation can be found in the config file.
|CALL_SYNTAX||RAW_TYPE, PRIM_FILE, CALL||these are pieces of shell code, wrapped into &&...&&. Make sure that '&&' are the last characters in the line.|
None. The DFO status cal_QC and sci_QC are written by the execution file, $DFO_JOB_DIR/execQC.
1.1 Check if AB exists
1.2 read from AB: RAW_TYPE, DATE
1.3 find name of product file from config file
1.4 check if $DFS_PRODUCT/<RAW_TYPE>/<DATE> exists; go there
2. if MODE=CALIB and DATE not older than TODAY-7d and RAWDISP_SUPPORT=YES: call qc_rawdisp.py for the AB; graphics go to $DFS_PRODUCT/RAWDISP/<DATE>
3. call configured QC report
[2.1 unpack fits.fz files from RAW section in AB (relevant for VIRCAM only) ]
2.2 Launch QC report with configured call
4. call procedure createCS (or the pgi $PGI_COVER if configured) if more than one graphics file is found
5. call scoreQC (unless called per detector)
6. Measure execution time in sec and write into AB (as TQCEXEC)
7. Update .tab files (QC report, coversheet, scores)
8. Call FINAL_PLUGIN if configured
|Last update: April 26, 2021 by rhanusch|