Common DFOS tools:

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:

- dropping call of checkConsist; config key PRINTER dropped

processQC, moveProducts; scoreQC; rawdisp2reference
getObInfo for OB related information

- calling rawdisp2reference at the end

[ used databases ] databases DFO_db (raw_comments | prod_comments) (insert/update); qc1_score (read)
[ used dfos tools ] dfos tools qc1Ingest to insert new entries into raw_comments and/or prod_comments; calls user-provided tool for displaying QC report; scoreHC to refresh score reports for HC monitor
[ output used by ] output certified or rejected products; uses scoreQC output
[ upload/download ] upload/download up: comments about raw or product files
topics: description | auto-certification | jumping | workflow | incremental mode | comments | configuration | hints


[ top ] Description

This tool is called to certify pipeline products. It is, like processQC, a wrapper around an instrument-specific and user-provided procedure. The tool scans all directories $DFS_PRODUCT/<RAW_TYPE>/<DATE>, as configured per MODE. These have been created by createAB, and filled by processAB and processQC.

The tool assumes that the user has some procedure (defined in PROC_CALL) which is able to load the graphical information which has been created during the execution of processQC. It then loops over all RAW_TYPEs specified, and on all ABs for that RAW_TYPE. Thereby the tool also checks for unsuccessful AB executions.

[ top ] Auto-certification. The tool offers auto-certification. This is a key concept towards combining objective and careful quality control with the efficiency requirement of mass production. This mechanism works with ABs being scored by scoreQC. The idea is to "delegate" certification to the scoring tool. This delegation requires a stable and complete configuration of scoring.

Per RAW_TYPE, the user can configure a certain certification mode:
certification mode action
ALL offer all ABs for certification
NONZERO offer for review only ABs having non-zero scores, or no scores, or the RAW_TYPES specified in configuration key CERTIF_RAW
RANDOM like NONZERO, plus offer N% ABs with zero scores, and at least one AB

There are two "extreme" situations:

- no scoring implemented, or scoring incomplete --> ALL (certification needs to be done by QC scientist). This can obviously be only a transient solution (for a new instrument or pipeline) or a niche solution (for rarely used modes without proper pipeline support).

- scoring fully implemented and complete --> NONZERO or RANDOM.

Strongly recommended is RANDOM: this is a choice which still offers the full QC report for a subset of ABs. This is a trade-off between the competing goals of review completeness and "review fatigue" caused by the need of reviewing many similar reports. Depending on the situation, the user may choose to have offered between 5% and 50% of all zero-score QC reports. Recommended value is 20% for instruments running in stable operations.

For new instruments ALL is the better choice.

[Technically, the system variable $RANDOM is used to derive the AB selection. Naturally, both the selected ABs, and their number, will differ from call to call. Combining RANDOM and jumping (see below), you may therefore select a certain 20% AB set, and immediately afterwards another set.]
All operational QC accounts should have a fully implemented scoring system.
All operational QC accounts should run the RANDOM certification mode.

CERTIF_MODE is configurable. Exceptions from that standard choice can be configured for certain RAW_TYPEs. CERTIF_FRAC is applied for CERTIF_MODE=RANDOM only.

After certification, the box CERTIF in the AB monitor is filled as follows:
OK certified
REJ rejected
AUTO auto-certified

If a product comment has been entered (see here), it will also be displayed in the CERTIF box.

In AUTO certification mode, the tool calls getStatusAB with option -e (quick edit) to update only the CERTIF column in the AB monitor.

[ top ] Jumping. The tool offers the option to jump forth and back within the certification loops. Once the user has jumped, there is a security mechanism to avoid unintentional exit from the tool (since products may not yet been reviewed). The user needs to confirm the exit. The tool then either exits on confirmation, or re-starts the certification loop, with the option for the user to jump to any product not yet certified.

Auto-certification and jumping can be combined: in auto-certification mode, the user can jump to any RAW_TYPE, and any AB. From the selection on, the certification mode for the selected RAW_TYPE is evaluated again.

In NONZERO certification mode, a jump is offered at the very end of the certification workflow. If chosen, the tool then starts again, in forced mode ALL, to offer full choices for selection and jumping.

[ top ] Workflow options. The tool has the following configurable options:

The tool pops up the text file with OB start times and comments (extracted by the getObInfo tool).

Unless it finds an AB selected for automatic certification, the tool asks for certification:

Certified (y/yc | n | j[ump]/e[xit]) (y)

Entering "y" (the default) results in certification, without further questions. "yc" brings you to a small menue where you can enter raw and/or product file comments (see here).

The provision of a product comment is enforced under the following conditions:

The reason for enforcing a comment is that all these cases are considered exceptional and potentially caused by an issue. Therefore both the rejection or certification should be accompanied by some information provided as a comment.

When enforcing a comment, the tool checks for already existing comments (if one is found, the tool just continues). Comments could already exist because 'certifyProcust Light' has been run, or because the bulk mode for comment ingestion has been used.

Please keep in mind to always provide significant comments (not "AB scores red" or "parameter XY above threshold", but e.g. "flat-field lamp flux fixed the next day" or "RON ok, statistical outlier").

Typing "n" means REJECTION, and the tool will enter the rejection workflow:

Removing a product name from $DFO_CAL_DIR/VCAL means it is not available any longer for other associations. 'y' is the right answer if there is a problem with the raw file (e.g. no light found). 'n' may be the right answer if the raw file is ok but the pipeline recipe failed (because e.g. of inappropriate parameter choice, or a bad algorithm etc.). In such case you may want to deliver (pack) the raw file with a science file, and this is only possible if the virtual product name is found in $DFO_CAL_DIR/VCAL.

A special case is that the pipeline has failed and no products have been created. Then, the tool enters the same workflow as under rejection. At the end the tool asks:

Continue with next AB (y)/j[ump]/e[xit]) (y)

The tool writes into a trending log ('tlog') which is created, per AB, in $DFO_AB_DIR. This is a mechanism to store a certification flag. It is used by the TQS tools scoreQC and scoreHC.

There are also optional calls of user-provided procedures for the following cases:

[ top ] Preliminary certification of calibrations in incremental mode (certifyProducts light). If DATE = TODAY, CALIB data are processed by autoDaily incrementally. Since the flow of CALIB data is not yet finished, CALIB data from TODAY should normally not be certified and moved (instead, you wait until the next day). However, there might be cases when a provisional certification ("certifyProducts light") makes sense, in particular if comments are provided as feedback to Paranal operations. Then, you can use the blue button on the dfoMonitor which calls the workflow of the light mode of certifyProducts. Comments and certification flags are displayed on the AB monitor, which is also exported to the qcweb site. You can stop anywhere in the workflow, and jump into it again later. No flag is written into DFO_STATUS which means that officially you have not yet certified this date and need to visit this date again later. Depending on the status of your certification, you may then only need to enter and exit again, in order to proceed with moveProducts.

Note that for certifyProducts light, in case of rejection, the fits files are not deleted, contrary to the normal situation: if they were deleted, then autoDaily would create them again.

In unusual operational situations you may want to call preliminary certification for dates older than TODAY. There is no button foreseen on the dfoMonitor, but you can find the corresponding launcher scripts in $DFO_GUI_DIR until the corresponding date is finished. Look out for 'launch_certifyPLight_<date>_CALIB.esh' and call it on the command line.

[ top ] File comments. After options 'n' and 'yc' you can enter comments, about the parent raw files and/or the products. In either case the workflow is as follows:

Entering and editing comments:

Ingesting comments, updating score reports:

Display of comments:

Product comments are also stored per AB in a text file called $DFO_LOG_DIR/<date>/<ab>.cmt. This text file is found by merge2Fits and included in the LOGS ancillary fits file.



How to use

Type certifyProducts -h for on-line help, certifyProducts -v for the version number, and

certifyProducts -m CALIB -d 2024-07-12

to certify all pipeline products for mode CALIB on that date,

certifyProducts -m CALIB -d 2024-07-12 -L

for the light option (no flag entry in DFO_STATUS, not to be called from the command line).


Use dfosExplorer or dfosInstall.

The MIDAS procedure automode.prg is attached as an example only. It is *not* designed as a common tool. Your version is expected under $DFO_PROC_DIR.

[ top ] Configuration file

config.certifyProducts defines:
Section 1: general
DET_YN YES | NO enable/disable detailed investigation
PRINT_YN YES | NO enable/disable asking for printing
XTERM_GEOM_NIGHT 100x40+400+10 xterm window size (for viewing the NIGHTLOG file)
MONITOR_UPDATE YES | NO refresh AB monitor with CERTIF flag (after each AB); optional, default is YES
DISPLAY_SCORE YES | NO point the browser to the score table if the score is non-zero; optional, default is NO
CERTIF_MODE ALL | NONZERO | RANDOM offer for review:
- ALL: all ABs
- RANDOM: non-zero scores, null scores, plus CERTIF_FRAC% randomly selected ABs (recommended)
- NONZERO: non-zero and null scores only
CERTIF_FRAC 20 fraction of zero score ABs to be selected for display (for CERTIF_MODE RANDOM only); supported: 5/10/20/30/40/50; default is 20.

Section 2: Procedure calls
Syntax of procedure calls: everything between &&...&& is evaluated.
- the last characters have to be &&
- only one such line per item can be defined

FAILURE_CALL &&echo "call your own procedure here"&&

optional call of procedure in case no product is found

PROC_CALL &&inmidas -P -j "@@ $DFO_PROC_DIR/automode.prg $PRODUCT"&& mandatory call of user-owned procedure for displaying QC reports per $RAW_TYPE
DET_CALL &&rtd $PRODUCT&& optional command to call detailed investigation
DELETE_CALL &&echo "call your own procedure here"&& optional call of procedure in case 'rejected'
Section 3: List of RAW_TYPEs to loop
For each of the RAW_TYPEs listed here, the tool calls $PROC_CALL; only the RAW_TYPEs per MODE (CALIB or SCIENCE) are called; FILE_EXT specifies the index and extension of the primary = representative product and is used to create $PRODUCT, the complete product name used by $PROC_CALL
Section 4: List of trending tables with entries for RAW_TYPE
- entries in those tables will be hidden if QCPASSED_YN = N
- You can have multiple (or no) entries.
TREND_TABLE BIAS qc1_bias.dat these local tables are optional and transient, the final QC1 parameter set goes to the QC1 tables. Presently there is no command-line option for qc1Ingest so that their entries cannot be hidden automatically.
Section 5: List of RAW_TYPES which always should be COMPLETEly reviewed (overriding the CERTIF_MODE configuration)
RAW_TYPE e.g. FLAT: this RAW_TYPE is always certified in COMPLETE mode

Status information

The tool writes the DFO status cal_Certif.

Workflow description

certifyProducts -m CALIB -d <DATE>:

1. Read all RAW_TYPEs from config file for selected MODE

2. Loop over all $DFS_PRODUCT/<RAW_TYPE>/<DATE>; use AB_list as reference for expected products

2.1 pop up OB comments
check if subdir <RAW_TYPE> exists; if not: skip
2.3 check for configured CERTIF_MODE, CERTIF_FRAC and CERTIF_RAW entries; decide which ABs to review
2.4 check if product files exists

2.4.1 No: ask if raw comment is to be entered; enforced to enter "product comment" (actually a comment about pipeline failure)
2.4.2 call optional FAILURE_CALL;
2.4.3 offer to continue with next AB, jump, or exit

2.5 product files exist: call the configured procedure (PROC_CALL) to check the products
2.6 if configured: ask if detailed tool should be called (DETAILED_CALL)
2.7 certification: ask "QC passed for product (y/yc/n/j/e)"
2.8 certification=yc: ask if comments about raw or product files should be added; raw comments are always optional;
product comments are mandatory under the following conditions:

if yes, or if enforced: insert into file raw_comment or prod_comment; offer to call enterCommentsBulk; products certified; continue with next AB or next RAW_TYPE

2.9 if answer is 'y': products certified; continue with next product/AB or next RAW_TYPE
2.10 if answer is 'n': products rejected

2.10.1 ask if comment is to be inserted in raw_comments (optional) and prod_comments (mandatory)
2.10.2 display other ABs from the same day which use any of these products; you may decide to review or delete those products as well
2.10.3 if configured: call DELETE_CALL, a user-provided procedure to e.g. investigate or delete the parent raw files (mostly obsolete with XXLight!)
2.10.4 if products have been deleted: ask if product name should also be deleted from $DFO_CAL_DIR/VCAL (association pool of virtual products)
2.10.5 ask if QC1 parameters should be hidden; if yes, and if configured, delete entries in local $DFO_TREND_DIR/<TREND_TABLE>
2.10.6 delete all products having the same root name
2.10.7 provide input for hideFrame
2.10.8 ask to call BADQUAL workflow for calChecker, and provide info for it

2.11 if answer is 'j[ump]':

2.11.1 a list of available RAW_TYPEs is displayed, the current one is highlighted; select RAW_TYPE (or hit return to stay in current one)
2.11.2 a list of available ABs is displayed, with setup information; select AB (or hit return to select the top one)
2.11.3 after selection, the tool continues with step 2.2 (AB selection), until all products are reviewed, or another jump is chosen
2.11.4 once the user has jumped, there is a security mechanism to avoid unintentional exit from the tool (since products may not yet been reviewed)

2.12 if answer is 'e[xit]': exit after confirmation

3. set DFO status to 'cal_Certif' or 'sci_Certif' (unless option -L is set)

4. Offer one last chance to jump (if selected, tool is called again, and you can jump at the next certification offer; this may be useful in AUTO mode, or if you want to add another comment)

5. Call 'rawdisp2reference' and check for new raw_type/setup combinations not yet existing as references 

6. If comments exist (raw_comments or prod_comments), ingest them all in the DFO database (tables raw_comments or prod_comments).

7. If comments exist, check if HC reports are affected. If so, call 'scoreHC -r <report>' for each of them.

You can jump forward or backward. You can e.g. focus on all files of a specific RAW_TYPE, or all files of a selected setup. This becomes useful if you have repeatedly executed a set of ABs and want to focus on certain products. After jumping, the tool always continues with the configured AB selection from the selected AB on. By opting repeatedly for 'j', you can jump forward and backward.

Setup information is read from the AB_list and is always displayed together with the AB name.

You can browse or hide ingested comments here.

[ top ] Hints.

Last update: April 26, 2021 by rhanusch