Common DFOS tools:
Documentation
make printable
Operations with DFOS: daily workflow, the simple model
The daily QC1 workflow for processing of calibration data is completely automatic. The dfos tool autoDaily provides the incremental and automatic CALIB processing. This is the most important components of the real-time QC system closing the QC loop with the mountain.
For tutorial purpose, we list here the basic steps of the daily workflow as you would execute them one by one on the command line. For CALIB processing, the first parts up to "execute QC jobs" are provided by autoDaily. See the description of that part here.
Let's assume you want to process a complete set of data from night 2014-11-15. You would start with the PRE-processing part of the daily workflow:
| 1. PREpare (all done with headers) | ||
| download headers | HdrDownloader -d 2014-11-15 | |
| create report | createReport -d 2014-11-15 | |
| create ABs: CALIB | createAB -m CALIB -d 2014-11-15 | |
filter data |
filterRaw -d 2014-11-15 -m SCAN | [done within createAB ] |
Next, the PROCessing part of the workflow starts, for mode CALIB:
| 2. PROCessing (CALIB) | ||
| create processing jobs | createJob -m CALIB -d 2014-11-15 | pipeline jobs (ABs) are written into a job file, for off-line processing |
| execute the job file | [execAB call in JOBS_NIGHT] | on-line or off-line |
| execute QC jobs | [execQC call in JOBS_NIGHT] |
included in job file |
Next, the POST-processing part for the CALIB mode is done:
| 3. POST-processing (CALIB) | ||
| certify products | certifyProducts -m CALIB -d 2014-11-15 | interactive certification |
| final distribution | moveProducts -m CALIB -d 2014-11-15 | move the certified products to $DFO_CAL_DIR/2014-11-15; includes renaming |
| product ingestion | ingestProducts -m CALIB -d 2014-11-15 | is usually done off-line, calling JOBS_INGEST |
The SCIENCE data (if any) are not processed. We currently create their ABs for verification and then store them for future reference, but they are not used operationally:
| 4. PRE-processing (SCIENCE) | ||
| create ABs: SCIENCE | createAB -m SCIENCE -d 2014-11-15 | ABs are created, verified (using verifyAB), and stored locally and on qcweb. |
Now we finish the daily workflow:
| 5. POST-processing (SCIENCE and ALL) | ||
| final distribution | moveProducts -m SCIENCE -d 2014-11-15 | move the ABs and logs to $DFO_SCI_DIR/2014-11-15 |
| finish workflow | finishNight -d 2014-11-15 | clean up temporary information, store statistics and logs |
Structuring the workflow
The workflow can be structured in several ways:
- the daily workflow is divided into three parts: PRE (pre-processing or preparation); PROC (pipeline and QC processing); POST (post-processing).
- the PROC part is likely to be executed off-line (e.g. over night), while the PRE and the POST parts essentially can be done on-line.
For the simple operational model discussed here, we assume all steps to be executed interactively. This is the baseline for the automatic execution which is documented here.
| part | PRE | PROC | POST | ||||||||
| mode | all | CALIB/SCIENCE | all | ||||||||
| workflow steps 1-3 | download headers, create report | CALIB: create ABs; filter data | create jobs | process ABs, download data | create QC report, scoring | certifi- cation | move | rename | ingest products | update ABs | |
| workflow steps 4-5 | finish night | ||||||||||
| SCIENCE: create ABs; filter data | move | verify ABs | update ABs | ||||||||
| DFOS tool | HdrDownloader, createReport | createAB, filterRaw | createJob | [JOBS]; ngasClient | [JOBS]; scoreQC | certify Products | moveProducts; ingestProducts; verifyAB | finish Night | |||
| Last update: April 26, 2021 by rhanusch |