Common DFOS tools:

dfos = Data Flow Operations System, the common tool set for DFO
*make printable

Operations with DFOS: daily workflow, the simple model

The daily QC1 workflow for processing of calibration data is completely automatic. The dfos tool autoDaily provides the incremental and automatic CALIB processing. This is the most important components of the real-time QC system closing the QC loop with the mountain.

For tutorial purpose, we list here the basic steps of the daily workflow as you would execute them one by one on the command line. For CALIB processing, the first parts up to "execute QC jobs" are provided by autoDaily. See the description of that part here.

Let's assume you want to process a complete set of data from night 2014-11-15. You would start with the PRE-processing part of the daily workflow:

1. PREpare (all done with headers)
download headers HdrDownloader -d 2014-11-15  
create report createReport -d 2014-11-15  
create ABs: CALIB createAB -m CALIB -d 2014-11-15  
filter data
filterRaw -d 2014-11-15 -m SCAN [done within createAB ]

Next, the PROCessing part of the workflow starts, for mode CALIB:

2. PROCessing (CALIB)
create processing jobs createJob -m CALIB -d 2014-11-15 pipeline jobs (ABs) are written into a job file, for off-line processing
execute the job file [execAB call in JOBS_NIGHT]

on-line or off-line
the job file contains calls of processAB, starts ngasClient (data download)

execute QC jobs [execQC call in JOBS_NIGHT]
included in job file

Next, the POST-processing part for the CALIB mode is done:

3. POST-processing (CALIB)
certify products certifyProducts -m CALIB -d 2014-11-15 interactive certification
final distribution moveProducts -m CALIB -d 2014-11-15 move the certified products to $DFO_CAL_DIR/2014-11-15; includes renaming
product ingestion ingestProducts -m CALIB -d 2014-11-15 is usually done off-line, calling JOBS_INGEST

The SCIENCE data (if any) are not processed. We currently create their ABs for verification and then store them for future reference, but they are not used operationally:

4. PRE-processing (SCIENCE)
create ABs: SCIENCE createAB -m SCIENCE -d 2014-11-15 ABs are created, verified (using verifyAB), and stored locally and on qcweb.

Now we finish the daily workflow:

5. POST-processing (SCIENCE and ALL)
final distribution moveProducts -m SCIENCE -d 2014-11-15 move the ABs and logs to $DFO_SCI_DIR/2014-11-15
finish workflow finishNight -d 2014-11-15 clean up temporary information, store statistics and logs

Structuring the workflow

The workflow can be structured in several ways:

  • the daily workflow is divided into three parts: PRE (pre-processing or preparation); PROC (pipeline and QC processing); POST (post-processing).
  • the PROC part is likely to be executed off-line (e.g. over night), while the PRE and the POST parts essentially can be done on-line.

For the simple operational model discussed here, we assume all steps to be executed interactively. This is the baseline for the automatic execution which is documented here.

mode all CALIB/SCIENCE all
workflow steps 1-3 download headers, create report CALIB: create ABs; filter data create jobs process ABs, download data create QC report, scoring certifi- cation move rename ingest products update ABs  
workflow steps 4-5     finish night
SCIENCE: create ABs; filter data   move   verify ABs update ABs
DFOS tool HdrDownloader, createReport createAB, filterRaw createJob [JOBS]; ngasClient [JOBS]; scoreQC certify Products moveProducts; ingestProducts; verifyAB finish Night
Sketch of the daily workflow
Last update: April 26, 2021 by rhanusch