Common DFOS tools:
Documentation
dfos = Data Flow Operations System, the common tool set for DFO
*make printable

Operations with DFOS: daily cronjobs

The automatic dfos system ensures that

Three core parts of dfos tools establish the automatic dfos system:

The following table gives an overview how these components work together (all tools work on latest date).

component    
calChecker
autoDaily
dfosCron -t JOBS_TREND
main step   tool      
reporting   hdr download HdrDownloader
yes
data report createReport+
yes
yes
calibration completeness   calChecker
yes
 
AB creation and processing    AB creation createAB  
yes
 
processing processAB/QC  
yes
 
scoring scoreQC  
yes
 
HC monitor affected plots trendPlotter/ JOBS_HEALTH  
yes
 
all plots trendPlotter/ JOBS_TREND    
yes
update frequency:
0.5 hr*
1 hr**
24 hr
multiple instances:
killer! younger one killed*** not an issue****

* daytime/nighttime
** while daytime calibs executing
*** based on the assumption that the older process is not hanging but still executing due to large data volume
**** trendPlotter enabled for parallel execution
+ final refresh upon histoMonitor

The following table details the autoDaily tasks.

1. PREpare
autoDaily: download headers dataLoader part of autoDaily
ngasMonitor ngasMonitor detect new data
no new data found: exit; new data found:
create report createReport -d 2020-11-15 refresh data report
create ABs: CALIB createAB -m CALIB -d 2020-11-15  
filter data
filterRaw -d 2020-11-15 -m SCAN [done within createAB ]

Next, the PROCessing part of the workflow starts, for mode CALIB:

2. PROCessing (CALIB)
create processing jobs createJob -m CALIB -d 2020-11-15 pipeline jobs (ABs) are written into a job file, for off-line processing
execute the job file [execAB call in JOBS_AUTO]

calls:
processAB --> ngasClient (data download)
execute QC jobs [execQC call in JOBS_AUTO]
included in job file
update HC monitor [execHC call in JOBS_AUTO] included in job file
Last update: April 26, 2021 by rhanusch