Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable New: see also:
 

v2.4:
- forced refresh (requires update of dfosCron and trendPlotter); more
- option -C disabled

v2.5:
- interacts with new BADQUAL interface of calChecker

v2.5.1:
- call HdrDownloader in incremental mode


workflow tool; tips and links under 'Operations'

[ used databases ] databases none
[ used dfos tools ] dfos tools launched by cronjob; calling ngasMonitor, createAB, createJob, processAB, processQC, execHC_TREND
[ output used by ] output CALIB ABs and their products; new QC1 parameters, updated trending plots
[ output used by ] upload/download list_calBadQual.dat on qcweb (<instr>/CAL/php): upload of updated list
topics: description | workflow (overview) | date list | execution: NORMAL TIMEOUT FORCED | output | logs | usage | config | operational hints

autoDaily

[ top ] Description

This tool is the standard wrapper tool providing the automatic, incremental daily dfos workflow. It provides automatic processing of CALIB data, all the way from detection of new data down to processing and scoring.

The tool calls ngasMonitor to search for new CALIB data. If nothing is found, it exits. If new CALIB data are found, the tool creates new ABs, executes them, ingests QC1 parameters, creates QC reports and scores, and finally updates any new HC report found by scoreQC. This strategy ensures that at the end of execution all HC plots are properly updated. Typically the tool is called as cronjob, once per hour. Since it works on new data only, it processes the data in an incremental way, providing an up-to-date feedback on the HC monitor and at the same time a good load balance.

The tool can be started externally from the HC monitor pages, using a php script. This mode is called "forced" mode and is designed for PSO staff to launch an upgrade of the HC monitor when they need urgent feedback and don't want to wait for the cronjob to start.

The tool supports the BADQUAL workflow of calChecker, by adding new ABs for TODAY to the list of offered ABs on the calChecker BADQUAL list (see there).

[ top ] Workflow. This is the workflow of autoDaily:

  check for new fits files
download and processing updating new HC jobs
autoDaily
calling ...   scoreQC
(as part of QC reports)
execHC_TREND [trendPlotter]
ngasMonitor createReport createAB createJobs + processAB + processQC  
finding ... new fits files new QC1 parameters new HC jobs

[ top ] Date list list_data_dates. autoDaily reads a file $DFO_MON_DIR/list_data_dates . This file is filled by ngasMonitor with processing dates. autoDaily loops over that date list. In incremental mode, that list usually contains only the current date, unless there is a delivery backlog (the data transfer is disturbed) and files are delayed. With the configuration key FLASHBACK=NO you can avoid the date list having dates which are older than two days.

You can also edit the date list to contain a custom set of dates (e.g. for reprocessing). Then you call the tool as 'autoDaily -D' from the command line.

[ top ] Execution flow. The tool then executes the following steps:

I. Normal execution mode:

for each DATE in $DFO_MON_DIR/list_data_dates :

[ top ] II. Execution if previous session has not yet finished ($OTHER_TIMEOUT): the tool checks for another session still being executed. This could happen if many ABs have been created and are still executing. Then, the tool goes into a "wait" loop and checks every 60 sec for the other autoDaily session being finished, until a total configurable TIMEOUT time has elapsed after which it terminates (notification email is sent).

With this strategy, it is guaranteed that autoDaily will always keep the HC reports up-to-date within 1 hour, no matter if it is executed under

See also the workflow table.

Since autoDaily is called again after 1 hour, the timeout parameter should not be longer than 60 minutes (actually a maximum value of 50 minutes is hard-coded).

[ top ] III. Forced execution: the tool can also be called from the HC monitor, with a php form that starts a password-protected dialog. If confirmed, the user can set a trigger which is recognized by a cronjob on the operational machine and used to start an autoDaily session with the usual functionality decribed above. (For security, there is no direct access from the web server to the operational machine.) The external user can follow the log file and also receives mails with the start and finish information. The intention of this mode is to allow Paranal staff to launch a refresh of the HC monitor on a short notice, e.g. when they have created new calibration data and wait for quick feedback on the HC monitor.

For the forced mode to work properly, the following components are needed:

what provided by
1. php interface on www.eso.org/qc/<instr>/reports/php autoDaily (uploaded every time upon execution)
2. cronjob to check for trigger set by php script dfosCron -t autoDaily (to be run once per minute)
3. execute the forced mode autoDaily -F

Sketch of the forced mode of autoDaily: The user creates a request for running autoDaily, using the php interface and creating a file HC_ENFORCE. A monitoring cronjob on the muc machine discovers this "trigger file" and then launches 'autoDaily -F'. Processes on the web server and on the muc machine are strictly separated. It is not possible to launch a muc process from the web server. Execution of the processing job on the muc server is controlled by the cronjob. The trigger file must have correct name and a defined content, otherwise it is ignored.


Workflow:

- Click the button 'forced refresh', on any of the HC reports.

- You are prompted for authentication.

- The dialogue "Forced refresh" asks you to enter your email address (can be operational or personal). It is used to notify you via email about start and end of the processing. The defaults are per instrument: the telescope account, the QC name, and the name of the QC scientist.

Then you are asked for confirmation:

Next, the trigger is set. As soon as the monitoring cronjob picks up the trigger (after one minute at most), the process will be started on the operational machine of QC Garching (if no other similar process is already running).

You can watch the progress of the processing behind the marked link.

While waiting, the message is the following:
When the process is started, it sends a mail to the email address just entered. Then it displays first the execution plan ...

... and later the progress. Once finished, it sends again an email notification, with the complete processing log and some statistics (number of processed files, list of HC reports updated).

This file auto-refreshes every minute.

Cases of conflict between forced and scheduled instances of autoDaily:

Case

Behaviour

forced autoDaily, without scheduled autoDaily interfering nothing special, just like a command-line call
forced autoDaily runs into a scheduled autoDaily session forced autoDaily will be terminated (since its ultimate goal is already being achieved by the scheduled instance); email about termination sent to originator
scheduled autoDaily runs into forced autoDaily session scheduled instance goesin wait loop, just like for any other pre-existing instance.
forced autoDaily session runs into another forced session second session waits until $TIMEOUT is reached, then it terminates, email sent to originator.

 


[ top ] Output. The tool returns, for all processed dates:

The next steps in the daily workflow (certifyProducts and moveProducts for CALIB) are manually launched from the dfoMonitor since they require interaction and decisions.


[ top ] Execution log. The execution log is found under $DFO_MON_DIR/AUTO_DAILY. This log file mainly consists of the merged log files of all participating dfos tools. Since its full execution may take hours, the log file is kept temporary during runtime, as $TMP_DIR/autolog. After termination, it is written into the final AD_<execution date>.log. During execution, the temporary log file can be watched on the dfoMonitor (click the red button, or the 'log' link on the right top part). In forced mode, the log file can be followed on the browser window.

The autoDaily logs are kept separate from the cronjob logs (under CRON_LOGS) since otherwise it might become difficult to find them.

The tool creates another type of log file, the AB creation log, as ABL_<date>.log, also under $DFO_MON_DIR/AUTO_DAILY. It list each single AB created, with the creation timestamp. This is useful for the incremental mode and helps checking

Outdated log files (2 years or older, by year) are auto-deleted.


[ top ] How to use

Type autoDaily -h for on-line help, and autoDaily -v for the version number.

The standard way of calling the tool is as cronjob.

You can also call the tool from the command line, or via the php interface linked to the HC monitor.

Call 'autoDaily -D' if you want the full execution of a custom date list $DFO_MON_DIR/list_data_dates.

[ top ] Configuration file

The tool confguration file (config.autoDaily) defines:

USER giraffe needs to be set for cronjob enabling
DISK_SPACE95% stop if disk space occupation exceeds $DISK_SPACE
OTHER_TIMEOUT 50 timeout parameter for waiting for other dfos tools (in minutes); should be 50 or less!
PGI_PREPROC pgi_fork optional, user-provided plugin script to manipulate the workflow in a non-standard way
FLASHBACK NO YES|NO: if YES, tool will backwards-process dates older than 2 days if they were incomplete and ngasMonitor reports a new file (default: NO; evaluated only if ENABLE_INCREM=YES)

[ top ] Operational hints.

[ top ]

Last update: November 9, 2015 by rhanusch