Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:
  v2.4:
- PHOENIX notification removed

dfoMonitor createReport

   
[ used databases ] databases DFO_db (daily_comments); obs_metadata..data_products
[ used dfos tools ] dfos tools getStatusAB, ngasClient , qc1Ingest
[ output used by ] output $DFO_MON_DIR/FINISHED
[ output used by ] upload/download upload to qcweb (monitor/FINISHED)

histoMonitor

Description

This tool creates and maintains a historical record of the DFO operations for a particular instrument. It is also known as the "nautilus". Why?

The histoMonitor creates a local web site that is also exported to the QC web server qcweb. It has two main parts:

1. DFO history

2. Package history (for tradition)


DFO history. The DFO history is the history of the daily processing workflow. It comes in three levels:

Level 1: This page is dynamically created each time the tool is called.

Screenshot of the histoMonitor level1 page [click to enlarge]

Level 2: This page is incrementally updated. Usually, with finishNight a new entry is created. Once a date is removed from the dfoMonitor, it shows up permanently on the histoMonitor. There is one line per date, e.g.:

DATE NR cdb? sci? SM_VM report statistics ABs comments
2017-10-06 NR SM report 20/3 done (14/14) Begin of operations

Level 3: This page is the AB monitor. Until 2020-12-01, this page contains all ABs, both science and calib. Beyond that date, only calib ABs are visible. The page is a coversheet to all log, graphical and scoring information that is stored permanently in the local file system:

This page has exactly the format as the usual AB monitor, it is created by getStatusAB scanning all ABs in $DFO_LOG_DIR/<date>.


Package history (supported until 2011-10)

The package history is sorted by periods. Two periods constitute a year.

The package history comes in two levels.

Level 1: the main page, same as above. This page is dynamically created each time the tool is called.

Level 2: the period page. It lists all released packages for that period. The relation package - period is established on the basis of data acquisition: all data acquired in a given period constitute a package for that period (depending on size, this package could also come in partial packages for that period). Data for the same run_id but taken in a different period build another package, sorted under that period.

For each package, a set of parameters, files or directories is linked, like e.g. the archive list, the TOC, the observing reports etc.


DFO database table. The daily comments are written in, and read from, the database table daily_comments that is part of the DFO database (formally this is just another QC1 database table). The database is used as central storage and backup facility here, but you may also browse entries.

filling tool:
histoMonitor
writes into:
daily_comments
entries visible:
browser: http://archive.eso.org/bin/qc1_cgi?action=qc1_browse_table&table=daily_comments
entries displayed in histoMonitor and data reports (by createReport)
maintenance:
none needed (histoMonitor checks for existing entries and deletes them); there is qc1Hide just in case

Output

The histoMonitor creates a set of HTML pages under $DFO_MON_DIR/FINISHED. The pages are collected under <year>/<ym> as <ym>.html where ym stands for e.g. 2008-01 (for the data processing history), or under <period> as <period>.html (for the package history tree). The comments are stored in text files, per day, under the same directory tree.

For each date, there is also the standard AB monitor.


How to use: initial setup

The tool creates and maintains a local web site. To obtain the first set of files might take some time, afterwards it goes much faster.

Step 1: call the tool for one complete, recent month, e.g. 'histoMonitor -m 2007-12'.
It will scan all days belonging to this month.

Per date:
information is extracted from $DFO_LOG_DIR/<date>
the tool creates the AB monitor
nightlog: check in getObInfo if NLT applicable: if yes, link to NLT URL; if not, check in $DFO_LST_DIR if found locally:
- if found, it is linked;
- if not found, but headers exist: data report is created from scratch and linked;
- if not found, and no headers are found: headers are downloaded with ngasClient;
- if not found, no headers found, no download possible: fits files are downloaded (!), their header extracted, the fits files deleted;
ingestion: either DFO_STATUS is scanned, or the list files from the ingestion are searched
statistics and SM_VM flags: taken from the report
AB monitor: created from the ABs

Note: for recent months, you will most likely have all information in your system, and the only performance issue is getting the AB monitor. For older months, with no hdr information in your system, things become more time consuming, with the header extraction becoming the dominant factor.

Step 2: Once you have a feeling how this works and how long it takes, do the same for the other months of the same year, by hand. You can also call 'histoMonitor -y 2007' to do it in one run for the whole year.

Step 3: The final step is to do this for more and more years, backwards in time, or to just call 'histoMonitor' that does the complete scan in one go (but this may take a couple of hours!).

Step 4: Fine tuning. Reviewing the full set, you may want to decide to optimize e.g. the length of the statistics bar. Re-running the tool, once the complete data reports and AB monitor pages exist, is not critical and can be done anytime.

As a rule of thumb, a full month w/o AB monitor refresh takes a few seconds, a full year about a minute.


How to use: daily increments

To add a new date, typically after finishNight:

call 'histoMonitor -m <yyyy-mm>' where the month corresponds to the date to be added. This will effectively add the new date (all steps done listed under 'step 1' above) and embed it properly in the pre-existing dates.

How to use: other calls for the DFO workflow


Exported version

The qcweb version in general is visible only ESO internally. Since packages contain sensitive information (at least within the proprietary period) the package part of histoMonitor is also password protected, like e.g. the nightlog information on the calChecker output.


Configuration file

config.histoMonitor defines:

Section 1
SCI_SCALE 5 scaling factor for calculating the width of the science statistics bar
SCI_MAX 20

maximum number of science files to display by scale
this value should probably not exceed 20-30
The HTML width of the cell will be:
- n_science * SCI_SCALE if n_science below SCI_MAX
- SCI_MAX otherwise

Notes:

- the tool provides a marking of the current last date visible in the dfoMonitor;

- you can add useful information to the main overview page by filling an optional file info.histoMonitor (HTML code supported);

- the dfoMonitor has a link to the current last page of the histoMonitor;

- the histoMonitor web site is exported to qcweb.hq.eso.org/~qc/<instr>/monitor/FINISHED/histoMonitor.html that is visible ESO internally only;

- the tool offers a comfortable navigation bar for the DFO workflow that however needs to be updated, under certain circumstances, on all or many other pages:


Last update: April 26, 2021 by rhanusch