Health Check Monitor:
strategy

Here is the strategy for transfer the pipeline-processed QC information to the Health Check (HC) Monitor:

• once per hour, a script is run to search, per instrument, for new calibration data
• if found, they are pipeline-processed, their QC1 parameters are extracted into the QC database
• if none found: stop here and try again one hour later
• after one hour, any new calibration data, plus the ones which failed upon earlier attempts, are processed
  (failure could happen e.g. because the raw data are not yet accessible, or require other calibration data that have not yet been processed)
• on Paranal side, send once per hour all available QC opslog data
• QC Garching has a cronjob running to detect and parse new QC OPSLOG information
• update Health Check plots, export to HC Monitor
• inspect HC Monitor to assist with daily QC checks [PSO and QC]

As part of the Calibration Plan, certain calibrations (HC calibrations) are executed solely for the purpose of checking the instrument status. They are supplemented by calibrations drivenby the calibration plan and triggered by the science observations of the night.

Both pools of calibration data can be used for extraction of HC information. The exact configuration, i.e. the selection of calibration data and QC1 parameters to go into the HC Monitor, depends on the mutual agreement of the Instrument Scientist and the QC scientist, for each instrument separately.

Any acquired data on Paranal are put into a staging area. From here the transfer process starts which has priorities defined (ToO/RRM science data have highest priority, then calibrations, then ordinary science data, then test/acquisition data). Data arrive in the Garching staging area from where they are ingested into the Primary Archive (PA) and Secondary Archive (SA). QC can access all data in the Primary Archive.

QC scans for new calibration data once per hour, pipeline-processes them, derives QC reports, extracts QC1 parameters and stores them in the QC1 database. For most of the QC1 parameters, scores are derived which are based on configured thresholds and designed to detect outliers and issues automatically.

The on-line pipelines on Paranal also produce QC1 parameters. These data are collected in opslog files and transferred via ftp. They are included as a fall-back solution in case of problems with data transfer. They are derived with default processing parameters and a static calibration database. They are less accurate than the final QC1 parameter values as determined by the Garching pipelines and therefore have no permanent value

On each QC machine in Garching, there is a cronjob installed searching for and scanning newly arrived OPSLOG files once per hour.

Triggered by a cronjob, or when new ops log parameters are detected, a QC tool called trendPlotter is fired and combines the QC1 parameters as read from the QC1 database, and the most recent QC1 parameters from opslog files, into a report. All HC plots have a common design and architecture and are collected under the HC monitor.

The monitor comes in two views: the graphical view, and the quick-look view based on scores. Both versions are based on the same input data, but the quick-look report concentrates on data from the last week only, and replaces parameter values by scores (either 0 = ok, or 1 = nok). The quick-look view strongly reduces information and complexity and essentially feed backs "all ok" or "problem detected". It is designed to catch the attention on relevant outliers.

The Health Check Monitor should be inspected both by the QC scientist and by the Paranal daytime astronomer/TIO. There are overview pages per instrument, linked as "score overview" to the navigation bar.There is also an overview of all instrument scores, linked as "ALL INSTRUMENTS" and dubbed the "shiftleader's page".