Common DFOS tools:

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new:

see also:

Introductions by Mark Neeser:

scoreQC with HC (trendPlotter + scoreHC)

[ used databases ] databases qc1..qc1_score; qc1..qc1_columns
<instrument> qc1_db tables
[ used dfos tools ] dfos tools qcdate; qc1Ingest
[ output used by ] output used by trendPlotter / scoreHC; AB monitor ; certifyProducts
[ upload/download ] upload/download none
topics: description | output | output: MEF | output: explore | output: investigate | thresholds | factsheet | coversheet | configuration | operations | hints and pitfalls | technical

The qc1_plot cgi interface has the default plot format png, with gif and ps also supported. Before 2013-10-01 the default was 'gif'. All historical score symbols (in the score html reports) created before that date have 'format=gif', while all symbols created later have 'format=png'. This is transparent for the user.


[ top ] Description

enabled for parallel execution

Note that in general the scoring system (as configured in config.scoreQC) is NOT backwards compatible! This is because:
- thresholds are configured for the CURRENT instrument performance;
- setup values as interpreted in the configuration are read from the ABs and may evolve.

Since the scoring system is meant to support the closed QC loop with Paranal, there is generally no need to maintain previous versions of the scoring configuration.

This tool compares a configured set of QC1 parameters for a given product (calibration or science) to configured thresholds. It also permits comparison of these new values to existing values for similar data ("trending"), using dynamic plots of the QC1 plotter.

It is the key tool to automatic certification (in the daily workflow) and to the score-based Health Check monitor (within the concept of closed-loop QC).

The principle behind scoreQC is to configure thresholds for the most important QC1 parameters. These thresholds may either come from specifications, or from statistics, or from experience. If a new parameter value complies with the thresholds, it is scored '0', otherwise '1'. A score is a quality flag. Scoring, as used by QC, effectively counts outliers.

In general, more than one QC1 parameter per AB can be configured and scored. The choice of QC1 parameters depends on the raw_type. It can be fine-tuned to apply to

There might also be products with more than one (detector-) extension. In general scoring then needs to be configured for each detector/extension separately.

All scores for all detectors and QC1 parameters are finally added and give the final score, which ideally is 0.

Scores apply to an AB, so effectively all products of that AB are scored in total.

Technical remarks.
All configured QC1 parameters must be stored in the QC1 database. They must be saved in the same table (this is a technical limitation) and must be available in a single record, uniquely defined by mjd-obs. This is usually automatically achieved since there is a close correspondence between the qc1Ingest call and the scoreQC call within the same QC procedure (see below).

Scores are inserted into a technical QC1 database table called qc1_score (find it here) to provide a central storage of the scores. This is required for the HealthCheck QUICK-LOOK reports which display scores rather than plots, as provided by trendPlotter.

Extended documentation.
A web-friendly version of the talks given by Mark Neeser introducing scoreQC can be found here: scoreQC and scoreQC with HC (trendPlotter + scoreHC)

Using multiple configuration files.
Generally the tool configuration is contained in one file, config.scoreQC. For exceptional cases, one can configure multiple configuration files, which makes particularly sense if part of an instrument has one detector and the other part multiple detectors, using the MEF parameters. Currently these are UVES and SPHERE. This way of supporting multiple configuration is suitable for parallel processing of QC jobs (remember the tool is called within processQC and therefore needs parallel execution capabilities as well).

The standard config file config.scoreQC is still needed in that case. It is marked for multiple configuration with the key MULTI_CONFIG=YES. The classification pattern for the different cases is configured as CONFIG_STRING, and the names of the various config files are stored in the CONFIG_CASE section. See all details in the section about configuration. It is important to note that some other tools read config.scoreQC. Therefore, some configuration information needs to remain in the main config file. In particular, the mandatory section HC_PLOT is read by scoreHC and needs to be complete, as before.

It is also possible to use a pgi for more flexibility if the MULTI_CONFIG=YES mechanism doesn't work for you (since it assumes the config file switch based on the RAW_TYPE line in the AB). An example is the FORS2 case of having two different detectors (blue sensitive or normal) that are not reflexted as different RAW_TYPEs but are distinguished by their detector parameters. In that case, the pgi can be coded such as to recognize the different situations and then switch to the proper config file. The pgi name is configured as CONFIG_PGI.

[ top ] Output

Parameter score report. The tool output comes in two formats: there is one HTML page (named by the AB name and the extension .html), and one text file (with the same information but more suitable for automatic reading; extension .tlog). The HTML page comes with color-coded scores. Green marks score value 0, yellow 1 (or the value configured as YELLOW_SCORE), and red anything larger than that. Below you find the main table of an example report:

Score report HELP
TPL_ID:	    GIRAF_cal_expfname
TPL_NAME:   GIRAF Calibration Exposure
setup:      Medusa1_H525.8B 
time range: 2009-10-27 ... 2010-01-25 

AB | ALOG | PLOG | QC1_plotter | factsheet

mean_xdiff HC
mean_ydiff HC
flux HC
explore >>
1st QC report & coversheet: QC | COVER
Score data: details ...
score result: 0/4 best: 0/4
powered by QC [scoreQC v1.7]

The colored symbols are linked to a dynamic QC1 plot for that particular combination of QC1 parameter, detector and setting values, having the actual value marked blue and with configurable statistics options. Click on any of the squares below to see a demonstration.

The dynamic QC1 plots have a special feature: they display the configured thresholds for the scoring (in blue) and the database mean (in red). The threshold lines (see an example) can be asymmetric with respect to the mean value. (This feature is not available on the standard QC1 plotter interface where you instead can choose a variety of error thresholds which are always symmetrical to the selected average.) The score-threshold feature ensures that the graphical output and the output on the score page are consistent.

The explore link is offered with a much more convenient option to quickly explore the behaviour of a prticular QC1 parameter. More ...

The squares have tooltips with: the actual QC1 value, the configured thresholds, the average of the database values ("db_avg"), and the number of database values found ("db_num").

You can configure thresholds to be taken from trendPlotter results (see below). Such scores are marked in the score report as HC, e.g.:

mean_xdiff HC
mean_ydiff HC
flux HC

For convenience, the HC markers are linked to the corresponding HC plots. With the same version, the QC1 parameters come with tooltips (comments visible if the mouse is moved over the parameter name). These tooltips are extracted from the comment column in the qc1 database interfaces.

Coversheet. The section "QC report(s)" links to, or displays, QC reports and, if available, to the coversheet. The coversheet is an HTML file collecting links in a user-defined way to all QC reports. It is created by processQC. The coversheet mechanism has been improved with v1.6 and works as follows:

Scoring text file. The text file version (extension .tlog) is used e.g. by getStatusAB to catch the total score and display it on the AB monitor. It is also used by certifyProducts to select products for semi-automatic review and certification.

Score overview.
The scores as determined by scoreQC are collected and re-grouped by the tool scoreHC. The result is displayed in an overview file and linked to the HealthCheck Monitor, as e.g. for the case of SINFONI here.

HC_list. The list $DFO_JOB_DIR/execHC1_CALIB_<date> is created and maintained by scoreQC (in INCREMENTAL mode only). It contains for each AB the name of the corresponding HC report and is used for incremental updates of the HC monitor, executed by autoDaily.

[ top ] Output for MEF files

In case of MEF files, scores are displayed per detector. A second part of the report repeats this information, sorted by detector ("detector score report").

In the score report, one can configure the arrangement of detectors (NCOL, NROW) and add aggregate parameters AVG (average of that QC1 parameter across all detectors) and RMS (scatter). Here are a few examples:

CRIRES (MEF=4, NROW=1, NCOL=6: 4 detectors + AVG + RMS)
mean_flux HC

HAWK-I (MEF=4, NROW=2, NCOL=3: 4 detectors + AVG + RMS)
qc_atx0 HC
qc_zpoint_peak HC

VIRCAM (MEF=16, NROW=4, NCOL=5: 16 detectors + AVG + RMS)
qc_darkmed HC
qc_darkrms HC
qc_ron12 HC
qc_stripe_rms HC

The exact arrangement of symbols is controlled with the configuration key DET_VALUES (see here). The product NCOLxNROW defines the matrix in which you then list, starting in the upper left corner, all possible detector IDs, the additional symbols AVG and RMS, and if necessary the empty placeholder EMPTY, until you reach the lower right corner. In the above examples, both CRIRES and HAWKI do not require EMPTY entries but VIRCAM does (positions defined by row 3, column 4 and row 4, column 4).

Detector score report. The second report part ("detector score report") is added for convenience. It contains the same information as the parameter report but grouped per detector. This helps to identify problems with detectors:

Detector score report
Scores are collected per detector.

[ top ] Output: explore button

ScoreQC has the explore button. This is a feature that opens a form that is managed by a php script (under If you click on 'explore' below, you see the form opening where you select a QC1 parameter for research. Then the form opens (disabled here but you can click here for the real form and a real 'explore' interface).

Score report HELP
setup:		1_1_2pts/225kHz/lg_7+1FIB_1_580.0 
time range: 	2011-12-14 ... 2012-03-13 

AB | ALOG | PLOG | QC1_plotter | factsheet 

1. Parameter score report
Scores are sorted per QC1 parameter.
Point your mouse on QC1 parameter name for short documentation.
Grey squares: unscored parameter. The large orange square links to a dynamic plot for all detectors.
Smaller squares link to AVG and RMS values (if configured).
HC plot(s): MOS_FFLamp_Stability_L | U
1st QC report & coversheet: QC| COVER
Score data: details ...
score result: 0/8 best: 0/8
powered by QC [scoreQC v1.7]

Once you select a QC1 parameter, the php script displays a table with plot parameters fixed by and read from the score report (the QC1 parameters, the setup parameters, and the detector options if MEF). The score thresholds for the first detector are displayed, and the MJD-OBS corresponding to the score report:

Fixed plot parameters: QC1 table:   uves_fib_fflat_scoreQC
Plot items:
xaxis:   mjd_obs
yaxis:   fib_max
Setup keys (de-select to plot all values):
  MEF data
Select a chip:        
[Thresholds apply to first detector.]
bin:   1x1
det_read_speed:   2pts/225kHz/lg
slit_num:   1
lambda_central:   5800
Special (displayed in blue if selected):
score thresholds  
(first detector):  
8.0, 8.0
marked mjd (in blue):   55954.49673190

Below that table there is a second table extracted from the well-known QC1 plotter interface:

from: to:
y_min: auto OR value:
(choose 'auto' to include ALL data points)
y_max: auto OR value:
(choose 'auto' to include ALL data points)
average: none mean median least-square
special: civil dates

You can use this interface for plotting data sets related to the score report, e.g. for a selected time range, selected detectors, plot ranges or setup parameters. It offers plotting capabilities without the full capacity of the QC1 interface, but with much higher convenience.

[ top ] Output: investigate the configuration

The option -I[nvestigate] is a useful feature when you need to modify the configuration file config.scoreQC. For a given AB, the origin of all QC1 parameters is investigated. Each line in config.scoreQC that would relate to the particular RAW_TYPE and the QC1 parameter is displayed:

QC1 param or
matching line number
setup string DET thresholds or trendPlotter configuration  
1_1_2pts/225kHz/lg_6FIB_._520 DET=U 4000.0 14000.0  
828/119: 1_1_2pts/225kHz/lg_7.*_1_580 DET=U HC: config.tp_MOS_LAMPS_Stability_LAMP3_OD+FF_REDU_DHC #4 hc_fact:1 thresh:5000,9500
fib_effic_abs 1_1_2pts/225kHz/lg_7.*_2_580 DET=U 5000.0 9500.0  

with the line marked in red that was used for the actual scoring. The following parameters are displayed:

the QC1 parameter,
the setup configuration,
the detector,
the lower and upper threshold (if static)
the name of the trendPlotter configuration file, the report number, the hc_factor (if any), and the configured thresholds (if dynamic).

For the matching line, its line number is displayed in blue, so that you can navigate directly to that line once you open your editor. If the configuration is dynamic, the corresponding line in the trendPlotter configuration file is marked in green.

The output is an html file that is generated only on demand and therefore resides in $TMP_DIR/score_investigate.html. It is linked to the AB monitor (link is called score_investigate). You call

scoreQC -I -a <ab_name>

and then click on that link, from any current local AB monitor. Find an example here.

[ top ] Threshold definition.

There are 3 options to configure score thresholds:

- case 1: specified thresholds, configured in config.scoreQC by numbers ("spec"),
- case 2: dynamic thresholds, configured within a user-provided script ("dynamic"),
- case 3: HC result thresholds, taken from trendPlotter result files ("static").

Specified thresholds. This is the classical way. Thresholds are specified in config.scoreQC, per QC1 parameter, detector and setup value. A typical line from that part of the configuration file looks like:

QC_PARAM RAW_TYPE QC1 parameter setting detector THRESH1 THRESH2 comment (optional)
QC_PARAM SCI_POINT_ECH_RED qc_nord 860.0 DET=U 18 19 # any outlier indicates an extraction problem

This means: For raw_type SCI_POINT_ECH_RED, QC1 parameter qc_nord, setting 860.0 and detector U, scoring will be 0 if the value is between 18 and 19, otherwise 1. This parameter is the number of orders, and this check has been set up to control the spectral format. If qc_nord is outside the specification, something went wrong dramatically.

Dynamic thresholds. You could also configure a dynamic thresholding:

QC_PARAM RAW_TYPE QC1 parameter setting detector THRESH1 THRESH2 comment (optional)
QC_PARAM SCI_POINT_ECH_RED qc_nord 860.0 DET=U pgi_QCNORD_U # dynamic thresholding

The executable pgi script pgi_QCNORD_U must reside under $DFO_BIN_DIR. It is used to calculate thresholds dynamically (e.g. as function of slit width) and return it to the main tool as "echo $THRESH1 $THRESH2". Find an example of a dynamic scoreQC plugin in the distribution. The pgi script is invoked from scoreQC with the parameter $SETUP (configuration section 2.3), which can be used within the plugin for calculating the thresholds. Also, the AB name is exported and could be used to extract further information if necessary.

The main power of dynamic thresholding is in the case of many settings to be configured separately (typical for spectroscopy). Rather than listing thresholds for each possible central wavelength, you would rather like to determine/calibrate once and forever a function giving you qc_nord for any input wavelength, calculate lower and upper thresholds (e.g. +/- 10%) and then use this for scoring. Also, you could create your own lookup table and search it with your plugin script, rather than writing all this into a single config.scoreQC file.

Static thresholds. This is the way of having scores based on the most recent results from the HC plots. The configuration looks as follows:

QC_PARAM RAW_TYPE QC1 parameter setting detector THRESH1 THRESH2 HC_REPORT HC_PINDEX HC_FACTOR comment (optional)
QC_PARAM FIBER_FLAT qc_nfib L682.2 DET=1 HC HC DARK 1 1 (optional) # static scoring based on HC results

If THRESH1/2=HC, the scoring is based on the content of the file $DFO_TREND_DIR/report/statFiles/stat_text_<HC_REPORT> (stat_text_DARK in this example). That file is maintained by trendPlotter and contains the result table (which is also displayed in the HC reports). These thresholds depend on the trendPlotter configuration, e.g. VAL=<thresh1>,<thresh2>, or PER=<val>). Effectively the scoring is then enforced to be consistent with the trendPlotter configuration.

Score results. These are best seen in the output text file. Each QC1 parameter has at least one output line, e.g.:

QC_PARAM det score threshold string db_average db_number
qc_nord det=U score=0 thresh1=12,val=13,thresh2=13 db_avg=13 db_num=82
qc_nord det=U score=0 thresh1=12,val=13,thresh2=13 db_avg=13 db_num=82
qc_nord PARAM_SCORE=0

Here we actually have two lines for the two detectors. Each detector gets its score, per QC1 parameter. The sum of these is PARAM_SCORE.

Further down in the scoring log there is a section "Detector scoring" which groups all scores per detector. It delivers the detector score, DET_SCORE.

Both kinds of aggregate scores can be used to find non-zero scores, and to see whether they are related to a specific detector, or to a QC1 parameter.

Finally there is the total scoring section. All score values are added to give the TOTAL_SCORE. There is also the TOTAL_NUMBER of scored values (n_detector x n_parameters) which is used to evaluate the significance of a TOTAL_SCORE = 0 result. The BEST_SCORE line gives the best possible result for that particular raw_type etc. Both TOTAL_SCORE and TOTAL_NUMBER are displayed and evaluated by getStatusAB to mark the AB score on the AB monitor.

Rules for threshold configuration. The configuration of thresholds is the core of scoreQC. An attempt has been made to make it as precise as required, and at the same time as comfortable as possible.

Specification of the thresholds is made per MATCH_KEY which is a string matching partly or completely the setup string. This is the string as extracted from the AB, RAW_MATCH_KEY section. You can specify any string which is then used to match (via 'egrep') the setup string. Furthermore, the order of the configuration is followed by the tool: whenever the first match is found, it is used. Finally, you can specify to use the configured string (e.g. U_BESS), or to not use it (e.g. ^U_BESS) which leads to application to all non-matching setup strings), or to match any string values (ANY).

Note the special meaning of '^' (negation) in this context!

Some examples:

QC_PARAM FFLAT mean_width Medusa1 DET=1 5. 6.2

Here all Medusa1 settings are thresholded the same way (spec scoring). This is the simplest case.

FFLAT mean_width Medusa1_L543.1 DET=1 5.2 5.6
QC_PARAM FFLAT mean_width Medusa1 DET=1 5. 6.2
#all others

Here a HC setting (Medusa1_L543.1) is checked first to match, and thresholded more stringent than all others (spec scoring). Note: If configured in reversed order, the HC setting would never be matched since always the more generic string would match.

Another option with the same result would have been to configure "^Medusa1_L543.1" or "ANY" in the second line.

FFLAT mean_width Medusa1_L543.1 DET=1 5.2 5.6
QC_PARAM FFLAT mean_width Medusa1 DET=1 pgi_meanwidth
#all others

Here the first HC setting is scored (spec.) as in the previous example, but all other settings are dynamically scored with the user-provided tool pgi_meanwidth.

4. Dynamic HC configuration
FFLAT flux Medusa1_L543.1 DET=1 HC HC FFLAMP 1 1 (optional)
# take HC thresholds
QC_PARAM FFLAT flux Medusa1 DET=1 5.2 6.0 # all others

Here the HC setting gets scored with the HC results (stat), while all others are scored against fixed thresholds. Note that HC_FACTOR is optional and is only needed to compensate a COMPUTE_RULE (as applied by trendPlotter).

5. Configuration for MEFs
SCI_POINT_ECH_RED wlen_start 520.0 DET=L 4140 4145 red_lower detector
QC_PARAM SCI_POINT_ECH_RED wlen_start 520.0 DET=U 5170 5216 red_upper detector

Case of multi-extension files (MEFs): similar syntax applies for the DET specification. Either you specify DET=ANY, or DET=<value>.

[ top ] Factsheet.

Each score report has a link "factsheet" to a separate HTML page collecting information about the QC procedure and reports used. The factsheet is specific for each data raw type. Its content is controlled in the QC_DOCU section of the configuration. The factsheet has three main sections:

There are also links to the instrument and pipeline manuals.

The factsheets are created by scoreQC on the fly and go to http://qcweb/<instr>/factsheets/factsheet_<RAW_TYPE>.html. Check out an example here.

Factsheets are designed to give the same look and feel as the pages maintained by qcDocu. Together they form the "QC documentation system".


Scoring report as HTML file: <ab_name>.html; scoring text file: <ab_name>.tlog (both in $DFO_AB_DIR).

These products are catched by moveProducts and finally stored in $DFO_LOG_DIR, as the ABs, pipeline logs etc. The HTML scoring report is archived within the LOGS ancillary files.

The tool writes score results into the database table qc1..qc1_score.

Dynamic QC1 plots. The score squares link to dynamic calls of QC1 plots. These have the current mjd_obs marked as blue filled dot.

HC reports. Configure links to HC plots in sect 4.1 of the config.scoreQC file, under the tag HC_PLOT. The score report then links to the corresponding HC plots. This section is mandatory and important: it should contain all reports on the HC monitor, and is used by autoDaily to find the HC reports that should be updated after a new AB has been pipeline-processed, QC processed and scored. Then the list of the HC reports from this configuration, selected using the RAW_TYPE from the AB, is used to update the plots and the quick-look score versions. Find this section in the config file here.

How to use

Type scoreQC -h for on-line help, scoreQC -v for the version number,

scoreQC -a <ab_name>

to launch scoring for a specific AB. Call

scoreQC -a <ab_name> -I

to investigate the configuration for that AB.


scoreQC -T <tmp_dir>

for a specific $TMP_DIR (useful for debugging), otherwise a subdirectory will be created during runtime and deleted afterwards.

You can call the tool on the command-line. The standard use is to embed it into your QC procedures, right after the qc1Ingest call.

scoreQC can only be called for ABs which are still in the certification stage: ABs in $DFO_AB_DIR, products in $DFS_PROD_PATH. Same conditions apply as for calling the QC procedures.

Note that there is a local scoreManager file ($DFO_TREND_DIR/reports/scoreManager/scoreManager.html). This is an interactive GUI to manage scores. Here you can even score ABs which are already stored in $DFO_LOG_DIR. This file is maintained by scoreHC, find more information there.

[ top ] Configuration file

Note: If you are using multiple configuration files, the general information for the scoreQC tool is taken from section 1 of config.scoreQC. Section 4 is read from config.scoreQC. Sections 4.1 and 4.1A of config.scoreQC are also read from scoreHC and need to be complete, always.

The information of sections 2 and 3 is read from the specific config files (as specified in section 1 of the generic config file).

If you use the CONFIG_PGI mechanism, you have to provide two complete sets of config files for the respective scoring situations, and use the pgi for switching. The PGI must use the exported variables $AB and $JTMP_DIR (!) to support parallel execution.

config.scoreQC defines:

Section 1: general

size of graphical symbols

50 MEF: size of detector table in pixels
YELLOW_SCORE 1 maximum score for yellow color (0 displays green, anything above YELLOW_SCORE is red)

format description for HTML detector table for MEF; CRIRES: 4/1; VIRCAM: 4/4; OCAM: 8/4; HAWK_I: 2/2 etc.
MEF_NCOL 1 number of columns for detector table (default: 1)
MEF_NROW 1 number of rows for detector table (default: 1)
UPDATE_YN YES YES: update AB status in AB monitor (default, makes always sense in incremental dfos operations)
NO: turn update off (makes sense in massively parallel QC processing like PHOENIX)
SHOW_ICON YES | NO YES: show the (downsized) QC plot as icon (default: NO); NO: show text (better for many or large QC plots)

YES: data come from SEF files but logically are MEF (VIMOS or FORS2)

if YES, their PROD_ROOT_NAME from the AB is made 'fuzzy', multiple mjd_obs values are replaced by the last one.

ALL_DETECTORS YES | NO YES (default): display orange all-detector symbol (MEF only)
NO: suppress (in exceptional cases)

Optional keys to define multiple config files; they are always read from the main config file.

MULTIPLE_CONFIG YES | NO NO by default; YES: multiple config files supported
CONFIG_PGI pgi_scoreQC optional key, for MULTIPLE_CONFIG=NO; to be provided in $DFO_BIN_DIR; must use exported variables $AB and $JTMP_DIR (!) to support parallel execution
CONFIG_STRING &&sed "s/X/Y/g"&&

If MULTIPLE_CONFIG=YES: string to define the distinction between the different cases.
It must be based on RAW_TYPE which is read by the tool from the AB; you can then further edit the RAW_TYPE using this CONFIG_STRING to distinguish the cases. Assume that your code is concatenated to
'cat $AB | grep RAW_TYPE | awk '{print $2}' | ${CONFIG_STRING}'
The string must come embedded in &&...&&, no chars permitted after final &&

Make sure to have a single, unique VALUE returned which then is used to match the proper config file in the next section.

Define the config files based on the returned VALUE of CONFIG_STRING, or use the string 'other' (lower-case) to mark the case of no match:


CONFIG_CASE VALUE config file name (any name, no strict convention)
CONFIG_CASE other config.scoreQC_OTHER (could also be config.scoreQC)

Section 2: QC1 database configuration

2.1 QC1 table names

RAW_TYPE BIAS as read from the AB
QC1_NAME giraffe_bias name of QC1 database table, must be unique per RAW_TYPE
RANGE 90 time range in days (backwards from TODAY) for qc1db queries: this is the time range displayed in the dynamic queries (e.g. 90 or 180)
TMODE CURRENT | HISTORY mode for range determination (optional)
CURRENT: total range is TODAY-RANGE; this is the default
HISTORY: total range is MJD+/-(RANGE/2) where MJD is read from the AB; this makes sense for historical data only (reprocessing)!
ALLOW_INCOMPLETE YES | NO for MEF only, optional: if YES, enforce correct scoring if number of QC1DB entries is lower than expected
2.2 detector configuration (MEF only)
For MEF files, list here the name of the extension/detector parameter, and all possible values, including the aggregates (like AVG and RMS); use EMPTY as placeholder if required
DET_ID chip name of extension parameter, as stored in QC1 database (e.g.: chip, quad etc.)
DET_VALUES CHIP1 all possible values for DET_ID, as existing in the QC1 database
  CHIP2 etc.  
  EMPTY (as placeholder for an empty position)  
2.3 Mapping of setup keys (FITS name vs. qc1db name)
This is needed to translate FITS key names (read from the AB) into corresponding QC1 database column names
AB_SETUP INS.SLIT.NAME setup key name in AB (section RAW_MATCH_KEY)
ins_slit_name setup column name in QC1 database
QC1_TYPE char | number data type of QC1_COLUMN (char or number)
QC1_MODIF &&awk '{print $1*10}'&& or
&&sed "s/DICHR#/DI/"&& etc.
optional string, with commands to modify setup values (to match them with qc1db entries which may come in a different format than the fits key values)
strings must come embedded in &&...&&, no chars permitted after final &&

3. Configuration of scoring
Per RAW_TYPE, list all QC1 parameters to score, as function of setup values and detector_id values, and the static thresholds to be used for scoring

Typically, there is one group of QC1 parameters per RAW_TYPE; combination of RAW_TYPE and QC1 parameter name must be unique; at least one line per QC1 parameter is required (ANY), several lines are supported (useful for fine-tuning of thresholds).

The <specific string> is an expression evaluated by egrep; you can use any substring of the match key. In case of multiple lines, the tool evaluates them one by one; you can go from specific values (e.g. from HealthCheck settings) to more general ones, including ANY at the end which then means "anything else". You can also de-select strings ("^Ks" meaning "anything but Ks")

QC1_PAR median_master etc.  
MATCHKEY value | ^not_value | ANY

MATCHKEY string used to find the setting, composed from the RAW_MATCH_KEY values in AB
value: any 'egrep'-compatible pattern
tool loops across values in configured sequence

DET DET=<value> where value = ANY | <specific value> extension specification; must be DET=1 (single-extension), or match the DET_VALUE definition;
a DET value not specified results in scoring failure.
Thresholds: case1 (spec. scoring)
THRESH1 number lower threshold value
THRESH2 number upper threshold value
Thresholds: case2 (dyn. scoring)
<name of pgi_script in $DFO_BIN_DIR> pgi_script is used to dynamically to calculate THRESH1 and THRESH2
Thresholds: case3 (static scoring)
THRESH1 HC sets thresholding to the limits defined in the trendPlotter health check report
THRESH2 HC sets thresholding to the limits defined in the trendPlotter health check report
HC_REPORT name of HC report which is linked to this score as given by REPORT_NAME in trendPlotter's config_tp_<report_name>
HC_PINDEX plot index within HC_REPORT as given by P_INDEX in trendPlotter's config_tp_<report_name>
HC_FACTOR (see comment below) optional factor to compensate for COMPUTE_RULE if any is used, the HC_FACTOR is the inverse; default is 1 which can be omitted

4. Configuration for HC and dynamic QC1 plots

4.1 HealthCheck plots

#HC_PLOT This section should be a complete list of all HC reports that should be updated by autoDaily if an AB has created new QC1 parameters and new scores.
Note that this section must be included and is monitored for completeness on the scoreManager. In case of multiple config files, this section must be included in config.scoreQC.
RAW_TYPE BIAS as above
HC_TAG HC_BIAS displays as title if mouse over link (must be one word!)
HC_URL GIRAFFE/reports/HEALTH/trend_report_DARK_HC.html you can add multiple links per RAW_TYPE, line by line; all assume as root

4.1A This section required for MULTI_CONFIG=YES and multiple RAW_TYPEs per HC_PLOT


Configure here those trendPlotter REPORT_NAMEs which have different RAW_TYPES for different plots. As an example, UVES has reports with mixed plots from the BLUE arm and the RED arm. Then you need to specify here for each report the name of the proper config.scoreQC file.

The following keys come as columns, one per plot:

REPORT_NAME BIAS as defined in the trendPlotter config file config.tp_BIAS
P_INDEX 1 same
RAW_TYPE BIAS_BLUE applicable RAW_TYPE for plot #1
CONFIG_NAME config.scoreQC_blue applicable scoreQC config file

4.2 Dynamic QC1 plots

NOTE: This section has been simplified by dropping PLT_AVG and PLT_SIG. They were defined as columns #4 and #5, while the former columns #6 and #7 (YMIN and YMAX) should replace them. This task is done automatically by scoreQC, and the user is informed via email to check and fine-tune the automatic result.

These are QC1 plotter calls, which create a dynamic QC1 plot with the current mjd_obs data point(s) marked blue

RAW_TYPE BIAS as above
QC_PARAM median_master QC1 parameter to plot (multiple per RAW_TYPE supported)
YMIN AUTO | <number> QC1 plotter y_min (supported: <number> or AUTO (default))
YMAX AUTO | <number>

QC1 plotter y_max (supported: <number> or AUTO (default))


4.3 QC_DOCU part: link between RAW_TYPE and QC report documentation

NOTE: for all three, have only one entry, take first/most relevant one in case of multiple choices
As DOCU_EXAMPLE, take arbitrary example from current data stream.

Entries are arranged in a table per RAW_TYPE, with the following columns:

RAW_TYPE BIAS as above
DOCU_LINK bias.html name of corresponding HTML page under<instr>
QC_TUTORIAL bias_qc1.html name of corresponding tutorial QC page under<instr>/qc
could be "none"
DOCU_EXAMPLE 2010-01-11/r.GIRAF.2010-01-12T08:41:15.061_tpl_0000.fits.gif date/name of example QC report under<instr>/plots

Links to the instrument manual, and to the pipeline manual:
INSTR_MANUAL link to the instrument manual
PIPE_MANUAL link to the pipeline manual

* NB. If a HC_FACTOR is defined (other than 1), then the associated trendPlotter report configuration file (config.tp_<report_name>) must also include a notice of this compute rule. Therefore, in section 3.2 of config.tp_<report_type>, you should add the column CR_COMPENSATE YES (default = NO)

[ top ] Operational aspects

Multiple config files. These are supported since v2.0. They are required for special cases when an instrument has several arms which differ fundamentally in MEF parameters (MEF_NCOL, MEF_NROW, ALL_DETECTORS, YELLOW_SCORE, DET_ID, DET_VALUES). Currently this is the case for UVES (BLUE arm: one detector; RED arm: two detectors) and SPHERE. One might also want to use this feature for other cases, when there is no difference between these parameters but one wants to improve readability of the configuration and may want to distinguish config files by mode etc. Make sure to always have the parameters MULTIPLE_PARAMETERS, CONFIG_STRING and CONFIG_CASE in the main config file config.scoreQC. Also, the HC_CONFIG section must be included there. The other parameters could be stored in the case-specific config files.

In exceptional cases, if more flexibility is needed (like FORS2), you could use the CONFIG_PGI which is supported only if MULTI_CONFIG=NO (they exclude each other for logical reasons). The pgi has to be provided by you. It should decide about the proper config file to choose, based on information available in the AB other than the RAW_TYPE (since that is already evaluated by the MULTI_CONFIG=YES mechanism).

Hierarchy within dfos. With scoreQC, the following levels of QC and certification information are provided by dfos tools:

1. highest: AB monitor (per DATE and MODE), created by getStatusAB; giving you an indication of the total AB score

2. AB level: scoring log (<ab>.html) and scoring text file (<ab>.tlog); giving you an indication of the individual QC and detector scores, plus dynamic QC1 db links, plus coversheet link

3. QC report: this is the traditional QC report with all the detailed boxes, displays etc. you create with your QC procedures

This hierarchical approach is thought to implement "information on demand": if all lights are green, you trust the system. If you find non-green cases, you can go to lower levels until you see all details you need to assess the products.

Choice of QC1 parameters and thresholds. The choice of QC1 parameters depends on experience. The "art of scoring" is to find, per raw_type, a set of parameters and thresholds which is not overly restrictive (giving you lots of false alerts) and not overly relaxed (then missing a problem). This is based on experience.

This system is designed for automatic certification, focusing on outliers (non-green cases).

Workflow description

1. Read info from AB

2. Prepare
2.1 Check for MULTIPLE_CONFIG:

2.1.1 if YES, decide about the proper config file and read all info from there
2.1.2 if NO, check for CONFIG_PGI; if found, execute

2.2 Read config information for RAW_TYPE
2.3 Map QC1 parameter names and qc1_db names
2.4 Find DET_IDs if MEF files are scored

3. Query qc1_db to have current and trending values
3.1 Check for multiple entries; give warning if more than one is found; take latest
3.2 Find qc1_db values for current products
3.3 Find qc1_db values for similar products in configured time window (CURRENT - RANGE), for dynamic plots
3.4 Query qc1_columns for comments

3.5 Prepare tlog and html log

4. Create factsheet

5. Do the scoring
5.1 Loop over QC1 parameters:

5.1.1 prepare HTML structure
5.1.2 prepare dynamic qc1_db calls
5.1.3 Loop over DET values Loop over configured setup values, distinguish value | ^value | ANY evaluate query results, get statistics (SUM, AVG) find threshold values (from configuration, or from configured plugin, or from HC statistical result files) thresholds found: do the scoring no thresholds found: skip write score and related info into tlog MEF: add DET=ALL score write result into qc1..qc1_score

5.1.4 End of DET loop

5.2 End of QC1 parameter loop
close the QC1 parameter HTML table
5.4 add detector scoring
5.5 add HC plot link(s)
5.6 add coversheet link(s)

6. MEF only: add the second (DET) table

7. Add entry in AB (scored ...)

[ top ] Hints and Pitfalls (thanks to Mark for this section!)
  1. COVERSHEET. For QC reports to be displayed on the Parameter score report page as a coversheet, the QC report must have the FULL product name (e.g. r.SINFO.2008-10-10T11:02:00.226_tpl_0000.png).
  2. Types of QC1 parameters. Section 2.3 of config.scoreQC maps the set up keys (MATCH_KEY) in the AB to those used in the QC1 database. Make certain that their type (char, number, etc.) matches the type set in your QC1 database.
  3. Setup definitions in config.scoreQC (I). You are bound to the setups as defined in the MATCH_KEY section in the AB. Try to avoid finding special characters there (like '+', '/' etc.) since they might have a special meaning for 'egrep' which is used here. If this is not possible:
  4. Setup definitions in config.scoreQC (II). The tool evaluates score rules sequentially. When you want to configure different score rules for different setups, you should go from specific scores to more general scores (or simply be complete if this is possible). For instance, you could specify scoring for a specific setup (since it is fed daily as a Health Check setup), and then use ANY with more relaxed thresholds for all other setups.
  5. Listing of QC1 parameter names. In the QC_PARAM section, always group together all entries per combination of RAW_TYPE and QC1_PAR. This is important for MEF instruments. E.g., group together all 16+2 entries for a given VIRCAM RAW_TYPE and QC1_PAR. If you group, for whatever reason, only 16 of them together and the 'avg' and 'rms' values somewhere else, you may end up in multiple repetitions of the score symbols in the HTML output.

[ top ] Technical aspects

The php script for the 'explore' interface is created by scoreQC, with option -P. Check out more. You need authorization by the Group Head, unless in emergency. Follow the dialog which is self-explaining.

The php script is named score_cgi.php and resides under In that directory there is also a README file with more explanation. You need to login to stargate1 as qc and 'cd qc/ALL/php' in order to read it.

Last update: April 26, 2021 by rhanusch