Common DFOS tools:

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:
-optional non-alphabetical sorting of set-ups in HTML overview pages, incremental update of outdated plots


-added html navigation, output promoted to become reference frames site

- added definition of RAW_TYPEs as read from the OCA rules

- possibility to select individual plots per AB




This tool manages the selection and publication of RAWDISP plots as references. Chosen plots are copied to $DFS_PRODUCT/RAWDISP/REFERENCE/, renamed, and published on the web server at<INSTRUMENT>/reports/RAWDISP/REFERENCE. Thereafter, the reference plots are found by getStatusAB when creating the RAWDISP web pages and are linked to the current plots for comparison. For usage in overview pages, also the thumbnail images are renamed, saved, and published.

The tool can be called from the command line for a specified date after moveProducts. It then expects ABs in $DFO_LOG_DIR/<DATE>. It only offers certified ABs and ABs from pipeline-unsupported modes for selecting reference plots. The RAWDISP plots are expected to be on the operational machine in $DFS_PRODUCT/RAWDSIP/<DATE>/. If this directory was already removed by finishNight, the user is asked whether the needed plots shall be re-created.

The tool is also called automatically at the end of certifyProducts while ABs are still in $DFO_AB_DIR. It then checks whether RAWDISP plots from the just certified ABs are available for settings which are currently without references. It also checks whether new plots are available for references that are older than half a year. If there are RAWDISP plots that fulfil these conditions then the user is asked for confirmation to add these plots to the already existing references.

The naming scheme for the reference plots is


<RAWTYPE>, <DO_CLASS>, and <SETUP> are taken from the RAW_TYPE, RAWFILE, and RAW_MATCH_KEY sections of the relevant AB. Header key names in the RAW_MATCH_KEYs are removed, any existing '/' in the set up is replaced by '_'. <EXT> is the fits extension number (0 for the primary header unit) in the raw file. Newly created reference plots replace automatically any existing references with the same name.


The renamed reference plots exist locally at


and on the web server at<INSTRUMENT>/reports/RAWDISP/REFERENCE/ .

Thumbnail images are locally at



A listing of all reference plots exists as $DFO_MON_DIR/RAWDISP/list_RAWDISP_references. It matches the plots with their original names, raw types, DO classes, instrument set-ups, and DFO dates.

Every time new reference plots have been published, a set of overview HTML pages of all plots is created. The entry page can be found at $DFS_PRODUCT/RAWDISP/REFERENCE/rawfiles_START_REFERENCE.html and at<INSTRUMENT>/reports/RAWDISP/REFERENCE/rawfiles_START_REFERENCE.html and is linked behind 'REF'.

How to use

Type rawdisp2reference -h for on-line help, rawdisp2reference -v for the version number.

The standard usage is

rawdisp2reference -d <DATE>

for selecting plots from <DATE> as references. It may eventually be useful to re-create the RAWDISP plots even in the case that the local directory still exists. This can be enforced with the -F option:

rawdisp2reference -F -d <DATE>

To update the overview HTML pages without adding new reference plots use the option -H:

rawdisp2reference -H

The -I option (incremental update) is normally only used within certifyProducts:

rawdisp2reference -I -d <DATE>

Configuration file

The configuration file config.rawdisp (same as for must exist in $DFO_CONFIG_DIR. It can contain optionally some keywords for fine-tuning the behaviour of rawdisp2reference.

Any number of


lines mark those raw types which RAWDISP plots are never considered as reference plots.

Raw types that shall not be considered when rawdisp2reference is called from certifyProducts can be specified with


This is useful for raw types that are eligible for reference plots but have an infinite number of set-ups (like infrared darks). It is not necessary to repeat the NO_REFERENCE raw types in this section as these are automatically added.

The number of set-ups per table row in the HTML overview pages can be controlled via


The default value is 5 which is typically a good value for SEF instruments. For MEF, this key should be set to 1.

The set-up strings in the HTML overview pages can be controlled via


The default value is "NO" which means that only the values of the header keys are displayed. With "YES" also the keyword names are included which can, however, result in rather long strings. Note that on "mouse over" the keyword names are always included.

Set-up strings are sorted in alphabetical order in the HTML overview pages. An alternative sorting can be configured per raw type:


with <ORDER> being a space-separated list of character strings which must be part of the <SETUP> values (as in the file names). The UNIX shell command 'grep' is then used to select the reference plots. For example


would sort flats in the order UBVRI. (The underscore is part of the set-up string and needed for identifying the set-ups uniquely.) And


would show first all Medusa set-ups, then IFU, and then Argus.


The tool comes as part of the normal dfos installation.

Operational aspects

Removal of existing reference plots

In order to remove existing reference plots, the following steps are needed:

Overview HTML pages

The exported overview HTML pages form the reference frames web site for a given instrument. The page rawfiles_START_ALL.html is linked (via 'REF') for each instrument to the CAL, HC, and QC web sites. The reference pages are updated automatically by the rawdisp2reference tool.

Last update: April 26, 2021 by rhanusch