Common Trending and QC tools:

tqs = Trending and Quality Control System
*make printable   new: see also:

pyqc.tar v2.1 (June 2018)


- description of library QClib
- pyQC script structure

- step-by-step installation tutorial
- repository of example and tutorial scripts
[ used databases ] databases none
[ used dfos tools ] dfos tools processQC, writeQC, scoreQC
[ output used by ] output used by certifyProducts
[ upload/download ] upload/download none

pyQC: Python QC tools

files description version date download
pyqc.tar complete installation 2.1 2018-06-12 download
QClib library with generic classes and functions 1.5.5 2018-06-12 download
plot modules instrument-independent module plotting cuts through 2D frames, histogrames, etc. 2.1.1 2018-06-12 download instrument-independent plotting module similar to but with more flexibility 1.0.2 2014-04-04 download instrument-independent module creating 2D images from a single detector (raw frame, master, comparison to reference) 1.3.3 2014-04-08 download instrument-independent module for detector monitoring
(needs also configuration
1.2.6 2016-02-19 download instrument-independent module creating a simple 2D image from a raw or product file; does not need additional configuration 1.5.4 2014-12-17 download instrument-independent module creating an overview of all (or of a selected number of) extensions in a raw or product file; does not need additional configuration 1.1 2017-03-20 download instrument-independent module for plotting flux growth curves of skylight flats 1.0 2014-05-20 download
repository of general example and tutorial scripts


QC1 procedures are used to generate graphical reports of pipeline products, to calculate additional QC1 parameters that are not provided by pipeline recipes, and to control ingesting and scoring of QC1 parameters (by calling writeQC and scoreQC). They are invoked by processQC within the DFOS workflow. Due to the uniqueness of each instrument (and also due to historical development), each instrument has its own set of QC1 procedures and different scripting languages are in use. On the other hand, there are several tasks in each procedure that are common to other procedures from the same instrument or other instruments. There are also some common datatypes (mainly bias, dark, and flat frames) which do not differ much from instrument to instrument.

The pyQC tools are an offer for Python programmers. They aim at:

Overview of concept

PyQC has three main components. On the lowest level, there is a collection of Python functions and classes (QClib) that offer standardized procedures for typical tasks (reading an AB, opening raw and product frames, plotting a cut through a 2D image, etc.). This library is imported into plot modules which typically produce a figure with several sub plots. The plot modules are coded so that they can be used for different raw types and also for different instruments; raw-type- and instrument-dependent definitions are contained in dedicated configuration files. Finally, there is typically one wrapper script per raw type that imports plot modules and executes them. The interaction between the components is illustrated in the following figure:

Wrapper script 1: RAW_TYPE_1
Configuration file
for RAW_TYPE_1
Plot module A Plot module B
Conf file 1 QClib Conf file 1 QClib

Wrapper script 2: RAW_TYPE_2
Configuration file
for RAW_TYPE_2
Plot module A Plot module C
Conf file 2 QClib Conf file 2 QClib

The above example has wrapper scripts for two raw types. There is one configuration file associated with each raw type. The first wrapper script uses plot modules A and B; the second wrapper also uses module A but adds functionality from module C instead of B. All modules use code from QClib.


The installation pyqc.tar offers the following Python files:

1. QClib a library of Python functions and classes for QC an example library of additional instrument-specific Python functions and classes

2. Plot modules:
creates a figure per detector with cuts through raw and product frames, histograms, etc. Useful for master frames (dark, bias, flat). Instrument-independent. Example plot VIMOS.
Plot 6 can be configured so that it contains either col/row averages (as before) or a 2D image. Example plot.
similar to but with more flexibility regarding plot configuration. Instrument-independent. Example plot.
visualization of detector monitoring raw frames and product tables (gain and linearity); instrument-independent. Example plot.
creates 2D images from a single detector (first raw frame, master, comparison to master). Instrument-independent. Example plot.
creates a 2D image from a single raw or product frame. Instrument-independent; can be called from the command line with some options; does not require additional configuration. Example plot.
creates an overview from all (or a selected number of) extensions of a single raw or product frame. Instrument-independent; can be called from the command line with some options; does not require additional configuration. Example plots: KMOS master dark, KMOS master flat, KMOS illumination correction.
creates flux growth curves for a sequence of skylight flats. Requires some additional configuration which can be put in the same configuration file as for qc_mstplt. Instrument-independent. Example plot: KMOS sky flats.
creates one figure with 2D images of all 4 VIMOS detectors. VIMOS-specific example script. Example plot.

All plot modules can be used standalone or can be called from wrapper scripts.

3. Wrapper scripts:
example for putting all pieces together. Imports and calls,, and

4. Configuration files: contains the connection between raw types and configuration files. example configuration file (from VIMOS) for using the above scripts. configuration file for

How to use

See also the step-by-step tutorial.

1. Download the tar file pyqc.tar. Unpack it into a dedicated directory of your dfo machine (e.g. into <HOME_DIR>/python). All files stay in the same directory.

2. Copy and edit configuration files: there is typically one configuration file per raw type. It is also possible to have several configuration files per raw type that are split by instrument setting (see The connection between raw types and configuration files is given in The latter file must keep its name.

3. The scripts expect the usual DFOS environment. ABs are expected to be in <DFO_AB_DIR>, raw and product files must be in the places specified in the AB. The AB name is given as command line option when calling a script, e.g.:

<HOME_DIR>/python/ -a <AB> .

Output is written as png files into the current working directory. Typical command line options accepted by the scripts are:

--version   print version number and exit
-h, --help   print help on command line parameters and exit
-a <AB>   name of AB [required]
-i, --ingest   ingest QC1 parameters using writeQC, may require additional coding [optional, only for wrapper scripts]

4. For making the scripts operational, add their calls to your config.processQC.

More details and support material

More details on pyQC: description of classes and functions within the library QClib and an outline of the structure of wrapper scripts and plot modules.

The official Python documentation. External packages that are frequently used: matplotlib, NumPy/SciPy, PyFITS.

Editing of scripts and own additions

Feel free to use and edit pyQC scripts and modules as you like. If you make changes to instrument-independent plot modules like it is recommend to make a copy (e.g. and edit that copy. If you would like to make changes to it is recommended to copy the affected functions or classes to an instrument-specific library file (e.g. and to import this module in your scripts after qclib.

Suggestions for additions to are highly welcome if they are of general interest and can be coded intrument-independently. The same is true for changes to and other modules and for additions in the form of new modules.


Thanks to Mark and Wolfgang for starting to code QC procedures using Python and to John for testing pyQC with HAWK-I.

Send comments to <>
Last update: June 13, 2018