Common DFOS tools:

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: Note:

- new column 'pipefile'; mark multiple entries

- supporting new mode -S (static calibrations)

[ used databases ] databases qc_metadata..qc_products; qc1_daily_comments; ngas..ngas_files
[ used dfos tools ] dfos tools ngasClient dpDelete histoMonitor phoenixMonitor
[ output used by ] output $DFO_MON_DIR/productExplorer.html and $DFO_MON_DIR/productExplorer1.html
[ output used by ] upload/download none



This is a helper tool for exploring and managing the content of the qc_metada..qc_products database. This database stores all QC-ingested product fits files (calib, science: until 2011-10, ancillary: until 2014-05). It offers a user-friendly way to query ("explore") its content, and fine-tune the query. There are options to download the results or to delete entries.

It launches a direct isql query to qc_metada..qc_products.

It displays its result in a html table which can be sorted and filtered in the standard way, plus an executable download file, plus a deletion file for ingested products (to prepare for ingestion of an improved version).

It offers a way to explore the content of static calibrations. These are calibration products which (in most cases) have no parent raw files and cannot be easily found with the normal date-oriented query.

How to use

Type productExplorer -h for on-line help, productExplorer -v for the version number,

productExplorer -d 2017-10-15

to explore calib for that date and the default $DFO_INSTRUMENT,

productExplorer -d 2017-10-15 -D 2007-10-30 -i CRIRES

for the specified time range and instrument,

productExplorer -d 2017-01-01 -p LINE_TABLE

for the specified pro_catg value only, and

productExplorer -d 2017-01-01 -p LINE_TABLE -s grism_256

to filter from the result set those files with setup value "grism_256". This filter option is done after the isql download, at the unix level, for performance reasons.

To launch a query for static calibrations, you need to know its product category (pro_catg) since there is no flag 'static calibration' in the database. Then, you can launch a query using the above options but replacing the date by the flag -S ('static'):

productExplorer -S -p FLUX_STANDARDS


productExplorer -S -p REFERENCE_DISP_SOLUTION -s Medusa1_L385.7nm

if you want to constrain the setup.

In the "static mode", the tool may take much longer (since there is no date constraint). Also you may run into files which do not follow the standard naming convention (since they were ingested before the convention), or into multiple instances of the same file, carrying different origfile names (see below, "multiple entries", and the operational notes). The tool checks for these cases one by one, before it does the final setup filtering (if specified). Be prepared for a longish log file which you may want to analyze.

Click [info for static files] on the result page to get some more information.

Result HTML file. The result file comes as an HTML file, similar in format to the (raw) data reports created by createReport. It uses color-coding to distinguish calib (green) from science (blue) files. It also supports ancillary (log) fits files (displayed yellow). In standard (date) mode, it displays daily comments just like the raw data reports.

Product report for 2008-03-13 (products ingested into NGAS)

number: 77
size (MB): 115.1 3.2 8.8 127.1
Date comment: Last night of old CCD Bruce; IFLATs taken; STD efficiency
[read query] | [read download file] | [read deletion file]
type* origfile setup** pipefile mjd_obs pro_catg ins_mode dp_id
VIRTUAL V.GIRAF.2008-03-14T02:29:04.977_005.fits   NULL 54539.163178 SCIENCE_RBN_SPECTRA MED V.GIRAF.2008-03-14T02:29:04.977_005.fits
SCIENCE GI_SRBS_287119_2008-03-14T02:29:04.977_IFU2_L543.1nm_o3.fits IFU2_L543.1nm_o3 r.GIRAF.2008-03-14T02:29:04.977_tpl_0001.fits 54539.103530 SCIENCE_RBNSPECTRA IFU S.GIRAFFE.2008-04-01T09:28:10.770
CALIB GI_PIME_080313A_Argus_L427.2nm_o4_LTT3218.fits Argus_L427.2nm_o4_LTT3218 r.GIRAF.2008-03-14T10:39:42.5009_tpl_0006.fits 54539.163178 STD_RCERRORS ARG M.GIRAFFE.2008-04-01T09:00:16.159

Filter and sort. The HTML table can be filtered (any string from the content can be used). You can filter e.g. to display only MBIA files, or files containing the string U_BESS etc. Sorting can be done by any column.

Virtual products. Virtual products (defining the calSelector datasets) are displayed. You can toggle between displaying and hiding them, nearby the 'Filter' option.

Multiple entries. In particular static calibration files could have been entered multiple times with different origfile names. These files, as being identified by the combination of pipefile, pro_catg and mjd_obs, are marked in red font color.

Download. The files can be downloaded, by using the executable file $TMP_DIR/mcalibDownload, either as it stands or after further editing. It uses ngasClient for the download. All downloads go into $DFO_CAL_DIR/DOWNLOAD and can then be further distributed by the user.

Deletion. The tool supports deletion of products. It provides the results of the query in a text file, with the proper syntax prepared for deletion of these files. However, without further editing, deletion will be performed only in simulation mode. Only if the user edits the deletion file to the full syntax, e.g.

dpDelete GI_MBIA_...fits -force

these files will get deleted. Deletion means physical removal and deletion in all database tables. Deletion is not hiding: after deletion, you cannot reconstruct the file content from information stored anywhere in NGAS. You can, however, ingest a new version of this file from your local file system, and this is the main purpose of the tool.

Obviously, the deletion mode should be used with care.


Use dfosInstall.

Configuration file


Operational aspects

Last update: April 26, 2021 by rhanusch