Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: See also :
 

v1.3:
- support coversheet file _cs.html
v1.4:
- dropped: INFO messages about PROCESSING_LOG and SCORE_RESULT
- content changes: no PROCESSING_LOG for SCIENCE; no SCORE_RESULTS, no nightlog extracts
v1.4.2:
- dropped: consistency checks for ANC files (see bottom of page)

- the tool is distributed through the utilities package (utilPack)
[ used databases ] databases PANL2 for nightlog information; asm (thru qcm) ; dataproducts..calib for pre-existing ANC files
[ used dfos tools ] dfos tools fitspack; qcm
[ output used by ] output twin ancillary fits files (LOGS and PLTS)
[ output used by ] upload/download none
 

merge2Fits

Description

merge2Fits is a helper tool which prepares all input for creating product ancillary fits files. These files are effectively a wrapper around multiple log files, or plot (PNG, GIF, JPG) files, designed to be stored in NGAS. That mechanism offers three advantages:

The reason for ingestion of product files currently is mainly backup. There might be future use of the calibration products in the calSelector associations.

With the ancillary fits files, QC is able to ingest three types of product fits files with one standard mechanism (ingestProducts/cdbIngest):

what where NGAS internal name starts with
calibration products $DFO_CAL_DIR/$DATE M.
science products $DFO_SCI_DIR/$DATE S. [terminated as of 2011-10-01]
ancillary files: logs $DFO_LOG_DIR/$DATE A.
QC plots $DFO_PLT_DIR/$DATE

How to use

Normally this tool is called by moveProducts. You can also call it on the command line.

Type merge2Fits for on-line help.

Type

merge2Fits -a <ab> -d <date>

to create a pair of ancillary fits files, one with the logs for the specified AB, the other with the QC plots.

Ancillary fits files: names

The ancillary fits files have the following name convention, automatically enforced by merge2Fits:

<MCAL_CODE>_<LOGS/PLTS>_<AB_root without instrument part>.fits, e.g.

GI_LOGS_2008-09-12T23:20:35.043_tpl.fits and
GI_PLTS_2008-09-12T23:20:35.043_tpl.fits

Note that 'LOGS' and 'PLTS' are reserved tags and cannot be used for any other product name!

Ancillary fits files: header structure

Ancillary FITS files follow the FITS Foreign File Encapsulation Convention (see http://fits.gsfc.nasa.gov/registry/foreign.html).

They have a primary array with the primary header, and multiple extensions, one per input file. You can see the complete header by typing

dfits -x 0 GI_LOGS_2008-09-12T23:20:35.043_tpl.fits

The primary header has, among other keys, HIERACH.ESO.PRO.ANC = T, which is required for cdbIngest to recognize it as ancillary file, plus INSTRUME and MJD-OBS, plus PRO.CATG="ANCILLARY_FILE", plus some other PRO keys.

Each extension has EXTNAME = <original file name> (e.g. GIRAFFE.2008-09-12T23:20:35.043_tpl.ab for the AB in the above example); PRO.CATG, set to something meaningful (e.g. ASSOCIATION_BLOCK). [Note that fitsort will deliver you only the last value found. If you want to see all PRO.CATGs in a given fits file, use 'dfits -x 0 ...fits | grep "PRO CATG"' .]

Ancillary fits files: content

The content of ancillary fits files is controlled by merge2Fits. The following cases apply:

1. LOGS

PRO.CATG extension content when? source created by?
ASSOCIATION_BLOCK .ab association block always createAB
ASSOCIATION_LOG .alog association log always createAB
PROCESSING_LOG .rblog pipeline/recipe log if processed pipeline
SCORE_RESULT .html scoring result if processed and scored scoreQC
PROCESSING_COMMENT .cmt extraction from NOTES files (about processed data) if mode=CALIB certifyProducts

2. PLTS: only if existing

PRO.CATG extension content when? source created by?
QC_PLOT (multiple possible) .png QC plot(s) in png format if existing QC procedures
QC_PLOT (multiple possible) .gif QC plot (s) in gif format if existing QC procedures
QC_PLOT (multiple possible) .jpg QC plot (s) in jpg format if existing QC procedures
COVERSHEET _cs.html coversheet to multiple QC plots if existing scoreQC

Ancillary files: ingestion and retrieval

Ingest an ancillary file with cdbIngest: cdbIngest <anc_file_name>. Operationally, use ingestProducts for this task (it automatically scans for calib, science, and ancillary products).

Retrieval of an archived ancillary file is done with ngasClient -c <anc_file_name>, just like retrieval of any other fits product file. Use fitsunpack afterwards to split it into its components.

Installation

The tool comes as part of the utilPack delivery.

Configuration file

none.

Operations

- In standard operations, this tool is wrapped into moveProducts.

[Turned off with v1.4.2:
- The tool automatically checks for previous versions of the just created LOGS and PLTS files in NGAS. If found, it warns you and then overwrites the older versions. Overwriting is implemented as combination of:

This is enforced since otherwise there might be inconsistencies between the archive backup and the local disk content. ]