Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:
  v3.3:
- enabled for MCAL ingestion for instruments that are operational with qcFlow
  new config key WITH_QCFLOW
- called by moveProducts
- dpIngest information
- how to ingest general calibration products
 

v3.2:
- PHOENIX only: new optional config key ENABLE_UPDATES
- up to 3 attempts if ingestion fails due to NGAS error related to disk space


[ used databases ] databases qc_products, to verify successful ingestion; qc1 for updating file names
[ used dfos tools ] dfos tools dpIngest; fitsverify
- off-line calls collected in JOBS_INGEST and managed by dfoMonitor
- status of ingestion is displayed on the dfoMonitor (open: yellow circle; done: green ok)
 
[ output used by ] output log files list_ingest and list_qc1update in $DFO_LST_DIR
[ output used by ] upload/download ingestion into NGAS

ingestProducts

Description

This tool ingests pipeline products into the NGAS archive, after they have been certified, distributed, and renamed. Calibration products (master calibrations) as well as science products (IDPs) are supported. The tool ingests all, or selected, products from the product directory. The product directory is $DFO_CAL_DIR/<DATE> for master calibrations, and $DFO_SCI_DIR/<DATE> for IDPs.

The tool is enabled to ingest IDPs (internal data products, being science products generated by QC through the phoenix process). This feature is documented in more detail on the PHOENIX branch of the DFOS documentation here.

With version 3, the tool is enabled to ingest master calibrations created by the PHOENIX process. The only difference to the normal (operational) ingestion of master calibrations is the configurable behaviour how to handle previously existing instances. This is documented in more detail on the PHOENIX branch of the DFOS documentation here. If you are not using the tool for a PHOENIX project, you can safely ignore this part.

The tool is normally called from the ingestion file under $DFO_JOB_FILE that is maintained by moveProducts (this is the usual operational setup for larger data sets). It can also be called from the command line.

The tool writes a log file list_ingest_$MODE_$DATE.txt into $DFO_LST_DIR. The monitor tools dfoMonitor and histoMonitor check this file and display whether or not a certain night with calibration products has already been ingested. (Note: versions earlier than v1.2 had the log file called list_ingest_$DATE.txt.)

It uses the DFS tool dpIngest that comes with the standard DFS installation. If the configuration key QC1_UPDATE is set to YES in the configuration file, the key cdbfile in the tables of the QC1 database is updated with the new name of the calibration product (since it has been renamed from the PIPEFILE format to the CALIBFILE format in the meantime). Find more information about dpIngest here.

If configured, the tool will send an email if errors or warnings are found in the log file.

The product data are ingested into NGAS. Their metadata description is extracted from the header keys and is stored in the qc_products database (qc_files).

In order for dpIngest to work properly, two entries in $HOME/.dbrc are required:

QC_PRODUCT_HANDLER                              acdb.hq.eso.org:2025 <user> <pass>
KEYWORDS_REPOSITORY     asestage.hq.eso.org:7500   <user> <pass>

Also, you need to make sure that the fitsio tools, in particular fitsverify, are in your $PATH. Type 'which fitsverify', and in case of 'not found', enter something like

export PATH=${PATH}:/opt/cfitsio/bin                        

in your $HOME/.qcrc.

cleanupProducts. The tool writes, in both modes (CALIB and SCIENCE), a call 'cleanupProducts -d $DATE' into the job file $DFO_JOB_DIR/JOBS_CLEANUP that is monitored by dfoMonitor and ready for execution after finishNight has been executed. Find more information about cleanupProducts here.

JOBS_CLEANUP:
cleanupProducts -d 2010-04-16
cleanupProducts -d 2010-04-17

Output

The output from dpIngest (messages about successful ingestion or errors) can be found in $DFO_LST_DIR/list_ingest_$MODE_$DATE.txt. The QC1_UPDATE function sends an email warning if an ERROR has been found. (The lists list_qc1update_$MODE_$DATE.txt have been terminated with v1.5.2 and can be removed.)

Error handling

The tool scans the log file $DFO_LST_DIR/list_ingest_$MODE_$DATE.txt for error messages from dpIngest and for unusual content. It reports about occurrences such as WARNING or ERROR before finishing. It also sends an email if SEND_MAIL=YES in the configuration file (highly recommended!). Error and warning messages must be carefully observed and understood in order to ensure proper ingestion of files.

The following error message from dpIngest is especially worth mentioning:

All other messages are likely caused by problems with connecting to the archive. It must be checked that files have been successfully ingested.

Example

ingestProducts -m CALIB -d 2004-10-12 ingest master calibrations for that date
[ingestProducts -m SCIENCE -d 2013-10-12 ingest IDPs, if PHOENIX enabled ]
[ingestProducts -m CALIB -d 2003-10-12 ingest MCALIBs, if PHOENIX and MCALIB enabled ]

How to use

Type ingestProducts -h for on-line help, ingestProducts -v for the version number, and

ingestProducts -m CALIB -d 2004-10-12

to ingest all (or specified) pipeline products for mode CALIB on that date into the product archive.

Type

ingestProducts -m CALIB -d 2004-10-12 -f

to call fitsverify only on the fits files in $DFO_CAL_DIR/<date>, and verify their fitsverify compliance. No other effect. If the tool returns an error (exit code != 0), you may want to investigate the files with an error, by following the instructions. In case the fitsverify error is a real one, and not just a warning, you should create a pipeline ticket, with the exact description of the error, to get the file structure fixed.

For MCALIB projects under PHOENIX, type

ingestProducts -m CALIB -d 2003-08-02 -D

to call the tool in DEBUG mode, with interactivity and confirmation of the important steps of deletion of old instances and ingestion of new instances. See here for more.

Configuration file

config.ingestProducts defines:

Section 1: general
INGEST_SWITCH ALL | SELECT | DESELECT ALL: ingest all products found
SELECT: ingest the products with pro_catg matching the values in section 2
DESELECT: ingest the products with pro_catg not matching the section 2 values
QC1_UPDATE YES | NO update the cdbfile key in QC1 database tables (default: YES)
SEND_MAIL YES | NO YES: an email is sent if ERRORS or WARNINGS are found in the log file.
WITH_QCFLOW YES | NO YES: usage of dpIngest option -q. Only for instruments that are fully operated with qcFlow.
# special config keys for PHOENIX:
PATH_TO_IT find details here  
CONVERTER    
ENABLE_UPDATES    
# parameters defining the stream
PROGRAM_NAME find details here  
COLLECTION_NAME    
RELEASE_TAG    

Section 2: List of PRO_CATGs

This is an optional list of PRO_CATGs that defines the products to be ingested (INGEST_SWITCH=SELECT) or not to be ingested (DESELECT).

PRO_CATG MASTER_BIAS  
PRO_CATG MASTER_FLAT  

Section 3 for phoenix-generated master calibrations only:
Definition of DELETEs of pre-existing masters

find details here

PHX_DELETE PRO_CATG INS_MODE

Status information and logging

The tool writes the DFO status cal_Ingested or sci_Ingested.

Operational aspects


Last update: March 22, 2024 by bwolff