Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO

new:

see also:

v3.0:
- supporting DEEP mode

v3.1.1:
- egrep syntax supported in PRODUCT_MARKER
v3.1.4:
- adapted to PHOENIX COMPLETE_MODE=YES

databases
dfos tools
output	$DFO_MON_DIR/FINISHED
upload/download	upload to qcweb (monitor/FINISHED); on SLAVE accounts, also scp to MASTER

PHO
ENIX

phoenixMonitor

Description

This tool creates and maintains a historical record of the PHOENIX IDP processing for a given instrument.

The phoenixMonitor tool is an adaption of the histoMonitor tool to the needs of the IDPs. Its output and workflow closely follows the histoMonitor.

Check out the IDP sites here: UVES | XSHOOTER | GIRAFFE | MUSE

Here is a list of differences with respect to the histoMonitor pages):

The output has no run (package) history, except for DEEP modes.
The output has an additional statistics page, with monthly number of created products (IDPs or MCALIBs).
The monthly output pages have two columns for the ingestion flags; for IDPs: one for the conversion, one for the IDP ingestion; for MCALIBs: one for renaming, one for product ingestion. In DEEP mode, these columns also exist.
The monthly output pages list only the SCIENCE ABs (for MCALIBs: only the processed CALIB ABs). For the DEEP mode, the pages are organized by run ID.
The column for the statistics on the monthly pages lists numbers for configurable instrument modes (e.g. for UVES IDPs: Echelle-point, Echelle-extended, Slicer, MOS; for GIRAFFE mcalibs: BIAS, DARK, FLATs, WAVE).
The linked AB monitor lists only the ABs relevant for the particular PHOENIX project (IDPs: science ABs; MCALIBs: selected CALIB ABs).
The column ABs lists setup info (configurable).
The last column 'comments' is not automatically taken over from the operational histoMonitor for that instrument, but includes daily comments specific for this PHOENIX project.

How to use:

The tool is automatically called by phoenix.

It can be called on the command line in the standard ways (-d, -m, -y). If called with -y, or without any parameter (total refresh), the user is asked for confirmation before actually starting. For DEEP mode, the tool is called with parameter -P.

Configuration file

config.phoenixMonitor defines:

Section 1 SCI_SCALE and SCI_MAX are identical to the histoMonitor. The other keys are specific. 'SCI' refers actually to main products, so it is also applicable to MCALIB projects.
SCI_SCALE	5		scaling factor for calculating the width of the products statistics bar
SCI_MAX	20		maximum number of product files to display by scale; The HTML width of the cell will be: - n_science * SCI_SCALE if n_science below SCI_MAX - SCI_MAX otherwise
SCI_CLASS	ECH_POINT		the 'CLASS' to which the bar refers (see below)
CHECK_CONV	YES		YES\|NO: check for CONVERTED log file (from conversion tool); default: YES
MARK_VIRTUAL	YES		mark ABs containing VIRTUAL products (default: YES)
MAX_TOTAL_NUMBER	<instr>	1000	mark bold if monthly number of fits files is higher than this threshold
PRODUCT_MARKER	<instr>	UV_SRED	string to mark and properly count IDP fits files, for statistics on main page; egrep syntax is supported like "GI_SIDP\|GI_SOBS", for cases with more than one possible identifier
			(the string should be chosen to reflect the primary product, avoiding intermediate or ancillary products)
The following two keys make sense if the MASTER/SLAVE model is used, and if this is a MASTER account:
HAS_SLAVE	YES		YES\|NO (NO); if YES, the tool checks for the marker PROCESSED_SLAVE in $DFO_SCI_DIR and thereby recognizes data content as coming from the SLAVE account (which is then marked by dark square).
SLAVE_ACCOUNT	muse_ph@muc09		indicates the SLAVE account
Section 2 Specific parameters for overview statistics
Multi-column configuration: CLASS	class of IDP/MCALIB product (short tag, self-explaining in the instrument context) # sequence does matter
PROC_INSTRUMENT	# as in config.phoenix
DECORATION	# some short tags to give this section some structure
DEC1	# single character (e.g. E for ECHELLE), or 'none'
DEC2	# / (slash) or 'none' (for structure)
DEC3	# BOLD or 'none' or 'CALIB' if this class is CALIB but should be processed as SCIENCE
DEFINITION	# sequence of piped grep statements, enclosed in '&&' (last char must be &)
#CLASS	PROC_INSTRUMENT	CLASS	DECORATION: DEC1	DEC2	DEC3	DEFINITION
CLASS	UVES	ECH_POINT	E	/	BOLD	&&grep ECHELLE \| egrep "OBJECT,POINT\|OBJECT" \| grep -v SLIC \| egrep -v EXTENDED
etc.
Section 3 Versions of AB_list_ALL files 3.1 SETUPS The AB lists are read in order to provide a display of setups which might be useful on the phoenixMonitor (e.g. display central wavelengths for UVES). The way they are displayed in the AB_lists may have changed over time. That evolution can be configured here. Check out $DFO_MON_DIR/FINISHED/$Y/$YM/AB_list_ALL_$DATE to see the format for a given date. It is identical to the format that was operational at that time.
Multi-column configuration: SETUP
PROC_INSTRUMENT	# as in config.phoenix
EPOCH	time string until which the described rule was valid (-lt); configure 'CURRENT' for the currently valid format
RULE	corresponding rule (enclosed by &&...&&, last character has to be &&)
#SETUP	PROC_INSTRUMENT	EPOCH	RULE
SETUPS	UVES	2007-04-01	&&grep ECH \| awk '{print $3}' \| sed "s/_/ /g" \| awk '{print $1}' \| sed "s/\.0//"&&
SETUPS	UVES	CURRENT	&&grep ECH \| awk '{print $3}' \| sed "s/_/ /g" \| sed "s/CD..//" \| awk '{print $6}' \| sed "s/\.0//"&&

PGI_SETUP	GIRAFFE	<name>	optional pgi to cope with more complex setup situations; if defined, provide it in $DFO_BIN_DIR; called right after the SETUPS rules above

3.2 SPECIAL You may want to mark the column 'ABs' for some special setups, data types etc. For instance, wide_slit data in UVES have a typical quality problem so the setups containg _WS are highlighted in red. The RULE is applied to the AB_list. Multiple entries are supported.
Multi-column configuration: SPECIAL
PROC_INSTRUMENT	# as in config.phoenix
TAG	short tag used for highlighting, must be unique
RULE	rule for that tag, usual syntax
#SPECIAL	PROC_INSTRUMENT	TAG	RULE
SPECIAL	UVES	WS	&&grep "_WS"&&
SPECIAL	UVES	ABS	&&grep "SCI_POINT_ECHABS_RED"&&

SPEC_TEXT: some explanation in the monthly overview table about the SPECIAL markings, HTML coded, enclosed by &&
#SPEC_TEXT	PROC_INSTRUMENT		SPEC_TEXT
SPEC_TEXT	UVES		&&<font color=red>WS</font>: wide_slit settings; <font color=red>ABS</font>: SCI_POINT_ECHABS_RED data (absorption cell)&&
4. For DEEP mode only
IS_DEEP	YES		YES: this is the account for a DEEP mode (optional, default: N)
DEEP_EXIST	YES		if YES, a link to a DEEP release is offered (only if IS_DEEP = N)
DEEP_RELEASE	MUSE_DEEP		if YES, this is the release name
FIRST_PERIOD	P60		marks first period (used for link on main monitor)
FIRST_ABSTRACT	http://eso.org/sci/activities/vltsv/musesv.html		special URL for first-period abstracts (if SV)
DEEP_CONTENT	DEEP_CUBES		tag used to link the $DEEP_RELEASE pages to the normal release

Notes:

- The phoenixMonitor web site is exported to qcweb.hq.eso.org/~qc/$RELEASE/monitor/FINISHED/phoenixMonitor.html that is visible ESO internally only;
- it is called automatically after call_IT -m <month> in $MONTH mode;
- it is called automatically after phoenix -d <date> (in DEEP mode after phoenix -r <run_ID>);
- if the dual-instance processing is implemented (for MUSE), the phoenixMonitor on the MASTER account also has the processing history of the SLAVE account included, marked by a dark-grey square on the daily overview page; in that case, the phoenixMonitor on the SLAVE account has only the dates processed by the SLAVE account.

Last update: April 26, 2021 by rhanusch

Common DFOS tools: Documentation

How to use:

Configuration file

Common DFOS tools:
Documentation