make printable	new:	see also:
	v1.5: - HdrDownloader called in separate directory	- output of this tool is operationally relevant for Paranal QC! - the tool is closely linked to calChecker
	Note: To enable calChecker for display of the tellTracker output page, you must set CHECK_TELLTRACK to YES in its config.calChecker file.	databases sched_rep for SM_VM flag dfos tools called fitsreport, HdrDownloader, getObInfo; output linked to calChecker output HTML output exported to qcweb upload/download download: twilight information from skycalc upload: HTML output exported to qcweb javascript javascripts jquery-latest.js, jquery.tablesorter.js, jquery.dataTables.js under ${JSCRIPT_URL}
topics: description | rules | results: issues overview enhancements all_matches | update | performance | output | how to use | configuration | workflow

tellTracker

You may also want to read the information linked to the public result page (here) and the general calChecker home page.

Description

The tellTracker monitor is a plugin to the calChecker monitor. It tracks the completeness of time-critical nighttime calibrations, like telluric standard stars, for all science OBs of the indicated date.

This plugin is only enabled for certain instruments with red or infrared spectroscopy. When enabled, you find it linked to the main calChecker output like this:

Rules

For all configured science data types, a matching telluric standard star is searched for. Matching is based on header information only. Unlike calChecker, the tellTracker accepts matches only for the same night. No quality information can be taken into account. This means that for instance a saturated telluric standard star cannot be recognized as such and will be used by the tool for a formally correct match.

After evaluation of the formal matches, two constraints are evaluated:

• the time constraint,
• the airmass constraint.

The configured values are indicated. Typical values would be 0.0875 (2 hr) for the time constraint, and 0.2 for the airmass constraint.

All configured science files are evaluated. This is generally a subsection of all science files measured during the night, because:

• there might be data types that are not configured for tellTracker (e.g. XSHOOTER UVB: they do not require a telluric standard star);
• science files with 60./060./0060.x PROG_IDs are suppressed.

Results

Results are displayed in terms of flags, in the last column RESULT. Note that tellTracker analyzes each single SCIENCE file (while calChecker analyzes ABs, regarding only the first file in a template). Like calChecker, the tellTracker has three flags, OK, NOK, MISS:

• OK is displayed if both constraints are fulfilled.
• NOK is displayed if a match is found but at least one constraint is violated (time and/or airmass). The violation is marked yellow, the compliance is marked green.
• MISS is displayed in red if no match could be found in the night. Matches to other nights are not considered.

NOK or MISS flags are further analyzed in the last column. Here, a label LEFT+79 indicates that 79 minutes are still left (defined by the time constraint) for taking a new telluric standard which would then improve the NOK or MISS flag to OK. A label LOST-13 indicates that the time window has already expired, but still it might be reasonable to take a standard star, e.g. in order to improve a MISS to a NOK case. A label LOST (without a number) indicates that there is no way to improve the result since the night is over.

If only one constraint is violated (combinations 'OK NOK' or 'NOK OK') preference for display is given to the time constraint, i.e. if there is one 'OK NOK' and one 'NOK OK' record, the former is displayed and evaluated.

During the night, there is in the update section some information regarding the remaining time until morning twilight starts.

If there is an unrecoverable issue for the current last night, a flag ! is displayed on the tellTracker tab.

Issues. An issue is defined as a NOK or MISS case. All issues, if any, are collected in the top part. That table is intended to provide a list of telluric standards to be taken. It is aggregated per combination of OBS_ID and setup. If a certain science OB has two different setups (e.g. XSHOOTER: 0.9x11_NIR and 100k/1pt/hg_0.9x11_VIS) and a total of 10 science files, there will be two entries in the issues section, one for the NIR and one for the VIS setup. If there are three OBs with two different setups with missing telluric calibration, there will be 6 entries in the issue table since the OBs might belong to different pointings and therefore to different airmass values.

If within an OB there are some files with 'OK NOK' and some with 'NOK OK', preference is given to the 'NOK OK' cases (time constraint violated but airmass ok) or to 'NOK NOK' since these are also found by calChecker. The 'OK NOK' issues (meaning AIRMASS is violated but time is ok) are also displayed as such but only if there is no 'NOK OK' issue.

The issues section is split into two parts, one for the recoverable issues (NOK+LEFT; MISS+LEFT; MISS+LOST) and one for the non recoverable issues (NOK+LOST). The first section should be used for scheduling the next telluric standard, the second section for OB comments or grades in the nightlog. The issue table has columns LST, RA and DEC to be used for input in the telluric star tool. The column UT is a short version of the ARCFILE column in the overview.

Overview. The overview table is complete for all science files of the night. It has the same columns as the issues table, plus a few more like ARCFILE, DATE, and DATA_TYPE. This table has the science and, for orientation, also the telluric standard stars listed. The latter objects are linked to their score page (if that is available at runtime).

The tables are sortable. Press Shift key for multiple column sorting. Default sorting is by UT (issues) and ARCFILE (overview). Restore default sorting with the browser refresh button.

Select SETUP and then ARCFILE to sort all TELL and science files per setup.

Table enhancements. The table output has features which go beyond HTML functionality and are provided by javascripts. Such enhancements are:

• All keys of the overview and issue tables are sortable. This enhancement is provided by the java library jQuery, in particular by the javascript tablesorter.
• The overview table also has a filtering enhancement, provided by the javascript dataTables, also being part of the java library jQuery.
• Some columns of the overview table can be toggled.

Filtering and sorting can be combined and can be useful for research on an issue. E.g. enter '50333' and see all files belonging to that OB_ID. Enter'TELL04' and see all files associating to that telluric star, including the TELL file itself. Then sort the output by AIRM, or dAirm. Do not forget to disable browser refresh for longer analysis (and to enable it again afterwards).

More technicalities about dataTables here.

Find an example output table here (click on any column to sort, enter a string to search/filter, refresh the browser to get back the original content):

Toggle: LST+RA+DEC | AIRM | DATA_TYPE | all of those

PROG_ID	M	OBS_ID	GRD	LST	RA	DEC	AIRM	ARCFILE	DATA TYPE	SETUP		RESULT FLAG	TELL	dTime	dAirm	remain
086.B-0407(A)	SM	501007	A!	08:06:59.936	09:59:21.9	02:06:40.6	1.24	XSHOO.2011-03-27T00:32:16.538.fits	SCI_SLIT_VIS	100k/1pt/hg/1x2_0.9x11_VIS		OK	TELL01	0.041	0.02
086.B-0407(A)	SM	501007	A!	08:32:32.119	09:59:21.6	02:06:37.0	1.19	XSHOO.2011-03-27T00:57:44.787.fits	SCI_SLIT_VIS	100k/1pt/hg/1x2_0.9x11_VIS		OK	TELL01	0.023	0.03
085.D-0144(A)	SM	442926	A	13:57:15.318	13:20:44.8	00:18:53.3	1.12	XSHOO.2011-03-27T06:21:34.766.fits	SCI_SLIT_VIS	100k/1pt/hg_0.9x11_VIS		OK	TELL03	0.040	0.07
085.D-0144(A)	SM	442926	A	14:10:08.428	13:20:44.7	00:18:57.6	1.13	XSHOO.2011-03-27T06:34:25.471.fits	SCI_SLIT_VIS	100k/1pt/hg_0.9x11_VIS		OK	TELL03	0.032	0.06
085.D-0144(A)	SM	442926	A	14:22:55.523	13:20:44.7	00:18:57.6	1.15	XSHOO.2011-03-27T06:47:11.025.fits	SCI_SLIT_VIS	100k/1pt/hg_0.9x11_VIS		OK	TELL03	0.023	0.04
085.D-0144(A)	SM	442926	A	14:35:49.637	13:20:44.8	00:18:53.3	1.17	XSHOO.2011-03-27T07:00:02.810.fits	SCI_SLIT_VIS	100k/1pt/hg_0.9x11_VIS		OK	TELL03	0.014	0.02
386.A-0620(A)	SM	503333	A!	15:09:25.140	13:11:30.1	-1:19:20.1	1.25	XSHOO.2011-03-27T07:33:32.648.fits	SCI_SLIT_VIS	100k/1pt/hg/1x2_0.9x11_VIS		NOK	TELL01	0.252	0.03	LOST
386.A-0620(A)	SM	503333	A!	15:16:02.224	13:11:29.9	-1:19:16.7	1.27	XSHOO.2011-03-27T07:40:08.661.fits	SCI_SLIT_VIS	100k/1pt/hg/1x2_0.9x11_VIS		NOK	TELL01	0.256	0.05	LOST
086.B-0407(A)	SM	501007	A!	08:07:04.950	09:59:21.9	02:06:40.6	1.24	XSHOO.2011-03-27T00:32:19.772.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL02	0.041	0.02
086.B-0407(A)	SM	501007	A!	08:32:37.133	09:59:21.6	02:06:37.0	1.18	XSHOO.2011-03-27T00:57:47.840.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL02	0.023	0.04
085.D-0144(A)	SM	442926	A	13:57:21.334	13:20:44.8	00:18:53.3	1.12	XSHOO.2011-03-27T06:21:39.265.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL04	0.040	0.07
085.D-0144(A)	SM	442926	A	14:10:13.442	13:20:44.7	00:18:57.6	1.13	XSHOO.2011-03-27T06:34:28.382.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL04	0.032	0.06
085.D-0144(A)	SM	442926	A	14:23:00.537	13:20:44.7	00:18:57.6	1.15	XSHOO.2011-03-27T06:47:14.131.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL04	0.023	0.04
085.D-0144(A)	SM	442926	A	14:35:54.650	13:20:44.8	00:18:53.3	1.17	XSHOO.2011-03-27T07:00:05.866.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL04	0.014	0.02
386.A-0620(A)	SM	503333	A!	15:09:30.154	13:11:30.1	-1:19:20.1	1.25	XSHOO.2011-03-27T07:33:35.688.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL04	0.010	0.06
386.A-0620(A)	SM	503333	A!	15:16:07.238	13:11:29.9	-1:19:16.7	1.28	XSHOO.2011-03-27T07:40:11.615.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL04	0.014	0.09
386.A-0620(A)	SM	503333	A!	15:22:42.317	13:11:29.9	-1:19:16.5	1.30	XSHOO.2011-03-27T07:46:45.547.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL04	0.019	0.11
386.A-0620(A)	SM	503333	A!	15:29:19.401	13:11:30.1	-1:19:18.9	1.32	XSHOO.2011-03-27T07:53:21.472.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL04	0.023	0.13
386.A-0620(A)	SM	503333	A!	15:35:57.488	13:11:30.1	-1:19:19.5	1.35	XSHOO.2011-03-27T07:59:58.768.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL04	0.028	0.16
386.A-0620(A)	SM	503333	A!	15:42:34.572	13:11:29.8	-1:19:15.4	1.38	XSHOO.2011-03-27T08:06:34.692.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL06	0.027	0.11
386.A-0620(A)	SM	503333	A!	15:49:11.656	13:11:29.9	-1:19:16.0	1.41	XSHOO.2011-03-27T08:13:10.617.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL06	0.022	0.08
386.A-0620(A)	SM	503333	A!	15:55:48.740	13:11:30.1	-1:19:19.3	1.45	XSHOO.2011-03-27T08:19:46.541.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL06	0.018	0.04
386.A-0620(A)	SM	503333	A!	16:02:26.827	13:11:30.1	-1:19:19.0	1.49	XSHOO.2011-03-27T08:26:23.132.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL06	0.013	0.00
386.A-0620(A)	SM	503333	A!	16:09:03.912	13:11:29.9	-1:19:15.8	1.53	XSHOO.2011-03-27T08:32:59.762.fits	SCI_SLIT_NIR	0.9x11_NIR		OK	TELL06	0.008	0.04
60.A-9022(C)		200193515	X!	09:05:47.568	10:08:48.4	06:43:02.8	1.22	XSHOO.2011-03-27T01:30:54.734.fits 0/6	STD_TELL_SLIT_VIS	100k/1pt/hg/1x2_0.9x11_VIS			Hip049704		TELL01
60.A-9022(C)		200193515	X!	14:55:35.876	14:38:48.2	07:54:39.8	1.19	XSHOO.2011-03-27T07:19:46.149.fits 0/6	STD_TELL_SLIT_VIS	100k/1pt/hg_0.9x11_VIS			Hip071615		TELL03
60.A-9022(C)		200193515	X!	16:21:01.872	12:47:53.6	-24:51:04.5	1.49	XSHOO.2011-03-27T08:44:57.927.fits 0/6	STD_TELL_SLIT_VIS	100k/1pt/hg/1x2_0.9x11_VIS			Hip062448		TELL05
60.A-9022(C)		200193515	X!	09:05:52.582	10:08:48.4	06:43:02.8	1.22	XSHOO.2011-03-27T01:30:58.135.fits 1/3	STD_TELL_SLIT_NIR	0.9x11_NIR			Hip049704		TELL02
60.A-9022(C)		200193515	X!	14:55:41.892	14:38:48.2	07:54:39.8	1.19	XSHOO.2011-03-27T07:19:49.558.fits 1/3	STD_TELL_SLIT_NIR	0.9x11_NIR			Hip071615		TELL04
60.A-9022(C)		200193515	X!	16:21:06.886	12:47:53.6	-24:51:04.5	1.49	XSHOO.2011-03-27T08:45:01.320.fits 1/3	STD_TELL_SLIT_NIR	0.9x11_NIR			Hip062448		TELL06
PROG_ID	M__	OBS_ID__	GRD__	LST	RA	DEC	AIRM__	ARCFILE	DATA TYPE	SETUP__		RESULT FLAG	TELL	dTime	dAirm	remain

Table with all matches. This additional output table provides a list of all possible TELL matches. It is thought to provide details in case of NOK issues. For instance, the main page lists the best possible NOK-OK match (dTime violated but dAirm ok), while the all-matches table also lists the other telluric matches, including the ones flagged OK-NOK. Find its link here: [all matches].

Update information and documentation

Last update: timestamp of creation of this page (UT)

Last header: name of last header file currently available; compare this timestamp to 'Last update' to see if monitor is up-to-date

Frequency: The tool is scheduled by a cronjob and is running during the Paranal night, from about 22:00 UT until about 11:00 UT, every 15 minutes.

Performance

As a specialized tool, tellTracker is tuned for good performance. Instead of OCA, it uses simple pattern matching (unix grep) for classification and association (like the long-term calibration section of calChecker). All configured keys are read with fitsreport into a single table which is then analyzed for data types, matches and setups. tellTracker typically executes in less than a minute. Its execution times are written in the QC1 database table exec_time and monitored for the case of XSHOOTER here.

Output

• exported tellTracker_$DATE.html under http://qcweb.hq.eso.org/<instr>/CALCHECK/TELLTRACK/tellTracker_$DATE.html
• local as $TMP_DIR/CALCHECK/tellTracker.html

How to install

This tool is not part of the standard dfos tool suite. Download the installation tarball from http://www.eso.org/~qc/dfos/ or directly from this link into $DFO_INSTALL_DIR/tellTracker. Then:

cd $DFO_INSTALL_DIR/tellTracker
tar xvf tellTracker.tar
cat README_tellTracker
mv tellTracker $DFO_BIN_DIR

The following should be done only if this is a new installation (not for updates!):

mv config.tellTracker tellTracker.cfg tellTracker_radec.cfg $DFO_CONFIG_DIR

Read the README file and follow the instructions. They are standard and straightforward.

How to use

Type tellTracker -h | -v for on-line help and version number. Type

tellTracker

to create the output (per default for $TODAY).

tellTracker -d(ate): for a specified date

tellTracker -N: ($TODAY, no header download)

tellTracker -d(ate) -O: (specified date, with header download)

tellTracker -V(erbose): verbose output (otherwise the tool is silent)

The verbose output goes to the terminal. It displays all matches per science file and may be useful for analyzing issues.

Note that for $TODAY the default behaviour is header download (with HdrDownloader), while for older dates the default is no header download (since those header directories are assumed to be complete anyway).

Configuration files

The tool has

• its own config file config.tellTracker, under the normal $DFO_CONFIG_DIR
• two config files in the fitsreport style, to download header keys
• one optional info file under $DFO_CONFIG_DIR

1. The tool configuration config.tellTracker defines:

2. Two configuration files are needed to define the classification and association keys, as well as the keys for the output table. One is called tellTracker.cfg, the other one tellTracker_radec.cfg. The latter file contains RA, DEC, and LST which are always used and need a fixed format and conversion. This file does not need any attention by the QC scientist. The other file comes as a fitsreport config file with minimum content. It has two parts, the first part being a standard part not to be modified by the user, and a second part for the instrument-specific setup definition:

As usual with fitsreport config files, you can e.g. define 'Delimiter' strings to make selection of these keys by unix grep easier. See more in the fitsreport documentation.

3. There is an optional but recommended info file, info.tellTracker under $DFO_CONFIG_DIR. Its text is displayed on top of the page, right part. Its content should be similar to the info file for calChecker.

Workflow

Details:

1. $DATE=$TODAY: download headers (unless disabled) with 'HdrDownloader -d $TODAY' (into $DFO_HDR_DIR/TELLTRACK to avoid interference with operational headers)
2. Classification
   2.1 call fitsreport and classify all files
   2.2 get OB grades, comments, SM/VM flags
   
3. Association
   3.1 read SCI_DATA_TYPE, TELL_MATCH, SCI_DEFSTRING, TELL_DEFSTRING from config.tellTracker
   3.2 loop across SCI_DATA_TYPEs
   3.3 get closest-in-time match of TELL and SCIENCE, calculate deltas, evaluate OK/NOK/MISS
   
4. Create HTML output tables
   4.1 extract issues per OB and setup
   4.2 provide full overview for science
   4.3 add telluric standard stars
   
5. scp result HTML file to qcweb

Operational hints:

- The tool will download headers with 'HdrDownloader -d $TODAY', without suppressing the last (possibly) incomplete template; in order to avoid unwanted effects with autoDaily (incomplete ABs), its headers are downloaded into $DFO_HDR_DIR/TELLTRACK; the call of 'HdrDownloader -d' is incremental, i.e. only new headers are downloaded.

- Make sure to set up the cronjob file properly: call the tool like

01,16,31,46 00-11,22,23 * * * . /home/xshooter/.qcrc; /home/xshooter/bin/tellTracker

- Make sure to call tellTracker before autoDaily: if not it will possibly interfer with filterRaw (as triggered by autoDaily) which means it will search for headers which are temporarily not available.