Common DFOS tools:
|dfos = Data Flow Operations System, the common tool set for DFO|
|make printable||new:||see also:|
|- output of this tool is operationally
relevant for Paranal QC!
- the tool is closely linked to calChecker
Note: To enable calChecker for display of the tellTracker output page, you must set CHECK_TELLTRACK to YES in its config.calChecker file.
|topics: description | rules | results: issues overview enhancements all_matches | update | performance | output | how to use | configuration | workflow|
The tellTracker monitor is a plugin to the calChecker monitor. It tracks the completeness of time-critical nighttime calibrations, like telluric standard stars, for all science OBs of the indicated date.
This plugin is only enabled for certain instruments with red or infrared spectroscopy. When enabled, you find it linked to the main calChecker output like this:
For all configured science data types, a matching telluric standard star is searched for. Matching is based on header information only. Unlike calChecker, the tellTracker accepts matches only for the same night. No quality information can be taken into account. This means that for instance a saturated telluric standard star cannot be recognized as such and will be used by the tool for a formally correct match.
After evaluation of the formal matches, two constraints are evaluated:
The configured values are indicated. Typical values would be 0.0875 (2 hr) for the time constraint, and 0.2 for the airmass constraint.
All configured science files are evaluated. This is generally a subsection of all science files measured during the night, because:
Results are displayed in terms of flags, in the last column RESULT. Note that tellTracker analyzes each single SCIENCE file (while calChecker analyzes ABs, regarding only the first file in a template). Like calChecker, the tellTracker has three flags, OK, NOK, MISS:
NOK or MISS flags are further analyzed in the last column. Here, a label LEFT+79 indicates that 79 minutes are still left (defined by the time constraint) for taking a new telluric standard which would then improve the NOK or MISS flag to OK. A label LOST-13 indicates that the time window has already expired, but still it might be reasonable to take a standard star, e.g. in order to improve a MISS to a NOK case. A label LOST (without a number) indicates that there is no way to improve the result since the night is over.
If only one constraint is violated (combinations 'OK NOK' or 'NOK OK') preference for display is given to the time constraint, i.e. if there is one 'OK NOK' and one 'NOK OK' record, the former is displayed and evaluated.
During the night, there is in the update section some information regarding the remaining time until morning twilight starts.
If there is an unrecoverable issue for the current last night, a flag ! is displayed on the tellTracker tab.
Issues. An issue is defined as a NOK or MISS case. All issues, if any, are collected in the top part. That table is intended to provide a list of telluric standards to be taken. It is aggregated per combination of OBS_ID and setup. If a certain science OB has two different setups (e.g. XSHOOTER: 0.9x11_NIR and 100k/1pt/hg_0.9x11_VIS) and a total of 10 science files, there will be two entries in the issues section, one for the NIR and one for the VIS setup. If there are three OBs with two different setups with missing telluric calibration, there will be 6 entries in the issue table since the OBs might belong to different pointings and therefore to different airmass values.
If within an OB there are some files with 'OK NOK' and some with 'NOK OK', preference is given to the 'NOK OK' cases (time constraint violated but airmass ok) or to 'NOK NOK' since these are also found by calChecker. The 'OK NOK' issues (meaning AIRMASS is violated but time is ok) are also displayed as such but only if there is no 'NOK OK' issue.
The issues section is split into two parts, one for the recoverable issues (NOK+LEFT; MISS+LEFT; MISS+LOST) and one for the non recoverable issues (NOK+LOST). The first section should be used for scheduling the next telluric standard, the second section for OB comments or grades in the nightlog. The issue table has columns LST, RA and DEC to be used for input in the telluric star tool. The column UT is a short version of the ARCFILE column in the overview.
Overview. The overview table is complete for all science files of the night. It has the same columns as the issues table, plus a few more like ARCFILE, DATE, and DATA_TYPE. This table has the science and, for orientation, also the telluric standard stars listed. The latter objects are linked to their score page (if that is available at runtime).
The tables are sortable. Press Shift key for multiple column sorting. Default sorting is by UT (issues) and ARCFILE (overview). Restore default sorting with the browser refresh button.
Select SETUP and then ARCFILE to sort all TELL and science files per setup.
Filtering and sorting can be combined and can be useful for research on an issue. E.g. enter '50333' and see all files belonging to that OB_ID. Enter'TELL04' and see all files associating to that telluric star, including the TELL file itself. Then sort the output by AIRM, or dAirm. Do not forget to disable browser refresh for longer analysis (and to enable it again afterwards).
More technicalities about dataTables here.
Find an example output table here (click on any column to sort, enter a string to search/filter, refresh the browser to get back the original content):
Toggle: LST+RA+DEC | AIRM | DATA_TYPE | all of those
086.B-0407(A) SM 501007 A! 08:06:59.936 09:59:21.9 02:06:40.6 1.24 XSHOO.2011-03-27T00:32:16.538.fits SCI_SLIT_VIS 100k/1pt/hg/1x2_0.9x11_VIS OK TELL01 0.041 0.02 086.B-0407(A) SM 501007 A! 08:32:32.119 09:59:21.6 02:06:37.0 1.19 XSHOO.2011-03-27T00:57:44.787.fits SCI_SLIT_VIS 100k/1pt/hg/1x2_0.9x11_VIS OK TELL01 0.023 0.03 085.D-0144(A) SM 442926 A 13:57:15.318 13:20:44.8 00:18:53.3 1.12 XSHOO.2011-03-27T06:21:34.766.fits SCI_SLIT_VIS 100k/1pt/hg_0.9x11_VIS OK TELL03 0.040 0.07 085.D-0144(A) SM 442926 A 14:10:08.428 13:20:44.7 00:18:57.6 1.13 XSHOO.2011-03-27T06:34:25.471.fits SCI_SLIT_VIS 100k/1pt/hg_0.9x11_VIS OK TELL03 0.032 0.06 085.D-0144(A) SM 442926 A 14:22:55.523 13:20:44.7 00:18:57.6 1.15 XSHOO.2011-03-27T06:47:11.025.fits SCI_SLIT_VIS 100k/1pt/hg_0.9x11_VIS OK TELL03 0.023 0.04 085.D-0144(A) SM 442926 A 14:35:49.637 13:20:44.8 00:18:53.3 1.17 XSHOO.2011-03-27T07:00:02.810.fits SCI_SLIT_VIS 100k/1pt/hg_0.9x11_VIS OK TELL03 0.014 0.02 386.A-0620(A) SM 503333 A! 15:09:25.140 13:11:30.1 -1:19:20.1 1.25 XSHOO.2011-03-27T07:33:32.648.fits SCI_SLIT_VIS 100k/1pt/hg/1x2_0.9x11_VIS NOK TELL01 0.252 0.03 LOST 386.A-0620(A) SM 503333 A! 15:16:02.224 13:11:29.9 -1:19:16.7 1.27 XSHOO.2011-03-27T07:40:08.661.fits SCI_SLIT_VIS 100k/1pt/hg/1x2_0.9x11_VIS NOK TELL01 0.256 0.05 LOST 086.B-0407(A) SM 501007 A! 08:07:04.950 09:59:21.9 02:06:40.6 1.24 XSHOO.2011-03-27T00:32:19.772.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL02 0.041 0.02 086.B-0407(A) SM 501007 A! 08:32:37.133 09:59:21.6 02:06:37.0 1.18 XSHOO.2011-03-27T00:57:47.840.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL02 0.023 0.04 085.D-0144(A) SM 442926 A 13:57:21.334 13:20:44.8 00:18:53.3 1.12 XSHOO.2011-03-27T06:21:39.265.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL04 0.040 0.07 085.D-0144(A) SM 442926 A 14:10:13.442 13:20:44.7 00:18:57.6 1.13 XSHOO.2011-03-27T06:34:28.382.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL04 0.032 0.06 085.D-0144(A) SM 442926 A 14:23:00.537 13:20:44.7 00:18:57.6 1.15 XSHOO.2011-03-27T06:47:14.131.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL04 0.023 0.04 085.D-0144(A) SM 442926 A 14:35:54.650 13:20:44.8 00:18:53.3 1.17 XSHOO.2011-03-27T07:00:05.866.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL04 0.014 0.02 386.A-0620(A) SM 503333 A! 15:09:30.154 13:11:30.1 -1:19:20.1 1.25 XSHOO.2011-03-27T07:33:35.688.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL04 0.010 0.06 386.A-0620(A) SM 503333 A! 15:16:07.238 13:11:29.9 -1:19:16.7 1.28 XSHOO.2011-03-27T07:40:11.615.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL04 0.014 0.09 386.A-0620(A) SM 503333 A! 15:22:42.317 13:11:29.9 -1:19:16.5 1.30 XSHOO.2011-03-27T07:46:45.547.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL04 0.019 0.11 386.A-0620(A) SM 503333 A! 15:29:19.401 13:11:30.1 -1:19:18.9 1.32 XSHOO.2011-03-27T07:53:21.472.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL04 0.023 0.13 386.A-0620(A) SM 503333 A! 15:35:57.488 13:11:30.1 -1:19:19.5 1.35 XSHOO.2011-03-27T07:59:58.768.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL04 0.028 0.16 386.A-0620(A) SM 503333 A! 15:42:34.572 13:11:29.8 -1:19:15.4 1.38 XSHOO.2011-03-27T08:06:34.692.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL06 0.027 0.11 386.A-0620(A) SM 503333 A! 15:49:11.656 13:11:29.9 -1:19:16.0 1.41 XSHOO.2011-03-27T08:13:10.617.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL06 0.022 0.08 386.A-0620(A) SM 503333 A! 15:55:48.740 13:11:30.1 -1:19:19.3 1.45 XSHOO.2011-03-27T08:19:46.541.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL06 0.018 0.04 386.A-0620(A) SM 503333 A! 16:02:26.827 13:11:30.1 -1:19:19.0 1.49 XSHOO.2011-03-27T08:26:23.132.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL06 0.013 0.00 386.A-0620(A) SM 503333 A! 16:09:03.912 13:11:29.9 -1:19:15.8 1.53 XSHOO.2011-03-27T08:32:59.762.fits SCI_SLIT_NIR 0.9x11_NIR OK TELL06 0.008 0.04 60.A-9022(C) 200193515 X! 09:05:47.568 10:08:48.4 06:43:02.8 1.22 XSHOO.2011-03-27T01:30:54.734.fits 0/6 STD_TELL_SLIT_VIS 100k/1pt/hg/1x2_0.9x11_VIS Hip049704 TELL01 60.A-9022(C) 200193515 X! 14:55:35.876 14:38:48.2 07:54:39.8 1.19 XSHOO.2011-03-27T07:19:46.149.fits 0/6 STD_TELL_SLIT_VIS 100k/1pt/hg_0.9x11_VIS Hip071615 TELL03 60.A-9022(C) 200193515 X! 16:21:01.872 12:47:53.6 -24:51:04.5 1.49 XSHOO.2011-03-27T08:44:57.927.fits 0/6 STD_TELL_SLIT_VIS 100k/1pt/hg/1x2_0.9x11_VIS Hip062448 TELL05 60.A-9022(C) 200193515 X! 09:05:52.582 10:08:48.4 06:43:02.8 1.22 XSHOO.2011-03-27T01:30:58.135.fits 1/3 STD_TELL_SLIT_NIR 0.9x11_NIR Hip049704 TELL02 60.A-9022(C) 200193515 X! 14:55:41.892 14:38:48.2 07:54:39.8 1.19 XSHOO.2011-03-27T07:19:49.558.fits 1/3 STD_TELL_SLIT_NIR 0.9x11_NIR Hip071615 TELL04 60.A-9022(C) 200193515 X! 16:21:06.886 12:47:53.6 -24:51:04.5 1.49 XSHOO.2011-03-27T08:45:01.320.fits 1/3 STD_TELL_SLIT_NIR 0.9x11_NIR Hip062448 TELL06 PROG_ID
Table with all matches. This additional output table provides a list of all possible TELL matches. It is thought to provide details in case of NOK issues. For instance, the main page lists the best possible NOK-OK match (dTime violated but dAirm ok), while the all-matches table also lists the other telluric matches, including the ones flagged OK-NOK. Find its link here: [all matches].
Update information and documentation
|Last update:||2017-02-22T07:12:07 (UT)||
Configured validity dTime:
|2.0 hr (0.083 d)|
|Last header: XSHOO.||2017-02-22T05:21:51.420.hdr||
Configured difference in airmass dAirm:
Last update: timestamp of creation of this page (UT)
Last header: name of last header file currently available; compare this timestamp to 'Last update' to see if monitor is up-to-date
Frequency: The tool is scheduled by a cronjob and is running during the Paranal night, from about 22:00 UT until about 11:00 UT, every 15 minutes.
As a specialized tool, tellTracker is tuned for good performance. Instead of OCA, it uses simple pattern matching (unix grep) for classification and association (like the long-term calibration section of calChecker). All configured keys are read with fitsreport into a single table which is then analyzed for data types, matches and setups. tellTracker typically executes in less than a minute. Its execution times are written in the QC1 database table exec_time and monitored for the case of XSHOOTER here.
How to install
This tool is not part of the standard dfos tool suite. Download the installation tarball from http://www.eso.org/~qc/dfos/ or directly from this link into $DFO_INSTALL_DIR/tellTracker. Then:
tar xvf tellTracker.tar
mv tellTracker $DFO_BIN_DIR
The following should be done only if this is a new installation (not for updates!):
mv config.tellTracker tellTracker.cfg tellTracker_radec.cfg $DFO_CONFIG_DIR
Read the README file and follow the instructions. They are standard and straightforward.
Type tellTracker -h | -v for on-line help and version number. Type
to create the output (per default for $TODAY).
tellTracker -d(ate): for a specified date
tellTracker -N: ($TODAY, no header download)
tellTracker -d(ate) -O: (specified date, with header download)
tellTracker -V(erbose): verbose output (otherwise the tool is silent)
The verbose output goes to the terminal. It displays all matches per science file and may be useful for analyzing issues.
Note that for $TODAY the default behaviour is header download (with HdrDownloader), while for older dates the default is no header download (since those header directories are assumed to be complete anyway).
The tool has
1. The tool configuration config.tellTracker defines:
|Section 1: general parameters|
||120||validity range for telluric matches in minutes (beyond +/-$RANGE minutes a match will be considered NOK)|
||0.22||airmass range for telluric matches (delta_airm beyond +/-D_AIRM will be considered NOK)|
Section 2: 2. Classification and association
|DATA_TYPE||SCI_SLIT_VIS||&&grep " VIS " | egrep "OBJECT ECHELLE,SLIT,NODDING|OBJECT ECHELLE,SLIT,STARE|OBJECT ECHELLE,SLIT,OFFSET|SKY ECHELLE,SLIT,OFFSET" | grep SCIENCE&&|
||STD_TELL_SLIT_VIS||&&grep " VIS " | grep STD,TELLURIC | egrep " 2_SEQ| 3_SEQ" | egrep "ECHELLE,SLIT,NODDING|ECHELLE,SLIT,STARE"&&|
Define for each DATA_TYPE the associated TELL_DATA_TYPE
and the matchkey;
The following lines are auto-inserted by tellTracker, to avoid unnecessary checks for directories on safweb1 (don't remove):
2. Two configuration files are needed to define the classification and association keys, as well as the keys for the output table. One is called tellTracker.cfg, the other one tellTracker_radec.cfg. The latter file contains RA, DEC, and LST which are always used and need a fixed format and conversion. This file does not need any attention by the QC scientist. The other file comes as a fitsreport config file with minimum content. It has two parts, the first part being a standard part not to be modified by the user, and a second part for the instrument-specific setup definition:
|General keys, always needed|
|DPR.CATG||needed for classification|
|MJD-OBS||needed for time matching|
|TEL.AIRM.START||needed for the airmass constraint; the mean airmass is evaluated|
|OBS.PROG.ID||used for display and for 60.A suppression|
|SETUP keys, user controlled; e.g.:|
|SEQ.ARM||these are typical keys for setup definition. Their index (e.g. $13) is used in section 2.1 of config.tellTracker to define matchkeys.|
As usual with fitsreport config files, you can e.g. define 'Delimiter' strings to make selection of these keys by unix grep easier. See more in the fitsreport documentation.
3. There is an optional but recommended info file, info.tellTracker under $DFO_CONFIG_DIR. Its text is displayed on top of the page, right part. Its content should be similar to the info file for calChecker.
- The tool will download headers with 'HdrDownloader -d $TODAY', without suppressing the last (possibly) incomplete template; in order to avoid unwanted effects with autoDaily (incomplete ABs), its headers are downloaded into $DFO_HDR_DIR/TELLTRACK; the call of 'HdrDownloader -d' is incremental, i.e. only new headers are downloaded.
- Make sure to set up the cronjob file properly: call the tool like
01,16,31,46 00-11,22,23 * * * . /home/xshooter/.qcrc; /home/xshooter/bin/tellTracker
- Make sure to call tellTracker before autoDaily: if not it will possibly interfer with filterRaw (as triggered by autoDaily) which means it will search for headers which are temporarily not available.
|Last update: April 26, 2021 by rhanusch|