make printable	new:	see also:
	v4.1: - display of multiple OB grades v4.1.1: - bug fix with INFO_FILE (here)	databases opc70, obs_metadata; ngas for availability check; DFO_db for comments dfos tools fitsreport, qcdate, getObInfo output used by histoMonitor, dfoMonitor, calChecker , ngasMonitor upload/download export to qcweb: data reports (html and txt) javascript javascripts jquery-latest.js, jquery.datatables.js under ${JSCRIPT_URL}
topics: description | modes: DATE | ALL | QUERY| export | help page | config: tool | config: fitsreport | workflow | public output example

createReport

enabled for parallel execution (multiple instances at a time do not interfer with each other)

Since the tool is called in automatic mode (by autoDaily and calChecker), its performance has been optimized with v4.0. Also, there is a safety mechanism to prevent automatic execution if NRAW > 2000 (hard-coded).

Description

This tool creates reports for QC. It has two main modes:

• create the daily data report for raw data (date mode)
• create all historical data reports and create reports per file properties (query mode)

Find its place in the daily workflow here.

Output

The tool creates the following output:

• in date and all modes: HTML report list_<ins>_<date>_data.html plus txt version under $DFO_LST_DIR/REPORT, plus exported versions on qcweb
• in query mode: HTML report $DFO_LST_DIR/REPORT/extract.html and file list

DATE mode:

• the output of the tool comes as set of HTML files, featuring color coding of types, comfortable navigation, display of OB grades and scores
• you can enter comments:
  • per raw file, using the action button enter comments: [ file ] ; these comments are written into the DFO database table raw_comments (same action offered in certification workflow) ; already entered comments display in the last column
  • per date, using the action button enter comments: [ date ] ; these comments are written into the DFO database table daily_comments (same ation as on histoMonitor); already entered comments display in the header part of the HTML pages
• you can launch a query for file properties, based on your local set of daily data reports
• you can re-create any historical report provided you have headers
• navigation:
  • use the navigation to open or close the initial nightlog section, to see the txt file or to see the full nightlog.
  • use the horizontal navigation 2007-12: , or the arrows to move forth and back in time; you can also jump to the first or last report
  • use the links bottom or top for vertical navigation within any given result page, and for incremental navigation (to the next/previous OB)
    Note: this is a new feature with v3.5, you need to update earlier reports if you want this feature for all reports.
• the report is auto-updated by finishNight after finishing the workflow, it then registers the last DFO flag and also links to the scores
• if there is a comment for the whole date (entered here or in the nautilus tool), it displays in the title section (provided you have updated the report after entering the comment)
• file types come color-coded, with the following cases:

science:

180.B-0806(B)

A 287269

CARINA_CORE_5

2

1

GIRAF.2008-01-01T07:26:48.497.fits

FLAMES_GIRAF_OBS001_0009.fits

SCIENCE

OBJECT,OzPoz

MOS

MED

Medusa1

L881.7

POS_1_67

1.323

3600

(0/4)

calib:

60.A-9022(B)

X 200170106

HD49798

1

1

GIRAF.2008-01-01T09:09:56.964.fits

FLAMES_GIRAF_OBS001_0010.fits

CALIB

STD,SimCal

IFU

IFU

IFU1

L682.2

POS_1_67

1.776

16.0

(0/4)

HC calib:

60.A-9022(B)

200117229

***

1

1

GIRAF.2008-01-01T14:20:42.358.fits

FLAMES_GIRAF_WAVE001_0003.fits

CALIB

LAMP,WAVE

MOS

MED

Medusa2

H525.8B

POS_1_67

***

710.

(0/4)

hidden:

H!  60.A-9022(B)

200117229

***

1

1

GIRAF.2008-01-16T16:19:50.640.fits

FLAMES_GIRAF_WAVE016_0001.fits

CALIB

LAMP,WAVE

IFU

IFU

IFU1

H525.8B

POS_1_67

***

710.

very low flux, hidden

All non-SCIENCE, non-CALIB or non-TECHNICAL files display grey.
All hidden files display with grey font colour and are marked by H!. They should have a file comment about the hiding reason.
Note: in exceptional cases (e.g. instrument is not yet operational), you could configure HIDE_FLAG = DISABLED; then no check for the hide flag is done, and of course no marking. A clear message underneath the title reminds you about this non-default situation. The advantage is a gain in performance (since the coding is done in a way that assumes that there is no or just a few files being hidden).

• the reports start with the availability flag (indicating if the file is already in NGAS)
• reports include the score flag and score result, linked to the score HTML file. The score information is included once the corresponding files (AB monitor, scores HTML and tlog) are moved to $DFO_LOG_DIR (i.e. after moveProducts -m CALIB). The automatic recreation of the data report is provided by histoMonitor.
• the OB grades for science data come color-coded, and display the OB comment if marked with a '!' and if the mouse is moved across it:

  B!	289833
  x|A!	1167023

  In the second row, there is the rare case of a multiple execution of an OB. In that case, all grades are displayed, and the comments give further information related to the multiple grades.
• the output tables created by v3.7+ are sortable and can be filtered; this enhancement is provided by the javascript datatables under $JSCRIPT_URL.
• the tool decides, based on the configuration in getObInfo, if nightlog information should be retrieved from the associations..night_logs database (the "old format") or linked to the nightlog tool (NLT, the "new format"). Roughly speaking, the switch between old and new scheme was on 2012-04-01 for UT2 instruments, on 2012-10-02 for the other UTs and VLTI. For the survey telescopes, there is only the new scheme, as well as for all future instruments. [The old PANL2 database has been switched off in 2017.]

ALL mode:

• for convenience, the all mode (-A) is offered. It takes the first and the current last date directory in $DFO_HDR_DIR. It then starts creating the data report for the first one, increments this by +1 and so on until it reaches the last one.
• with this strategy, it creates a continuous set of reports, even for those dates without any headers (and presumably without any data).
• This is useful for the nautilus and for the navigation bar in the HTML reports. It may also be useful in case of changes in the fitsreport configuration file: you may then want to re-create all reports to make full advantage of the query function.
• Of course this mode may take a while (hours ...) to execute.

QUERY mode:

• Use query: [launch] to start a query. You can also start with createReport -Q on the command line.
• This works very much like a database query, except that it is based on your local set of data reports. Technically, it uses 'egrep'-based pattern matching on the set of HTML data reports which you pre-select with the 'month' (time-selection) query. Each file in the result data set then has exactly one line with all the entries which you have chosen in previous createReport runs, plus HTML tags of course.
• You will be prompted for a "month"; this is a simple-minded date selection mechanism and could actually be:
  • a real month (e.g. 2008-05)
  • a year (e.g. 2008)
  • the full range (e.g. 200)
• Then, you will be prompted for a selection string. Here you can enter any string which can be used by egrep. The logical OR is implemented as e.g. "U_BESS|V_BESS". The logical AND operator is implemented as '&'. Other operators (like NOT) are not supported.
• An example: you want to find all LAMP,WAVE files in the L682.2 setting taken in 2008-01. You enter that month, and 'LAMP,WAVE&L682.2' on the second prompt. The tool will return a list of all LAMP,WAVE files in that setting and date range, formatted like in the standard reports, including all comments, OB grades etc.
• The tool has a memory which makes repeated (iterated) queries comfortable: it offers your previous selection (if any) as default choice.
• The result list is offered under the query: [read] link. There is also a text file with the names of all selected files, useful e.g. for data downloads or other workflow steps.
• This is (hopefully) a very powerful mechanism for investigation of non-standard situations; calibration completeness; listings per OB or run_ID; re-processing etc.

Export

The created reports are exported to the QC web server (http://qcweb.hq.eso.org), with the action buttons removed. They accumulate in the directory /qc/<instr>/REPORT which is created and managed by the tool webDircheck. Find an example page here.

Public HELP page

The reports are published on qcweb and linked to the calChecker result pages. Therefore they can be read by Paranal Science operations staff. There is a public help page linked under HELP with some hints about functionality and purpose of the pages.

How to use

Type createReport -h | -v for on-line help and version;

createReport -d 2005-12-30

to produce a standard raw file report for date 2005-12-30,

createReport -d 2015-12-30 -F

if NRAW is higher than 2000 and the tool refuses automatic report creation,

createReport -A

for re-creating all data reports,

createReport -Q

to call the query mode.

By default the tool runs with the temporary workspace $TMP_DIR/$$ which exists during runtime only. This protects a given instance (e.g. launched on the command line) against another one (launched from a cronjob tool). You can override this workspace, for debugging, by assigning a workspace with parameter -T:

createReport -d 2010-12-30 -T $TMP_DIR/test

Note:

• There is at least one config file to support the calls of fitsreport. For the DATE mode there is fitsreport_raw.cfg.
• There is also the optional report_raw.inf file (a text file to be pasted into your report).

Configuration files

- config.createReport

- fitsreport_raw.cfg

This file has the structure of fitsreport config files. Check out here for details.

Operational aspects

• createReport is called by the workflow tools calChecker and autoDaily. It provides up-to-date data reports based on the headers.
• With v4.0, its performance is much faster than before. Nevertheless it has a safety mechanism, it is not called in automatic mode if the number of headers exceeds 2000. Instead the user is warned via email and can call the tool on the command line, with flag -F.
• Version 4.1.1 fixes a bug that prevented the tool (for a long time!) from displaying the configured INFO_FILE. Now that this is fixed, please note: the entire content is displayed as one HTML line; pls. use '|' or similar as separator.

Workflow description: create a raw file report for QC purpose

1. call fitsreport, using fitsreport_raw.cfg, output in text file list1, scanning $DFO_HDR_DIR/<date>/*hdr

1.1 check if NHDR is higher than 2000; if so, stop and send email (unless called with flag -F)
1.2 check for file availability
1.3 check for hide flag
1.4 call getObInfo to get OB grades and comments
1.5 check for score information
1.6 check for raw comments

2. add a text header

4. add some statistical information

5. add a final part (created by ...)

6. write all this into the final report: $DFO_LST_DIR/list_${DFO_INSTRUMENT}_<date>.txt