Common Trending and QC tools:
Documentation

tqs = Trending and Quality Control System

make printable 


new:

v1.0:

- released

see also:

webCMS syntax; intro presentation

[ used databases ] databases none [classification.txt will eventually go to a database]
[ used dfos tools ] tqs tools webDocuSys; webNavbar
[ output used by ] output used by none
[ upload/download ] upload/download upload (scp) created web pages to standard QC web server

intro presentation

[ top ] webCMS

Description

The tool webCMS (web content management system) helps to create and maintain the QC web pages. The core ideas are:

The webCMS system maintains the HTML part of the tutorial web pages.

It does NOT maintain:

[ top ] Principles

As a content management system, webCMS has the structure of the pages (and the actual HTML code) encoded in the tool, while the content is stored in local text files and maintained by the user. It has a configuration file config.webCMS, a text file with the classification of the QC1 parameters (classification.txt), and text files with the content of the QC pages. The tool has its own directory $DFO_CONFIG_DIR/webCMS hosting all these files.

The file classification.txt will eventually be replaced by a database table. The tool also reads content from the file $DFO_CONFIG_DIR/webDocuSys/version_KPI.txt which will eventually be replaced by a database table. [KPIs are the QC1 parameters describing the instrument performance (literally: key performance indicators). There is a tool managing properties of the KPIs, webKPI (coming).]

Directory $DFO_CONFIG_DIR/webCMS
config file config.webCMS
text file classification.txt
branches BRANCH#1: HOME BRANCH#2: QC_PAGE BRANCH#3: PIPELINE
one sub-dir per QC page index page1 page2 page3 ... page1 page2
n text files per page ... ... ... ... ... ... ...

The QC_SITE is the entire set of QC pages per instrument visible as the web site www.eso.org/qc/<instr>. It is defined and reflected in this structure.

The QC_SITE comes in three branches: HOME, QC_PAGE and PIPELINE. This structure is hard-coded in the tool. QC_PAGE is the set of tutorial pages. They appear under the navigation bar lable "Trending & QC". PIPELINE the set of pipeline and data documentation that appears under the lable "Pipeline". HOME has just one page, index_<instr>.html.

There is another branch, COMMON, which collects the content of the common pages which are embedded as general content in certain QC pages. These are maintained by a priviledged account which is hard-coded as 'giraffe@muc02'.

1. QC_SITE definition

qualifier (name of the branch and name of the webCMS directory) branch (part of URL)
QC_SITE QC_PAGE qc
QC_SITE PIPELINE pipeline
QC_SITE HOME <special>

Each branch has its separate part of the URL. The QC_PAGE branch has 'qc', like www.eso.org/qc/GRAVITY/qc. Each branch consists of a set of web pages. The web pages for the 'qc' branch are collected under the navigation label "Trending&QC1". The web pages for the 'pipeline' part go to the label 'Pipeline'. The HOME branch gets a special, hard-coded treatment which can be ignored here.

2. Pages per branch definition
BRANCH PAGE VNAVBAR QC_TITLE
(=directory)
(=sub-directory)    
QC_PAGE bias_qc1 DEFAULT &&Bias&&
QC_PAGE lamp_qc1 DEFAULT
&&Calibration lamp stability&&
etc.      
PIPELINE pipe_reduc DEFAULT &&Science data&&
etc.      

A web page (PAGE) consists of one or several logical components. For the webCMS system, a component is the smallest logical unit. Its content is encoded as text file. All text files that constitute a web page are stored in the corresponding sub-directory PAGE, and are composed by webCMS into the web page. On the tutorial QC pages, a component typical refers to one HC page and can be recognized by its title.

The column VNAVBAR is filled with DEFAULT. Then the tool derives the name of the vnavbar from the PAGE name: for the PAGE bias.qc1.html, the vnavbar name becomes vnavbar_QC_bias_qc1.html if it is filled with DEFAULT. Only if the vnavbar has a different (non-standard) name, you fill this name.

As a special property, the pages of the QC_PAGE branch have some standard components (such as "Trending", History" etc.). Their existence is checked by the webCMS tool as a completeness and consistency check for the upcoming webDocuSys tool.

The way how the pages are organized (the sequence of web pages), and the internal structure of the web pages (the order of the components), is configured in config.webCMS.

3. Definition of the components of a web page

PAGE components (txt files) stored in (preceded by $DFO_CONFIG_DIR/webCMS/)
COMPONENT bias_qc1.html top_table.txt
general.txt
bias_level.txt
badpix.txt
readnoise.txt
structure.txt
QC_PAGE/bias_qc1/
COMPONENT lamp_qc1.html top_table.txt
general.txt
simlamp.txt
fflamp.txt
nasmyth.txt
arclamp.txt
QC_PAGE/lamp_qc1/
etc.      
COMPONENT pipe_calib.html

raw.txt
products.txt
cascade.txt
static.txt

PIPELINE/pipe_calib/

The examples for the component files are the original source files for the corresponding webCMS pages

http://www.eso.org/observing/dfo/quality/GIRAFFE/qc/bias_qc1.html,

http://www.eso.org/observing/dfo/quality/GIRAFFE/qc/lamp_qc1.html,

http://www.eso.org/observing/dfo/quality/GIRAFFE/pipeline/pipe_calib.html.


[ top ] Content

Most content in the text files is plain text, or related to interpreted high-level tags like TABLE or some ITEM identifiers.

TABLE content gets marked by table identifiers. ITEM identifiers are needed for some text files, there are just 2 or 3 and they appear at the begin of the file. A complete list of identifiers can be found here.

Figures get embedded into TABLEs, they have no type of their own. This makes sense because figures should always have a caption, and a figure with a caption naturally forms a table, or a row in a table. Other HTML elements like URLs or horizontal lines are embedded in TEXT and are interpreted as native HTML code. HTML tags should be kept to a minimum. Don't forget that the purpose of the webCMS tool is to eliminate the burden of HTML coding. In particular avoid anything like text decoration with colors. Likewise, font sizes have gone, they are replaced by classes and styles which are managed within the tool.

[ top ] Syntax

The syntax and structural components are listed here.


[ top ] Migration

1. Configuration

For a given page, you need to configure:

Then, enter the COMPONENTs in section 2.2 .

2. Create the directory $DFO_CONFIG_DIR/webCMS/${QC_SITE}/${PAGE}.

3. How to fill:

3.1 top_table.txt: use the provided example above, everything should be obvious.

3.2 general.txt: this is optional, and in many cases not needed. It is meant to provide some information before the structured part of the page. Here you can put information which is generally applicable to all components of that page.

3.3 Other components: at least one for QC_PAGE. For each component:

4. Continue with the next pages until you are complete for the QC_PAGE branch.

5. Continue with the pages for the PIPELINE branch.

6. Finally add the index_<instr>.html page in the HOME branch, it has a general.txt file only.

Common files:

Simplifications:

Content:

Checking:


[ top ] File classification.txt

The text file classification.txt under $DFO_CONFIG_DIR/webCMS is important. It is used by two tools: webCMS and webDocuSys. It will eventually go into a TBD database and then also have an interface to create/delete/modify content. Currently it is a simple text file structured like a config file.

It contains one row per QC1 parameter and HC plot. All QC1 parameters that are used in a HC plot should be listed here. This is the core idea of the upcoming webDocuSys tool. The documentation information for this QC1 parameter should then also be made visible on the tutorial pages (the QC_PAGE branch of webCMS), which are also linked to the HC plots.

If a QC1 parameter has entries in a QC1 database table but is not used in any HC plot, don't list it here. If you create a new HC plot, always add the corresponding records in classification.txt. If you remove a HC plot, remove all entries referring to that plot.

The table has some content which is not evaluated by the webCMS tool. It should nonetheless be filled consistently.

In the contect of this table, a HC plot is any of:

This boils down to the rule that every config.tp file should have at least one entry in classification.txt, with the exception of:

Often there will be more than entry, because all plotted QC1 parameters per HC plot should have one entry.

Content:

Find the operational classification.txt for GIRAFFE here. Each record has the following columns:

Qualifier Example comments
QC1_parameter giraffe_bias..sigma_raw <table>..<name>
HC_REPORT /qc/GIRAFFE/reports/FULL/trend_report_KPI_READNOISE.html either KPI or HC name; no FULL reports needed unless no _HC existing
HC_CLASS KPI

KPI | HC | CAL | ENG
where
KPI = instrument performance;
HC = instrument health;
CAL = calibration quality;
ENG = engineering parameter

TPL_ID FLAMES_giraf_cal_bias Template ID for parent raw data, as displayed on the score report (needed for webDocuSys)
RECIPE gimasterbias Pipeline recipe for processing these data, as displayed in the AB (enter 'none' if no recipe exists) (needed for webDocuSys)

HC_CLASS:

There are four classes. A QC1 parameter can have more than one class although this rare. Think of the classes as a role. For instance, the read noise of the standard read mode is likely a KPI while the same parameter, for a rarely used read mode, is a class HC or CAL.

A parameter used in a KPI plot has always class KPI. All other parameters describing instrument properties important for data quality are classified HC. Parameters that describe the data quality but are not instrument properties (like count level in a flat) are classified CAL. Parameters which are not related to quality but are needed e.g. for correlation (like temperature) are classified ENG (for engineering).

The item QC1_parameter needs to be filled by table..parameter in order to be unique per instrument.

The items TPL_ID and RECIPE are needed for webDocuSys.


[ top ] How to use

Type

webNavBar -h for on-line help,

webNavBar -v for the version number.

The standard mode is:

webCMS -p <page_name>

where page_name is entered without .html, e.g.:

webCMS -p bias_qc1

The tool reads the components as configured in config.webCMS under COMPONENT (section 3), with the sorting order preserved. The content is interpreted (TABLE names, IDENTIFIERs, QUALIFIERs etc.). If configured, a COMMON part is added. The page <page_name>.html is created, the vnavbar, the top and bottom parts and the title(s) are added.

Finally the page is transferred to the URL www.eso.org/qc/<instr/qc_new/<page_name>.html (or pipe_new for the PIPELINE branch or home_new for the HOME branch). The page is then available for comparison to the operational page which is under www.eso.org/qc/<instr/qc/<page_name>.html (or pipeline for the PIPELINE branch).

This mode is also called the development mode because you can compare the original and the new content, without overwriting the original content.

Once you have checked for completeness and for correctness, call

webCMS -p <page_name> -f

which will transfer the page to the operational URL. This is the 'forced' mode using the -f flag (as trendPlotter does), and the page will go life.

Now the migration for <page_name> is finished. Future maintenance will be done on the local text files in your operational muc environment. The original page (before migration) will probably still exist on your local PC but it will not be used anymore.

Calling the tool in global mode without parameter:

webCMS

This will create ALL configured and existing pages in a loop, in development mode. The tool reads the config file to get the branches and all page names and then loops over all page names.

Calling

webCMS -f

will finally refresh the entire website.

Creating a directory for a new page, and filling it with txt files, will have no impact if you forget to configure the new page in config.webCMS. Think of this as a "registration": an unregistered page does not exist for the tool.

In a later stage, and upon further trigger, the tool can be 'upgraded' to include information from webDocuSys in the QC tutorial pages. Check the GIRAFFE QC pages for a prototype implementation. This is currently not supported for any other instrument, so set INCLUDE_WEBDOCUSYS to NO.


[ top ] Output

The output is the configured and specified (if any) HTML web page on the $DFO_WEB_SERVER.

[ top ] Configuration

The tool has a tool configuration file, the file classification.txt (see above), and a folder tree HOME, QC_PAGE and PIPELINE. All these go to $DFO_CONFIG_DIR/webCMS.

The tool configuration file (config.webCMS; find the operational example for GIRAFFE here):


Section 1: tool configuration
INCLUDE_WEBDOCUSYS NO YES|NO: if YES dynamic queries for documentation are included in the QC pages, as provided by webDocuSys; recommended to set to NO initially

Section 2: Definition of content
2.1: List of pages, relevant for sorting
#QUALIFIER NAME VNAVBAR QC_TITLE
HOME
or
QC_PAGE
or
PIPELINE
complete name of page, with .html extension DEFAULT (unless you link something not following the convention navbar_QC_<page_name> &&QC title as displayed on top of the page; note that the instrument name is automatically added by the tool, so don't duplicate &&
example: QC_PAGE fmt_qc1.html DEFAULT &&Grating stability&&
       


2.2: Components of each page, relevant for sorting

 

COMPONENT # PAGE: name of HTML output page # COMPONENT: name of text file with encoded content  
example: COMPONENT fmt_qc1.html top_table.txt # top_table.txt describes the top HC navigation table, mandatory for QC_PAGEs
COMPONENT fmt_qc1.html general.txt # general.txt is an optional general introduction text, possible for QC_PAGEs only.
COMPONENT fmt_qc1.html temp.txt  
COMPONENT fmt_qc1.html gratings.txt  
COMPONENT fmt_qc1.html correlations.txt  

Send comments to <rhanusch@eso.org>
Last update: March 8, 2018