Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable

DFOS tools: components and documentation

tool
M?*
description version date download history
1. Infrastructure
.dfosrc
initialization file     download latest version operational for Giraffe (for quick reference)
dfosInstall
*
automatic installation for dfos tools 1.3 2019-05-28 download history
dfosExplorer
*
GUI for DFOS installation 1.5.2 2019-06-05 download history
dfosCron
*
tool to launch dfos cronjobs 1.7.1 2018-06-16 download history
2. Utility tools (DFS tools: red)
createCalibMap * create calibration maps 2.1.1 2018-06-25 download history
calselManager * manage calSelector OCA rules 3.2 2019-02-16 download history
dfoManager
*
review operational monitors 1.0.2 2018-06-25 download history
rawdisp2reference
*
store a RAWDISP set for reference 1.4.3 2018-09-07 download history
utilPack
*
package with utility tools 1.26.6 2019-07-11 download history
qcdate 
date arithmetics and conversion 2.0.1 2019-02-07
modhead, modheadcmt
add/edit fits key to a file header 1.0 2008-06-09
hide_raw_files   collect a list of raw file names and hides them, along with the reason for hiding 1.3 2019-04-30
enterCommentsBulk   enter raw or product file comments in bulk mode 1.0.1 2011-04-21
getObInfo   query old or new database for OB related information 2.0 2017-07-25
hideFrame
send a change/hide request to dbcm 2.0 2019-07-11
dfosBackup
create regular backups of DFOS software and configuration 1.0 2006-01-24
refreshVCAL
create virtual calibrations from scratch 1.0.1 2009-06-16
cleanupRawdisp   remove outdated RAWDISP plots on web servers 1.0.3 2018-09-07
cleanupProducts

replace product fits files by their headers

3.3 2016-06-22
dfosCleanPath
cleanup LD_LIBRARY_PATH 1.1 2009-02-12
dfosRLSmassage.py   prepare OCA RLS file for createCalibMap 1.1.1 2013-03-11
dfosVNCserver   checks for running vnc sessions; if not found, creates them 1.0 2013-03-05
ngasMonitor
*
NGAS monitor 3.0.1 2016-02-23
writeBreakpoint   manage breakpoints for calSelector 1.1 2015-03-21
3. Wrapper tools (combining workflow steps)
autoDaily
*
wrapper to control the first part of the standard DFO daily workflow 2.5.4 2018-06-16 download history
processPreImg
wrapper to control the PreImg process (not part of the standard dfos tool suite) 1.4.1 2018-09-07 download history
phoenix, distillery: see sections 6 and 7
4. Monitor tools
calChecker
*
calibration completeness monitor 4.6.7 2018-09-06 download history
tellTracker   telluric standard star tracker [optional tool] 1.5.3 2020-02-07 download history
histoMonitor
*
monitor for processing history 2.4 2020-10-22 download history
dfoMonitor
*
daily workflow monitor 5.1.1 2021-03-12 download history
mucMonitor
*
visualization of condor queue on muc blades 1.3.4 2019-08-19 download  history
getStatusAB
*
AB product monitor 4.1.7 2021-05-19 download history
productExplorer
*
query and manage the content of the product file archive 2.0.1 2017-09-15 download history
5. Daily workflow tools
dataLoader
*
download hdr and fits files
1.3.2 2015-01-13 download history
ngasClient
*
direct download of files from NGAS, using ngamsCClient or dpQuery 1.8.5 2020-03-03 download history
HdrDownloader
download hdr files
1.1    
createReport
*
create raw file reports 4.3.3 2020-02-07 download history
filterRaw
*
scan and filter raw files 1.3 2013-01-07 download history
createAB
*
create ABs 3.9.3 2021-02-26 download history
createJob
*
create an executable AB job file 2.3.5 2019-09-16 download history
processAB
*
process an AB 2.7.6 2021-06-26 download history
processQC
*
create QC report for an AB 1.6.6 2019-03-07 download history
certifyProducts
*
certify pipeline products 2.12.8 2021-10-20 download history
moveProducts
*
move pipeline products 2.2.7 2019-05-22 download history
renameProducts
*
rename products 1.8 2017-02-13 download history
ingestProducts
*
ingest products 3.2.1 2019-05-22 download history
verifyAB
*
verify calSelector associations 1.2.3 2019-05-22 download history
CalSelector   archive application to find associations for science files 3.0.4 2019-05-22  
updateAB
*
update ABs 1.5.6 2019-09-17 download history
extractStat
*
extract statistical data 3.0.1 2014-09-04 download history
finishNight
*
clean up 1.5.4 2021-05-11 download history
6. PHOENIX tools (required only for PHOENIX installations)
phoenix** * science processing 4.1.2 2021-01-18 download history
phoenixMonitor** * monitor for phoenix processing progress 3.2 2021-03-12 download history
call_IT** * wrapper tool for phase3 IngestionTool 1.4.2 2020-10-23 download history
7. OPSHUB tools (required only for OPSHUB installations)
distillery*** * create and process ABs for defined OPSHUB projects 1.1.4 2020-02-13 download history
8. Standard DFS tools (provided by the current DFS package)
ABbuilder
central OCA engine 1.1.6   [standard DFS installation]  
fitsreport
create file-by-file reports 1.3.4    
hotfly
update headers of fits files, or download updated headers 1.1    
dpIngest ingest product fits file 1.0  
dpDelete
delete archived product file 1.0    
dpQuery   download mcalib from NGAS 1.1.1    
ngamsCClient   download raw file from NGAS      
HdrDownloader
download hdr files
1.1    
qc1Ingest   ingest a set of QC1 parameters into the QC1 database 1.4    
IngestionTool**   ingest science products into phase3 infrastructure 3.1.1    
* monitored by dfosExplorer? ** required for PHOENIX only *** required for OPSHUB only

This page lists the available DFOS tools. The tools come for three different environments: DFOS (the standard installation used by QCG), PHOENIX (IDP science processing), OPSHUB (project-driven processing).

The tools come in two flavours:

There is the template initialization .dfosrc file needed by all DFOS tools to define the standard, common environment. Find details about the implementation here.

General installation

All components can be installed in three ways:

  1. recommended: use dfosExplorer which checks your installation, recommends upgrades or new installations and performs them on your request.
  2. also possible: type dfosInstall -t <tool_name>; dfosInstall performs the above actions for you
  3. download the tar file from the browser, unpack it, and follow the README file. The tar files contain the tool, template configuration files and additional components if applicable. The tar files are available in the column 'download'.

For environments DFOS, PHOENIX: the first option is recommended, alternatively option 2. The third option is always available but only necessary to create .dfosrc and obtain dfosInstall and dfosExplorer.

OPSHUB: use option 1 and then option 2.

Use the top navigation bar to find the tool documentation. 'Version' and 'date' give you the latest version and date available. 'Download' links to the tar file. 'History' displays the history file with useful information about the evolution of the tool, changes and backwards compatibility.

Check out here for the overall picture of how this all fits together.