The PHOENIX systems (in particular on muc08) are typically demanding more resources than the dfos systems. Therefore knowledge about other competing jobs is desirable for the scheduling of your own PHOENIX job. Likewise, there is information about the IDP ingestion jobs on those systems. If one is running, the system displays a blinking warning. See below for the related operational rules.

vultur_exec_cascade jobs. Next the mucMonitor lists the currently executing condor queues:

Scheduled vultur_exec_cascade jobs:

UID	STIME	JOB
uves	14:40	/home/uves/jobs/execAB_CALIB_2013-01-13
xshooter	14:38	/home/xshooter/jobs/execAB_CALIB_2013-01-13
giraffe	14:42	/home/giraffe/jobs/execAB_CALIB_2013-01-13

UID=user ID; STIME=start time; JOB=job file under $DFO_JOB_DIR being currently executed.

Main table. Then it displays the 'condor_q' output:

Condor nodes on muc02         slot4         slot8         slot12

busy | idle | reserved/not available

'busy' indicates that condor currently executes a job on that core; 'idle' means no current condor job; 'reserved/not available' means the core is not configured for condor jobs, or is currently not available. Note that this overview indicates only the condor situation. Some cores are reserved for interactive jobs, so they might be busy e.g. with certifyProducts or trendPlotter and would not be indicated on this monitor. Likewise, an idle condor core might actually be running a non-condor job.

The monitor includes the Ganglia load report (the last-hour overview of the load on that muc server).

If a node is condor-active, the job is displayed (e.g. processAB), along with the AB name and the user ID, in the Details panel:

Details: This is the output of 'condor_status':

The load is the last-minute average.

Queue. The 'queue' tab lists all jobs in the condor queue that are currently waiting (i.e. not dependency-locked but not yet executing due to the muc blade being core-limited).

Output

• mucMonitor_<hostname>.html (the main page, behind the first link) and mucMonitor2_<hostname>.html (the page behind the 'queue' link) in $DFO_MON_DIR, also exported to http://qcweb/~qc/ALL

How to install

• standard installation with dfosInstall

How to use

Type mucMonitor -h for a quick help, mucMonitor -v for the version number. Type

mucMonitor

on the command line to create or refresh http://qcweb/~qc/ALL/mucMonitor_<hostname>.html.

Operations

While calling on the command line is possible anytime, the tool should be running operationally in an infinite loop on one account per muc blade only. Remember it is a visualization of 'condor_q', and condor is an installation on a machine, not on an account. For instance, there is one condor installation on muc02, and no matter if giraffe, uves or xshooter call 'condor_q', they will all see the same result. This is also true for 'mucMonitor'. Calling it individually from each account would not break anything but cost unnecessary performance. On the other hand it doesn't matter which account is executing mucMonitor. For that reason, the output indicates the current execution node of mucMonitor.

Self-organized mode. Therefore the tool should run operationally in a self-organized way: If the browser has discovered the tool is not refreshed anymore (how?), you should start the tool on your own account and thereby "take over". Killing or starting a session on another account is not possible (unless in emergency you log in via qc_shift). Of course, if your own session is hanging, you can/should fix it right at the source.

The proper way of calling the infinite loop is

watch -n 10 mucMonitor

Do this in a dedicated window. It's a good idea to do it in a separate xterm so that you easily recognize it.If needed, you (but nobody else) can always terminate the session with Ctrl+C.

On the AB monitor, you also have a button to start the mucMonitor in the infinite loop. The AB monitor has a browser-independent way of finding whether a mucMonitor session exists on the muc blade: it greps in the 'ps' output for the string mucMonitor. If nothing is found, it displays the standard red cross as alert.

The output is linked to the dfoMonitor. The output is linked to the public monitor page as "MUC monitor".

Outdated.

The browser calculates the difference between the current time and the timestamp in the mucMonitor (same way as e.g. in the trendPlotter or calChecker pages). If the page is older than 1 minute (!), it turns red and tries to catch your attention. Thereby you can normally expect the output to be near-real time.

Timeouts. Sometimes the command 'condor_q' is hanging. The tool has a timeout mechanism: if after 6 seconds there is no condor_q response, it exits and tries again after the 'watch' time. The configuration file download has also a timeout mechanism.

Configuration file

Because of the self-organized way of operations, the tool needs a central configuration being identical for all participating accounts. This is why there is a master configuration file http://www.eso.org/~qc/dfos/tools/config.mucMonitor_<hostname> that is downloaded automatically each time the tool executes. It overwrites the local version under $DFO_CONFIG_DIR/config.mucMonitor_<hostname>.local. The download is done via wget and has a timeout protection (10 sec). On timeout, the local version is read instead. It is pointless to edit the local version (it gets overwritten upon the next execution).

The central configuration file should not be edited unless after coordination.

It contains the topology of the executing muc blade, and labels marking the role of the nodes ('condor_execution' or 'free'):

Special cases

There are the following special cases:

a) the PHOENIX processing on muc08/muc10/muc11 (having no autoDaily process): these machines have the PHOENIX_EXEC set to YES, to indicate if a phoenix job is being executed. While there is no general operational rule, it seems reasonable to wait with a task if another PHOENIX job is already running.

b) the MUSE processing on muc10/muse11, using DRS_TYPE=CPL (because of internal parallelisation); then there is no need to monitor the condor pattern. This is controlled by the optional key CONDOR_AB (by default YES, here set to NO.

c) the PHOENIX-enabled accounts have ING_WARNING=YES, giving a warning if an ingestion process is currently running, this to avoid having several of them running in parallel. If you see such a warning (blinking), please wait with your ingestion job until the blinking is over. In particular the MUSE IDP ingestion process is quite demanding for the current NGAS system.

This even applies to other mucs: before you start an IDP ingestion process, please check the mucMonitors of muc08, muc10, and muc11 (or check with your colleagues directly).