Data Acquisition Guide

This guide demonstrates different Data Acquisition use-cases with focus on the interaction with OCM to act as a guide when e.g. writing Sequencer templates/scripts.

For details on deployment please refer to section Deployment and section Server for how to configure OCM.

Prerequisites

The guide assumes the runtime environment has been configured and OCM has been deployed and is running.

Example configuration:

# example.yaml
cfg.instrument_id   : "TEST"
cfg.req.endpoint    : "zpb.rr://127.0.0.1:54162"
cfg.pub.endpoint    : "zpb.ps://127.0.0.1:54163"
cfg.dataroot        : "/dataroot/ocm/"

Note

Examples in this guide use the command line client ocmServerCtl provided with the ifw-hl/daq project. To facilitate use from scripts ocmServerCtl can provide return values in JSON format with the --json option. Alternatively the MAL API ocmif.OcmDaq may be used directly.

The instructions assume the following environment variables are defined (if defined the corresponding explicit option can be omitted). The endpoint URIs should reflect what is in your configuration, and using the example configuration above it would be:

  • $OCM_REQUEST_EP=zpb.rr://127.0.0.1:54162 (replaces option --rep)

  • $OCM_PUBLISH_EP=zpb.ps://127.0.0.1:54163 (replaces option --pep)

Futher, unless otherwise stated, all instructions in this section assumes OCM is deployed and is in state Operational/Ready which can be achieved with the following set of commands:

# Start redis-server if not already running
$ redis-server &

# Start ocm server using example configuration
# Note: if relative path is used (e.g. "example.yaml") the environment variable CFGPATH is
#       used to resolve the real file path.
$ ocmServer -c /absolute/path/to/example.yaml &

# Export environment variables to simplify ocmServerCtl interaction
export OCM_REQUEST_EP=zpb.rr://127.0.0.1:54162
export OCM_PUBLISH_EP=zpb.ps://127.0.0.1:54163

# Bring ocm operational
$ ocmServerCtl std.init
$ ocmServerCtl std.enable

To shut ocmServer down the signal SIGINT (Ctrl-c) or the command Exit() can be sent using e.g. the ocmServerCtl application:

# Send exit command
# or send signal SIGINT (Ctrl-C)
ocmServerCtl std.exit

Observing Status Changes

To observe the published state changes in OCM it is possible to use ocmServerCtl without a command with the --status option, in which case it will subscribe and remain running until stopped with Ctrl-c while printing any received topic samples to stderr. This is an example of the output during a Data Acquisition:

$ ocmServerCtl --status
no command provided -> will subscribe indefinitely
status: Operational;Active
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=NotStarted, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Starting, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Acquiring, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Acquiring, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Stopping, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Stopping, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Stopping, error=false
status: Operational;Idle
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Stopped, error=false

Note

The status is published by OCM for any change, some of which are not caused by state transitions. This may lead to the same status line being repeated multiple times.

The --status option can be passed when executing a command as well, but often the commands complete before any topic samples are received. The exception to this is the daq.awaitstate which only completes when the specified state is achieved or operation times out.

Automatic Stop Sequence

Automatic stop is the typical scenario when a Data Acquisition includes one or more primary data sources (usually detectors) that are configured with a fixed integration time. When all primary data sources stop (also referred to as Completed or Stopped in recif) as observed by OCM, which monitors all the sources, OCM will proceed and stop all metadata sources automatically. This condition can then be awaited on to then trigger other activities such as configuring the instrument for the next Data Acquisition.

Note

Multiple sources in each category are provided by space-separating each source, e.g:

"fcf1@zpb.rr://insws1:12345/daq fcf2@zpb.rr://insws2:12345/daq"

Data Acquisition is started with the daq.start command which takes two list of sources.

The following example starts a new Data Acquisition with the primary source named dcs1 and metadata source named fcf1. The names are only used to give a friendly name to a possibly anonymous URI and is so far not used for anything but logging activities.

$ ocmServerCtl --json daq.start "dcs1@zpb.rr://dcsws:12345/rec" "fcf1@zpb.rr://insws:12345/daq"
{
  "id": "TEST.2021-03-09T18:48:05.967",
  "error": false
}

Data Acquisition status can be checked with the daq.status command:

# Check status
$ ocmServerCtl --json daq.status "TEST.2021-03-09T18:48:05.967"
{
  "id": "TEST.2021-03-09T18:48:05.967",
  "state": "StateAcquiring",
  "substate": "Acquiring",
  "error": false,
  "timestamp": 36142213.78997446597
}

When dcs1 completes OCM will issue the equivalent of the StopDaq command to stop all metadata sources. To await the completion of all FITS files the daq.awaitstate command is issued with states ocmif.DaqState.StateAcquiring / ocmif.DaqSubstate.Stopped:

# Await Stopped state with a timeout of 60 seconds so the MAL timeout is set to 61 seconds (-t)
# note: State-names are case-insensitive.
$ ocmServerCtl -t 61 --json daq.awaitstate \
    "TEST.2021-03-09T18:48:05.967" StateAcquiring Stopped 60
{
  "timeout": false,
  "status": {
      "id": "TEST.2021-03-09T18:48:05.967",
      "state": "StateAcquiring",
      "substate": "Stopped",
      "error": false,
      "timestamp": 36142213.78997446597
    }
}

If the global --status option is provided, ocmServerCtl will subscribe to published status topics and print them to stderr (JSON command result goes to stdout):

# Await Stopped state with a timeout of 60 seconds so the MAL timeout is set to 61 seconds (-t)
# note: State-names are case-insensitive.
$ ocmServerCtl --status -t 61 --json daq.awaitstate \
    "TEST.2021-03-09T18:48:05.967" StateAcquiring Stopped 60
daq: id=TEST.2021-03-09T18:48:05.967 state=StateAcquiring, substate=Stopping
daq: id=TEST.2021-03-09T18:48:05.967, state=StateAcquiring, substate=Stopped
Done
{
  "timeout": false,
  "status": {
      "id": "TEST.2021-03-09T18:48:05.967",
      "state": "StateAcquiring",
      "substate": "Stopped",
      "error": false,
      "timestamp": 36142213.78997446597
    }
}

At this point the data sources are finished with their contribution and the merging process can be started.

Manual Stop Sequence

Manual stop is used either when there is no fixed integration time on all primary data sources or when no primary sources are used at all (in which case there is nothing to inform OCM when to stop except for the user).

In this example data is acquired from fcf1 only, so user must decide when to stop manually.

# Start
$ ocmServerCtl --json daq.start "" "fcf1@zpb.rr://insws:12345/daq"
{
  "id": "TEST.2021-03-09T18:48:05.967",
  "error": false
}

At a later point the Data Acquisition is stopped as follows:

# Stop the just started DAQ by specifying the correct id
$ ocmServerCtl --json daq.stop "TEST.2021-03-09T18:48:05.967"
{
  "id": "TEST.2021-03-09T18:48:05.967",
  "error": false
}

# Check status
$ ocmServerCtl --json daq.status "TEST.2021-03-09T18:48:05.967"
{
  "id": "TEST.2021-03-09T18:48:05.967",
  "state": "StateAcquiring",
  "substate": "Stopped",
  "error": false,
  "timestamp": 36142213.789974466
}