JSON Schemas¶

Warning

Schemas are still work-in-progress and may change between OCM versions.

Some commands allow transport of parameters in serialized JSON format. This section describes the schemas in use using JSON Schema.

Common Definitions¶

Some parts are reusable in multiple commands and is presented here:

Download JSON Schema.

Start DAQ¶

Download JSON Schema.

This schema is a composite schema that defines the properties parameter that is provided for the StartDaq() command.

It is foreseen that all parameters to StartDaq() will be provided as part of the following structure, in a future version of OCM.

Note

All time durations are specified in seconds.

Example:

{
  "awaitInterval": 5.0,
  "keywords": [
    {
      "type": "valueKeyword",
      "name": "OBJECT",
      "value": "OBJECT,SKY"
    },
    {
      "type": "esoKeyword",
      "name": "OBS TPLNO",
      "value": 2
    }
  ]
}

Root object description:

awaitInterval (number) [default: 5.0]: This is the interval and timeout, in (fractional) seconds, used when awaiting the completion of primary sources. If this number exceed the MAL timeout it will stop working.
keywords (List[FitsKeyword]) [optional]: These are the initial FITS keywords that will eventually be provided with the FITS file created by OCM primary HDU. See FITS Keywords for detail.

FITS Keywords¶

Download JSON Schema or view it here.

FITS keywords are provided as a JSON-array of objects, where each object describe one keyword. The following show how FITS standard value keyword and ESO hierarchical keyword are defined:

[
   {
      "type":"valueKeyword",
      "name":"OBJECT",
      "value":"OBJECT,SKY"
   },
   {
      "type":"esoKeyword",
      "name":"OBS TPLNO",
      "value":2
   }
]

FitsKeyword (object)

type (str)

valueKeyword: Is used for standard FITS keywords and keywords are limited to 8 characters. Trailing whitespaces are not necessary to add, but is allowed.
esoKeyword: Is used for ESO hierarchical keywords that use space-separated groups of subsystems. The prefix HIERARCH ESO is added automatically when keyword is written to FITS file and shall not be included.

name (str)

Logical keyword name.

value (Union[str, boolean, number])

For both valueKeyword and esoKeyword the value field provide the raw value, which can represent the following FITS types:

Table 5 JSON to FITS type conversion¶
JSON	FITS	Notes	Example
string	string	Do not use single-qotes `'`. Timepoints are represented as strings.	`"foobar"` -> `'foobar'`, `"2020-07-28T04:57:00.8836"` -> `'2020-07-28T04:57:00.8836'`
boolean	logical		`true` -> `T`, `false` -> `F`
number	integer or float	Dictionary will be used to format the value correctly (not yet implemented). integer range: `-9223372036854775807 ..` `9223372036854775807` float range: `-1.79769313486231e+308 ..` `1.79769313486231e+308`

Note

There is no built-in type in either JSON or FITS to represent a time point (date/datetime). DICD [RD1] standard time format is ISO 8601: YYYY-MM-DDThh:mm:ss.sss or YYYY-MM-DD when only date is shown.

When it is time to format the value a dictionary will be used to validate the keyword and perform the formatting of the raw value as well as to provide the comment field (see ctd project for additional details on the dictionary system).

Standard MAL API¶

Note

ocmServer hosts the standard commands stdif.StdCmds under URI path /std.

The standard interface is documented in the RAD User Manual and should be consulted for details and only the brief is provided here.

Interfaces¶

class stdif.StdCmds¶

Standard command interface.

Stop() → str¶

Init() → str¶

Reset() → str¶

Enable() → str¶

Disable() → str¶

GetState() → str¶

GetStatus() → str¶

GetVersion() → str¶

Exit() → str¶

SetLogLevel(info: LogInfo) → str¶

Data Structures¶

class stdif.LogInfo¶

level: str¶

logger: str¶

class stdif.StatusTopic¶

status: str¶

Data Acquisition MAL API¶

Note

ocmServer hosts the Data Acquisition control interface ocmif.OcmDaq under URI path /daq.

The following sections document the interface in a language agnostic manner. For data structures only the data members are documented, not the accessors generated by MAL. The names are not fully qualified as it is different across the supported languages.

Interfaces¶

class ocmif.OcmDaq¶

OCM Data Acquisition control interface.

StartDaq(id, file_prefix, prim_sources, meta_sources, properties) → DaqReply¶

Create and start new data acquisition. If id is not provided (left empty) ocmServer will create a unique identifer automatically.

The format for prim_sources and meta_sources is a space separated list of sources in the format: <name>@<rr-uri>, e.g. meta-source@zpb.rr://example:1234/daq.

The format might be changed in future version to be part of the JSON property parameter.

There must be at least 1 data source.

Fail-fast

If any source fails to start the command will fail.

Parameters

id (str) – Optional unique identifier of data acquisition.
file_prefix (str) – Optional file name prefix.
prim_sources (str) – List of primary data sources (e.g. detectors).
meta_sources (str) – List of metadata sources (e.g. FCF Device Manager or CCS).
properties (str) – JSON properties.

Returns

If id was provided that is returned as an acknowledgement, otherwise the id generated by ocmServer is returned.

Return type

DaqReply

Raises

DaqException – On fatal error.

StopDaq(id) → DaqReply¶

Stops data acquisition and keep any data acquired.

Fail-Fast

If all data sources fail the command will report error by throwing DaqException. A partially successful execution is reported using normal status reply (which includes information about any partial failure). This is done because it is prioritized to create a partially complete data product over discarding all acquired data.

Parameters: id (str) – Id of data acquisition to stop.
Return type: DaqReply
Raises: DaqException – On fatal error.

ForceStopDaq(id) → DaqReply¶

Like StopDaq() the command stops data acquisition and keeps any data acquired. The only difference is that if non-fatal error occurs the Data Acquisition is marked as stopped whereas StopDaq() would not.

Fail-slow

The command is resilient to non-fatal errors. If any non-fatal errors occur the error flag in the reply is set but no exception is thrown. If fatal error occur DaqException is thrown.

Warning

Any data source that failed to stop properly will not be able to provide data to the Data Acquisition or the final Data Product.

Although ocmServer is always left in a consistent state, any data source that failed to stop may not. Manual intervention my be necessary to restore a problematic data source to a functional state.

Parameters: id (str) – Id of data acquisition to forcibly stop.
Return type: DaqReply
Raises: DaqException – On fatal error.

AbortDaq(id) → DaqReply¶

Aborts data acquisition and discards any data acquired.

Fail-fast

If any error occur DaqException is thrown and the DAQ state remains in Aborting.

Parameters: id (str) – Id of data acquisition to abort.
Return type: DaqReply
Raises: DaqException – On fatal error.

ForceAbortDaq(id) → DaqReply¶

Like AbortDaq() the command aborts data acquisition and discards any data acquired. The only difference is that if non-fatal error occurs the Data Acquisition is marked as aborted whereas AbortDaq() would not.

Fail-slow

The command is resilient to non-fatal errors. If any non-fatal errors occur the error flag in the reply is set but no exception is thrown. If fatal error occur DaqException is thrown.

Warning

Although ocmServer is always left in a consistent state, any data source that failed to abort may not. Manual intervention may be necessary to restore a problematic data source to a functional state.

Parameters: id (str) – Id of data acquisition to forcibly abort.
Return type: DaqReply
Raises: DaqException – On fatal error.

UpdateKeywords(id:str, keywords:str) → DaqReply¶

Update keywords for specified Data Acquisition. For each keyword:

If it already exist*, it is updated with the provided value,
otherwise it is added.

*) OCM will compare keyword name as provided, while considering it it’s an ESO hiearchical keyword or standard FITS keyword.

Fail-fast

Command will not make any attempt to be robust against errors. Failures are treated as fatal and DaqException will be thrown.

Parameters

id (str) – Id of data acquisition to update.
keywords (str) – JSON encoded set of keywords. Refer to JSON keywords schema for details.

Return type

DaqReply

Raises

DaqException – On fatal error.

AwaitDaqState(id, state, substate, timeout) → AwaitDaqReply¶

Await a particular Data Acquisition state. This is mainly used to support sequencing of higher level coordination activities.

The command will reply when condition is fulfilled or times out. When it times out this is indicated in the reply with AwaitDaqReply.timeout set to true.

Condition is fulfilled whenever the requested state and substate is either achieved or no longer possible to achieve. Awaiting e.g. the condition. DaqState.StateMerging/ DaqSubState.Scheduled will be fulfilled also for DaqState.StateCompleted/DaqSubState.Aborted. Perhaps a more likely scenario is that multiple state transitions occurred before request was handled, such that a subsequent state is reported instead of the requested state/substate.

Note

OCM v1.0.0 implements all substates in DaqState.StateAcquiring and py:attr:DaqState.StateCompleted/DaqSubState.Aborted. As such it is not possible to await other states.

Parameters

id (str) – Id of Data Acquisition to await.
state (DaqState) – State of Data Acquisition to await.
substate (DaqSubState) – Substate of Data Acquisition to await.
timeout (float) – Duration in seconds to wait for condition to be fulfilled. Must be > 0.

Returns

Current Data Acquisition status and indication if operation timed out.

Return type

AwaitDaqReply

Raises

DaqException – On fatal error (e.g. invalid arguments).

GetDaqStatus(id) → DaqStatus¶

Get Data Acquisition status.

Parameters: id (str) – Id of data acquisition to get status for.
Return type: DaqStatus
Raises: DaqException – On fatal error.

GetActiveList() → List[DaqStatus]¶

Get list of active Data Acquisitions (i.e. not in :py:attr`ocmif.DaqState.StateCompleted`).

Returns: List of active Data Acquisitions.
Return type: List[DaqStatus]
Raises: DaqException – On fatal error.

Data Structures¶

class ocmif.DaqState¶

Enumeration of top-level Data Acquisition states, see The Data Acquisition Process for additional details.

StateAcquiring¶: State that span the initial point it is created in OCM through the phase where data is acquired until all inputs for the Data Product have been created. See also StateAcquiring.

StateMerging¶: State that spans not where a Data Product is created from the acquired data. Phase where Data Product is created. See also StateMerging.

StateCompleted¶: Data Acquisition is completed.

StateUndefined¶: Only used for technical reasons. This is not an expected state.

class ocmif.DaqSubState¶

Enumeration of possible Data Acquisition sub-states, see The Data Acquisition Process for additional details.

Note

Only a subset of the listed states are possible/valid in each DaqState.

NotStarted¶: Data Acquisition is created but not yet started.

Starting¶: Transitional state where Data Acquisition is being started. Some data sources might already be acquiring data. When all sources have started the Data Acquisition transitions to Acquiring.

Acquiring¶: Data is being acquired.

Stopping¶: Transitional state to Stopped. Some data sources might still be acquiring data and are about to be stopped. When all data sources have stopped the Data Acquisition transitions to Stopped.

Stopped¶: All data has been acquired.

Aborting¶: Transitional state where Data Acquisition remains while it is in the process of being aborted.

Aborted¶: If a Data Acquisition has errors or otherwise needs to be aborted the Data Acquisition also include the Aborted state.

NotScheduled¶: Before Data Acquisition is acknowledged by DPM it remains in NotScheduled.

Scheduled¶: Data Acquisition is acknowledged by DPM and is scheduled for merging (i.e. the Data Acquisition is in the backlog).

Transferring¶: Inputs for Data Product are being transferred to DPM host in preparation for merging.

Merging¶: Data Product is being created by merging input files together.

Releasing¶: Data Product is being released to specified receivers, which in the normal case is OLAS.

Completed¶: Nominal final state for a Data Acquisition (i.e. final state when Data Acquisition is not aborted).

Undefined¶: Only used for technical reasons. This is not an expected state.

class ocmif.DaqException¶

Exception used by OcmDaq.

id: str¶: Data Acquisition identifier.

message: str¶: Exception message.

class ocmif.DaqStatus¶

Contains the Data Acquisition reply.

id: str¶: Data Acquisition identifier.

state: DaqState¶: Data Acquisition state at the time the reply was sent.

substate: DaqSubState¶: Data Acquisition sub-state at the time the reply was sent.

timestamp: double¶: Timestamp of last status update.

error: boolean¶: Identifies if request was completed with non-fatal error.

message: str¶: Contains error message.

class ocmif.DaqReply¶

Basic reply type that also indicate non-fatal errors.

id: str¶: Data Acquisition identifier.

error: boolean¶: Identifies if request was completed with non-fatal error.

class ocmif.AwaitDaqReply¶

timeout: boolean¶: Indicates if the await request timed out before condition was fulfulled.

status: DaqStatus¶: Data Acquisition status when reply was sent.