1. Alarm

Document ID:

Revision:

2.2

Last modification:

Oct 22, 2025

Status:

Released

Repository:

https://gitlab.eso.org/cii/srv/cii-ias-bridge

File:

alarm.rst

Project:

ELT CII

Owner:

Marcus Schilling
Document History

Revision

Date

Changed/ reviewed

Section(s)

Modification

1.0

6.5.2020

jstrnisa/ bterpinc

All

Document creation.

1.1

5.7.2020

bterpinc

4

Added plugin tab example

1.2

15.10.2020

msekoranja

All

RIX updates

1.3

17.11.2020

msekoranja

4.3.2.

Replaced ias-webserver sender with iasWebServer Sender

1.4

13.1.2021

jrepinc

4.2

Alarm Configuration GUI updates

1.5

26.1.2021

jrepinc, msekoranja

4.2

Alarm Configuration GUI updates (RIXes PSI 19, PSI 21). Transfer function alarm priority paragraph added (PSI 39).

2.0

14.9.2022

mschilli

all

CII v3: new tooling and workflow for defining and monitoring alarms

2.1

18.03.2024

mschilli

0

CII v4: Public doc

2.2

22.10.2025

mschilli

0

CII v5: ported from Java to Py

Confidentiality

This document is classified as Public.

Scope

This document is a manual for the Alarm system of the ELT Core Integration Infrastructure software.

Audience

This document is aimed at Users and Maintainers of the ELT Core Integration Infrastructure software.

Glossary of Terms

API

Application Programming Interface

CII

Core Integration Infrastructure

CLI

Command Line Interface

CDB

IAS configuration database

DP

Data Point

GUI

Graphical User Interface

ES

ElasticSearch

JSON

Javascript object notation

OLDB

Online Database

URI

Uniform Resource Identifier

SVN

Subversion

IAS

Integrated Alarm System

DASU

Distributed Alarm System Unit

ASCE

Alarm System Computing Element

IASIO

Integrated Alarm System Input/Output

References

  1. ESO, Core Integration Infrastructure Requirements Specification, ESO-192922 Version 7

  2. ESO, Integrated Alarm System Architecture, ESO-293482 Version 2

  3. ESO, Integrated Alarm System Design, ESO-299387 Version 1

  4. ESO, Integrated Alarm System General information, https://github.com/IntegratedAlarmSystem-Group/ias/wiki

  5. https://www.eso.org/projects/elt/develop/cii/srv/latest/manuals/html/services.html

  6. https://www.eso.org/projects/elt/develop/cii/srv/latest/manuals/html/oldb.html

  1. https://github.com/IntegratedAlarmSystem-Group/ias/wiki/Transfer-functions-how-to

  2. https://github.com/IntegratedAlarmSystem-Group/ias/wiki/Transfer-function-in-python

  3. https://github.com/IntegratedAlarmSystem-Group/integration-tools/blob/develop/docs/DEPLOYMENT_GUIDE.md

  4. https://github.com/IntegratedAlarmSystem-Group/integration-tools/blob/develop/docs/WEB_APPS_DOCUMENTATION.md

  5. https://github.com/IntegratedAlarmSystem-Group/ias/wiki/ConfigurationDatabase

1.1. Overview

This document is the user manual for CII Alarm service. It explains how to set-up, start and stop the alarm system (chapter 3), how to configure alarms (chapter 4), and how to inspect and address alarms (chapter 4), including concrete working examples.

The advanced topics (chapter 5) are mostly only relevant for administrators.

1.2. Introduction

The CII Alarm System is based on the Integrated Alarm System IAS.

It operates on rules that observe the OLDB and decide whether an alarm should be raised. The alarm system is therefore fully configuration-based. An application never directly triggers an alarm. An alarm is represented by an OLDB datapoint as well. These datapoints are owned by the Alarm system, and you cannot write to them. You read these datapoints to find out which alarms are active or not.

IAS is a software whose main purpose is to get alarms and monitor points from different sources and generate alarms to present to the users who can range from operators in the control room up to engineers at their desks. The alarm system is a message-passing facility that routes information about abnormal situations detected from hardware or software to the user. For IAS architecture and design, please refer to the IAS architecture [2] (necessary to read to get to know IAS concepts) and design [3] documents.

IAS is based on pluggable architecture to integrate with external systems. To connect the IAS to the CII, the CII-IAS bridge was developed. The bridge consists of the following components:

  • OLDB-2-IAS - The long-running process that reads from OLDB and writes (via Kafka) to IAS.

  • IAS-2-OLDB - The long-running process that reads (via Kafka) from IAS and writes to OLDB.

Changes in CII 5:

The following components were retired

  • ciiAlarmCtl (replaced by ciiAlarmLoader plus cii-services)

The following components were added:

  • iasAlarmGui

The following features are not available in this version of the CII Alarm System

  • Acknowledgement

  • Shelving

  • Running the alarm system “in user space” (i.e. not as a system service)

1.3. Prerequisites

This section describes the prerequisites for using the CII Alarm service.

1.3.1. Set-up

An administrator (with root privileges) has to post-install the Alarm system on the host. The procedure is described in #Advanced-Topics/Installation below.

1.3.2. Start-Stop

Start the Alarm system services

cii-services start alarm

Stop the Alarm system services

cii-services stop alarm

1.4. Usage

The CII Alarm System operates on rules that observe the OLDB and decide whether an alarm should be raised. An application never directly triggers an alarm. An alarm is represented by an OLDB datapoint as well. These datapoints are owned by the Alarm system, and you cannot write to them. You read these datapoints to find out which alarms are active or not.

The general workflow is this:

  1. You define alarm rules

  • Step 1: You write an alarm definition file

  • Step 2: You pass the definitions to the alarm system

  • Step 3: You request the alarm system to read the new definitions

  1. You monitor and trigger alarms

  • Step 4: You monitor the alarm-datapoints

  • Step 5: You write to the input-datapoints

1.4.1. Write Alarm Definitions

Locations

  • The CDB is stored in /etc/cii/ias/

  • The alarm rules file can be stored anywhere (it is yours).

Create an alarm-rules file alarm-rules.yaml with some rules:

Example

/harmoni:
   /sensor1_alarm: !cfg.type:MinMaxAlarm {
      dp: /harmoni/fcs1/sensor1/value, lowOn: 30, highOn: 50,
      desc: Demo alarm for numerical value (lowOn, highOn)
   }
   /errorstates/tracking_alarm: !cfg.type:BoolAlarm {
      dp: /harmoni/fcs1/adc1/trackingfailure,
      desc: Demo alarm with a boolean input value
   }
   /hitemp/dcs_alarm: !cfg.type:MinMaxAlarm {
      dp: /harmoni/dcs1/detector/temperature1, highOff: 70.0, highOn: 100.0,
      desc: Detector High Temperature (highOff, highOn)
   }
   /device/error_alarm: !cfg.type:RegexAlarm {
      dp: /harmoni/dcs1/device/error_string, re: Error.*,
      desc: Demo alarm with string input
   }

1.4.1.1. Format

The alarm rules file is written in config-ng syntax. In the alarm rules file, each rule consists of these parts:

<name> : <type>
   - <input dp>
   - <prop 1>
   - <prop n>
   - <description>

Hints:
  name  : unique name of this alarm, must start with '/'
  type  : type of alarm function, see listing below
  input : path to plant datapoint, must start with '/'
  prop  : params to the alarm function, see listing below
  desc  : free text describing the alarm

1.4.1.2. Alarm Rule Types

These are the available types of alarm rules.

MinMaxAlarm
  Value range check, with optional hysteresis

  inputs:
      dp : address of an OLDB-datapoint of type number

  props:
      (specify at least one)
      lowOn   : Alarm activates when dp value goes below lowOn
      highOn  : Alarm activates when dp value goes above highOn
      lowOff  : Alarm deactivates when dp value goes above lowOff (Hysteresis)
      highOff : Alarm deactivates when dp value goes below highOff (Hysteresis)


BoolAlarm
  Alarm activates iff dp value is true

  inputs:
      dp : address of an OLDB-datapoint of type boolean
  props:
      <none>


RegexAlarm
  Alarm activates iff dp value matches regex

  inputs:
      dp : address of an OLDB-datapoint of type string
  props:
      re : regular expression

Hints

  • Arguments called “dp” are addresses of OLDB datapoints. Give the absolute path inside the oldb, but without the “cii.oldb://” prefix, i.e. the address shall start with a single slash.

  • The DP’s oldb-type must be compatible with the function-arg type. E.g., where an alarm function expects a double, avoid passing a String-DP (OldbString), but you can pass an Int-DP (e.g. OldbInt64Std).

Note: On the terminal, you can remind yourself of the Alarm Rule Types by running:

ciiAlarmLoader help

1.4.2. Inject Definitions

The ciiAlarmLoader tool reads the alarm definition file, and creates alarm config as well as other config in the format understood by IAS: supervisor config, converter config, etc. The IAS keeps this information in the IAS configuration database (CDB).

ciiAlarmLoader  load  alarm-rules.yaml
Note:

  • When you give it the standard name (‘alarm-rules.yaml’), you can omit the file argument to ciiAlarmLoader.

  • The alarm definitions are added to the CDB, if you have one already. If you want to start from scratch, run clear first, or use load –clear.

Unless you specify –skip-notify, the alarm loader will inform the alarm system services that they should re-read the alarm configuration.

Alarm rules for which the input-DPs are existing already will function instantly. For those with missing inputs, there will be a certain delay after the missing input-DPs were created (by any application, or by oldbloader, or by you e.g. using oldb-cli, see example below). The alarm system will discover the new inputs within 20 seconds.

Note:

IAS generally follows a fail-fast behaviour. If it detects bad config, the supervisor process will stop to indicate that the config needs fixing.

Example

You could create the input-DPs for the example rules file above by doing:

oldb-cli create  /harmoni/fcs1/sensor1/value  INT32
oldb-cli write   /harmoni/fcs1/sensor1/value  34

oldb-cli create  /harmoni/fcs1/adc1/trackingfailure  BOOLEAN
oldb-cli write   /harmoni/fcs1/adc1/trackingfailure  false

oldb-cli create  /harmoni/dcs1/detector/temperature1  DOUBLE
oldb-cli write   /harmoni/dcs1/detector/temperature1  56.0

oldb-cli create  /harmoni/dcs1/device/error_string  STRING
oldb-cli write   /harmoni/dcs1/device/error_string  "None"

1.4.3. Monitor Alarms

The Alarm System comes with these dedicated tools:

1. Alarm Gui (Qt application)

iasAlarmGui

2. Alarm Monitor service (system service)

http://<host_where_alarm_services_run>:5602/

3. ciiAlarmMon (command line executable)

ciiAlarmMon

Moreover, since the alarms are OLDB datapoints, you can monitor them with the OLDB API, OLDB GUI, or OLDB tools (all described in [6]), for example:

oldb-cli read  /harmoni/sensor1_alarm

oldb-cli subscribe  /harmoni/errorstates/tracking_alarm

1.4.4. Trigger Alarms

Since the inputs are OLDB datapoints, you can read/write them with the OLDB API, OLDB GUI, or OLDB tools (all described in [6]), for example:

oldb-cli read   /harmoni/fcs1/sensor1/value

oldb-cli write  /harmoni/fcs1/sensor1/value  99.9

Example 1

This example uses a built-in demo alarm, so will work on every installation

oldb-cli read  /sandbox/cii_diag/basic_alarm
Value:  NOMINAL

oldb-cli read  /sandbox/cii_diag/alarm_source_bool
Value:  false

oldb-cli write /sandbox/cii_diag/alarm_source_bool true
Value:  true

oldb-cli read  /sandbox/cii_diag/basic_alarm
Value:  PRIO_2

oldb-cli write /sandbox/cii_diag/alarm_source_bool false
Value:  false

Example 2

This example uses the alarms demonstrated earlier, so it can only work if you have followed the above examples.

In terminal A:

ciiAlarmMon

In terminal B:

# Current temperature
oldb-cli read /harmoni/dcs1/detector/temperature1
Value:  56.0

# Increase to 90 - no alarm
oldb-cli write /harmoni/dcs1/detector/temperature1 90

# Increase temperature above limit - alarm state will change
oldb-cli write /harmoni/dcs1/detector/temperature1 120.0

# Decrease to 90 - alarm still on
oldb-cli write /harmoni/dcs1/detector/temperature1 90.0

# Decrease to normal - alarm state will change
oldb-cli write /harmoni/dcs1/detector/temperature1 56

1.5. Advanced Topics

1.5.1. Design

Components:

  • cii-alarm-oldb2ias: receive OLDB points, and produce input values for IAS

  • cii-alarm-ias2oldb: receive alarm values from IAS, and write to OLDB points

  • ciiAlarmLoader: authoring tool for alarm config

  • ciiAlarmMon: terminal-based monitoring tool

  • cii-alarm-mon: web-based monitoring tool (systemd service)

  • Systemd Unit files:

    • cii-alarm-default-ias: unit files for IAS components

    • cii-alarm-bridge: unit files for CII components

    • cii-alarm-mon

Communication Flow:

               cii-ias-bridge                         IAS
              |--------------------------|           |------------------|

CTRL-APP -----> OLDB ---(sub)--> oldb2ias ---(udp)--> UDP-Plugin --+
                                                                   |
                                                                (kafka)
                                                                   |
                OLDB <---------- ias2oldb <----------(kafka)-- IAS Core
                 |                                                 |
               (sub)                                            (kafka)
                 |                                                 |
                 v                                                 v
               ciiAlarmMon                                IAS Alarm GUI
               cii-alarm-mon

1.5.2. Dynamic Prefixes

Like oldbloader, ciiAlarmLoader is able to dynamically rewrite oldb paths during the loading process. The standard prefix (via –prefix) will affect the path under which alarm points will be created, whereas the input prefixes (via –input-prefixes) will affect the paths under which input datapoints will be looked for.

Example:

By loading this alarm rule

/ins/alarm1: !cfg.type:BoolAlarm {
  dp: $INS/instrument/errorflag
}

with this command:

ciiAlarmLoader -prefix OPERATOR --input-prefixes "{INS:/harmoni}"

you get:

alarm point: cii.oldb:///OPERATOR/ins/alarm1
input point: cii.oldb:///harmoni/instrument/errorflag

1.5.3. Administration

1.5.3.1. Installation

The CII Alarm System is not part of the standard ECOS packages.

You can install the alarm system only on hosts where standard ECOS is already installed, and cii-postinstall has been performed, because it needs access to cii services like oldb and log.

Install 3rd-party libraries and cii-ias-bridge (as root):

dnf -y install python3-jep kafka ias
dnf -y install elt-ciisrv-ias-bridge

Configure your host (as root):

/elt/ciisrv/postinstall/cii-postinstall alarm

1.5.3.2. Start

To start the Alarm system

cii-services start alarm

1.5.3.3. Auto-start

Auto-start the Alarm system on boot (requires root privs)

cii-services enable alarm

1.5.3.4. Health Check

Check status of Alarm system

cii-services info

The relevant lines in the output are tagged with “[alarm]”.

1.5.3.5. Log Configuration

Redefine log levels (requires root privs)

# IAS-Bridge
/etc/cii/logconfig/cii-ias-bridge-logging.json

# Alarm Monitor
/elt/cii/logconfig/cii-alarm-mon-logging.json

1.5.3.6. Logs

Inspect Logs (requires root privs)

journalctl -e  -u cii-alarm-bridge  -u cii-alarm-ias

1.5.4. Troubleshooting

List alarm names

# List alarms defined in CDB
ciiAlarmLoader alarmids
# output:
#   /sandbox/cii_diag/basic_alarm
#   ...

List rules that were used for loading

# List source rules
ciiAlarmLoader rules

Trigger a CII alarm point:

# Read alarm point
testutil-alarm  point_read  /sandbox/cii_diag/basic_alarm

# Trigger an alarm point
# -> Note: ias2oldb owns these points. Accordingly, if ias2oldb is running
# it will revert your change within a few seconds. You writing these points
# is only reasonable (and allowed) when you are debugging something.
testutil-alarm  point_write  /sandbox/cii_diag/basic_alarm  PRIO_2

Check status of alarm system processes:

# Check processes active
systemctl status cii-alarm-*

Run an alarm system component in debug mode:

# Stop the respective service (as root), and run the component manually
systemctl stop cii-alarm-bridge-oldb2ias
/elt/ciisrv/bin/cii-alarm-oldb2ias -lc debug

Reset alarm definitions:

# Delete all alarm definitions (as root)
ciiAlarmLoader clear

Capture kafka messages and oldb events:

# Show alarm-related kafka messages and oldb events
testutil-alarm-stream
Supervisor refuses to start, or stops after start:

  • The supervisor purposefully terminates (as a fail-fast behaviour) if it finds the ias config is broken. Look for supervisor logs containing the phrase “marked as broken”. The idea behind this is to ensure that the alarm defs are indeed working as intended.

1.5.5. Work with IAS directly

For deeper debugging, you can work with IAS directly.

This is usually simpler if you first acquire the same environment that IAS uses:

# ... either by opening a new shell
bash --rcfile /etc/cii/alarm.env

# ... or by modifying your current shell
source /etc/cii/alarm.env

Run validity check of the IAS CDB:

# Check CDB sanity
iasCdbChecker -jCdb $ALARM_CDB_ROOT

1.5.6. Modify CDB directly

The IAS offers a number of additional configuration options, but they require an understanding of the internals of the IAS. Before modifying the IAS CDB directly, read the detailed description of the Alarm system architecture [2], and for all details on alarm configuration attributes see [4] and [12].

Table 1 contains a brief explanation of the configuration elements used in IAS. For the proper explanation of the elements, refer to the IAS documentation [2], [3], [4].

Table 1: Alarm system elements

Entity Name

Description

ASCE

An alarm system computing element that runs inside DASU and contains one transfer function.

DASU

An alarm system element that runs inside Supervisor and contains one or many ASCEs.

IASIO

An alarm system input/output entity.

Supervisor

A standalone process that contains one or many DASUs.

Transfer Function

A transformation function that calculates alarm states. It receives and outputs IASIOs.

Plugin

A plugin for IAS system that interfaces the IAS system with external systems (in CII case OLDB).

1.5.6.1. General Settings

General settings are configured in CDB/ias.json

Set LogLevel, refresh rate, etc. according to the IAS CDB documentation [12].

1.5.6.2. IASIOs

IASIOs are input and output entities of the Alarm system. To define a new IASIO, go to CDB/IASIO/iasios.json

For the IASIO, write the URI of the input data point (e.g. “/alarmtest/device/motor/input_int_dp”, i.e. the relative path of the input data point “cii.oldb:///alarmtest/device/motor/input_int_dp”) and set the type (e.g INT).

Create another IASIO for the output alarm, set the id (e.g. “/motor/input_int_dp_alarm”). The type must be set to “ALARM”. You can define if the operator can suppress/hide the alarm by setting canShelve to “True”.

Other values for both IASIOs can be left empty.

The fields shortDesc, docURL and emails are optional, they provide additional information on the IASIO. Also note that the fields canShelve, sound, and emails are only applicable to IASIOs of type ALARM. Template field allows templatization of the IASIO (refer to the IAS documentation on templatization).

1.5.6.3. Transfer Functions

Alarm system output IASIOs are calculated by Transfer Functions. Transfer functions are in the IAS domain and must be implemented and built there. IAS already provides a small set of Transfer functions. Please refer [8] to [9] and for details on Transfer functions. To import a new transfer function, go to CDB/TF/tfs.json and add a new entry.

To register a newly implemented transfer function, write the className (e.g. “org.eso.ias.asce.transfer.impls.MinMaxThresholdTF”) and implLang (e.g. “SCALA”).

1.5.6.4. DASUs

DASUs are containers for ASCE entities, that make alarm calculations using the Transfer functions. First DASU needs to be defined followed by its ASCE(s) with Transfer function to be used.

To define a new DASU, go to CDB/DASU/ and create a new file, or copy an existing one.

Set the name of the DASU file (e.g.”int_alarm_dasu.json”). Leave the other properties empty for the time being. The default logging level will be used and there will be no templatization of the DASU.

Go to CDB/ASCE/ and define a new ASCE file (e.g. “int_alarm_asce.json”). For the transferFunctionID choose one of the transfer function names defined earlier (e.g. “org.eso.ias.asce.transfer.impls.MinMaxThresholdTF”). In the outputId field enter the output IASIO defined earlier (e.g. “/motor/input_int_dp_alarm”). For the dasuId, write the name of the DASU defined above, (e.g. “int_alarm_dasu”).

For the Inputs, add as an input the IASIO name (e.g. “/alarmtest/device/motor/input_int_dp”) to be the (only) input of the ASCE. In the properties, specify the parameters for the transfer function (e.g. “org.eso.ias.tf.minmaxthreshold.highOn” with value 5, “org.eso.ias.tf.minmaxthreshold.highOff” with value 1)

Note: For the MinMaxThresholdTF transfer function, analogous lower limits minmaxthreshold.lowOff and minmaxthreshold.lowOn can be defined. If both upper and lower limits are defined in the same ASCE, the alarm will not distinguish between the input value going above or below the limits. To distinguish between these two cases, one needs to define two ASCE with the same input, one checking for the high value and the other checking for the low value.

The Alarm priority is determined by the Transfer function. Some implementations allow the priority to be configurable, e.g. org.eso.ias.asce.transfer.impls.MinMaxThresholdTF has a property named org.eso.ias.tf.alarm.priority for that reason. Valid values are: SET_CRITICAL, SET_HIGH, SET_MEDIUM (default), SET_LOW.

To add the ASCE to deploy in the DASU, write the ASCE name into the DASU file (in CDB/DASU/) of the respective DASU (i.e. int_alarm_asce)

To define the output of the DASU, edit the DASU file, and write the outputId as defined in the IASIO file (i.e. “/motor/input_int_dp_alarm”).

Finally, save your configuration changes, and ask the Alarm system to re-read its config, as described in section #Usage above.

1.5.6.5. Supervisor

To deploy DASUs we need to define a Supervisor first. Supervisor is a stand-alone process that hosts DASUs among with its ASCEs including needed Transfer Function processing.

To define a new Supervisor, go to CDB/Supervisors and create a new file, or duplicate an existing one.

Set the supervisor’s name (e.g. “test_supervisor”), and assign the hostname (e.g. “localhost”). In the field DASUs to Deploy, write the name of the DASU defined above, leave templateId and instance empty, meaning the DASU will not be processed as a template.

The template field defines what template to apply and the instance field defines the instance number used to apply when processing the template. For more information on templates refer to the IAS documentation.

Finally, save your configuration changes, and ask the Alarm system to re-read its config, as described in section #Usage above.

1.5.6.6. Plugin

The OLDB Plugin configuration foremost lists the set of IASIO IDs (each of them corresponding to an OLDB data point) to be monitored by the particular Plugin instance.

To define a new Plugin, go to CDB/Plugins/ and create a new file, or duplicate an existing one. Name the new plugin “CiiPlugin” and set the monitored system to “OLDB”. The values defines a set of data points to monitor. Add a new monitored value, and select the desired input. If needed change the “Refresh time” (for how long a value is valid if not updated) or other attributes.

Finally, save your configuration changes, and ask the Alarm system to re-read its config, as described in section #Usage above.

1.5.7. Run IAS directly

Running the alarm system fully “in user space” can be useful for local testing and integration tests.

The integration tests of the CII Alarm System provide some pointer how to do this.