Telemetry Republisher

Overview

The Telemetry Republisher RTC Component reads telemetry data in MUDPI format originating in the HRTC, and forwards (republishes) it using DDS reliable Multicast to one or more SRTC nodes.

Prerequisites

Telemetry data is published via RTI Connext DDS, so it needs to be installed.

Additionally a RTI DDS QoS XML file has to be provided and NDDS_QOS_PROFILES environment variable should point to it.

Here the environment variable setup for the RTC Tk provided (default) QoS file:

export NDDS_QOS_PROFILES=file://MyIntroot/resource/config/rtctk/RTCTK_DEFAULT_QOS_PROFILES.xml

The Telemetry Republisher works optimally for MUDPI traffic with Ethernet MTU of size 9000. Nevertheless it works also with different sizes. Thus it is recommended to first exercise Telemetry Republisher with default (by OS set) MTU, and afterwards change the MTU size for specific MUDPI traffic.

Customisation

The Telemetry Republisher component does not require any compile-time customization. It runs out of the box and only needs to be configured accordingly, please refer to section Configuration for more details.

Running

The Telemetry Republisher can be started (after deployment) by invoking the command:

rtctkTelRepub tel_repub_1 file:$INTROOT/run/myRtRepo/service_disc.yaml

Mandatory command line arguments are the component instance name (first) and the service discovery endpoint (second). Service discovery information is retrieved from the specified service registry located in $INTROOT/run/myRepo/service_disc.yaml.

The component can be stopped either by sending Exit command or by pressing Ctrl-C.

Commands

To initialise the Telemetry Republisher component send Init using the Client application:

rtctkClient tel_repub_1 Init file:$INTROOT/run/myRtRepo/service_disc.yaml

As a result the component reads its configuration from the Run-time repository, it creates entities for listening to MUDPI traffic on a UDP socket, and entities for publishing DDS agnostic topic. At this stage the reading and publishing is not yet started, but subscription to created DDS topic(s) can be established.

This can be seen in the logs. E.g. a log states that “connection” to the particular DDS topic (TestTopic0) i.e. corresponding DDS subscriber comes up:

[17:22:30:158][INFO ][tel_repub_1] on_publication_match (TestTopic0)
[17:22:30:158][INFO ][tel_repub_1] on_reliable_reader_activity_changed (TestTopic0): active_count: 1 active_count_change: 1 inactive_count: 0 inactive_count_change: 0

When a DDS subscriber “disconnects” to the particular (TestTopic0) Telemetry Republisher with name tel_repub_1 logs such a kind of message:

[17:24:55:625][INFO ][tel_repub_1] on_reliable_reader_activity_changed (TestTopic0): active_count: 0 active_count_change: -1 inactive_count: 0 inactive_count_change: 0
[17:24:55:625][INFO ][tel_repub_1] on_publication_match (TestTopic0)

Commands Enable/Disable are used to transition between states NotOperational.Ready and Operational.Idle where the Run command can be applied.

When sending command Run the republisher starts listening to the MUDPI traffic on the sockets and publishes aggregated topics via DDS. At this point it is possible to see log messages that are regularly printed. By default a messages is printed for each topic every 5000 samples, the message contains number of received MUDPI frames, number of frames per sample, receiving rate, estimated loop (sample) frequency, and the number of frames and samples skipped so far.

E.g. logs for MUDPI topic with TopicId 3 and 2, number of frames is 6:

[3] MUDPI processor rcvd 13434 packets (# frames 6) 990.58/s, elapsed 30.29 est. loopFreq 165.10 errors (0 : 1)
[2] MUDPI processor rcvd 15174 packets (# frames 6) 971.40/s, elapsed 30.88 est. loopFreq 161.90 errors (0 : 1)

The listening and publishing is stopped by sending the Idle command.

As all the Configuration for the Telemetry Republisher is static the Update command does not have any effect.

At component shutdown (e.g. using command Exit) statistical information is logged. For the DDS topic (TestTopic3) the log message contains information like total number of samples sent, number of ACKs and NACks, and ratio of NACks and sample sent. A too high ratio (>0.1) suggests problems with the network or subscriber saturation.

[09:03:56:796][INFO ][tel_repub_1] Statistics for DDS topic TestTopic3:
        Sent to 0 subscribers/readers
        unacknowledged_sample_count: 0
        full_reliable_writer_cache (times): 252
        high_watermark_reliable_writer_cache (times): 39
        unacknowledged_sample_count_peak: 50
[09:03:56:796][INFO ][tel_repub_1]  DDS DataWriter protocol status (TestTopic3): [sample pushed: 30524 (1607271744) HB: 4461 (142752) ACKs: 2949 (82572) NACKs: 275 (9820) ratio: 0.009009 Rejected: 0]

Configuration

Configuration for Telemetry Republisher component is stored as for other components in a file in YAML format. The configuration file name has to correspond to the name of the component instance. The configuration contains just the static part meaning that configuration can not be updated i.e. taken into account during running invoking Update command. If configuration is changed then the component should be restarted or reinitialized i.e. call Init.

The configuration can be divided into three groups: * common configuration * receivers configuration * DDS topics configuration

Common configuration

Example configuration (yaml): In the common configuration part can be specified the DDS QoS Library and profile to be used for setting QoS DDS entities like DDS participant and DDS publisher.

cfg attribute	Type	Description
dds_qos_library	RtcString	(optional) DDS QoS Library to be used for setting QoS of DDS entities like DDS participant and DDS publisher.
dds_qos_profile	RtcString	(optional) DDS QoS Profile to be used for setting QoS of DDS entities like DDS participant and DDS publisher.

In both cases the specified QoS Library and QoS Profile need to be contained in the file given by environment variable: NDDS_QOS_PROFILES. Both configuration parameters are optional. Example for a common configuration block:

static:

    dds_qos_library:
        type: RtcString
        value: RtcTk_Default_Library
    dds_qos_profile:
        type: RtcString
        value: RtcTk_Default_Profile
    ...

Receivers configuration

The Telemetry Republisher can listen to more receivers which are to be specified in mudpi_receivers section. Each receiver is specified as rcv_NN where NN goes from 00 to two digit number (index) of receivers - 1. E.g for two receiver we have: rcv_00 and rcv_01.

Note

rcv_1 wont work as index as it is just one digit, correctly this would be rcv_01. The index has to start with 00 (not 01). There should be no gap in numbers. E.g. rcv_00, rcv_01, rcv_09 will configure just two receivers.

For each receiver the following needs to be specified:

cfg attribute	Type	Description
ip	RtcString	IP address to be used for listening to. E.g one receiver can listen on one NIC the other on other. Important It can be just an IP address and not for example host name.
port	RtcInt32	Port to be listen to.
cpu_affinity	RtcInt32	(optional) CPU affinity (CPU/core number) for receiver thread.

Example configuring two receivers:

static:
    ...
    mudpi_receivers:
        rcv_00:
            ip:
                type: RtcString
                value: 127.0.0.1
            port:
                type: RtcInt32
                value: 6000
            cpu_affinity:
                type: RtcInt32
                value: 1
        rcv_01:
            ip:
                type: RtcString
                value: 127.0.0.1
            port:
                type: RtcInt32
                value: 6500
    ...

DDS Topic configuration

Similarly as for receivers, the Telemetry republisher can operate on many (DDS) topics. DDS topics are configured in topics section.

Each DDS topic is specified as topic_NN where NN goes from 00 to two digit number (index) of topics - 1. E.g for two topics we have: topic_00 and topic_01.

Note

topic_1 wont work as index is just one digit correct would be topic_01. The index has to start with 00 (not 01). There should be no gap in numbers. E.g. topic_00, topic_01, topic_09 will configure just two topics.

For each topic the following needs to be specified:

cfg attribute	Type	Description
name	RtcString	Topic name. Important: names should be unique per Telemetry republisher.
mudpi_topic	RtcInt32	Map to MUDPI topic id. Important: each MUDPI topic id needs to have corresponding DDS topic i.e. topic that has configuration that maps (=has defined mudpi_topic) to that specific MUDPI topic id.
rcv	RtcInt32	Receiver index where topic specified in mudpi_topic will be received. Important: receiver with particular index needs to be configured in mudpi_receivers section.
sim_freq	RtcInt32	(optional) If specified, and value is different than 0 the topic is generated with the specified frequency. As there is no need for corresponding MUDPI topic in this case mudpi_topic and rcv can be omitted. Important: The frequency should be reasonable not to get system too busy.

Example configuring three topics:

static:
    ...
    topics:
        topic_00:
            name:
                type: RtcString
                value: "TestTopic00"
            mudpi_topic:
                type: RtcInt32
                value: 0
            rcv:
                type: RtcInt32
                value: 0
        topic_01:
            name:
                type: RtcString
                value: "TestTopic01"
            mudpi_topic:
                type: RtcInt32
                value: 1
            rcv:
                type: RtcInt32
                value: 0
        topic_02:
            name:
                type: RtcString
                value: "TestTopic02"
            mudpi_topic:
                type: RtcInt32
                value: 1
            rcv:
                type: RtcInt32
                value: 0

The configuration can be always inspected using Configuration Tool e.g. to check if a certain configuration data point exists.

Errors

During the initialization i.e in On.NotOperational.Initialising several errors can occur:

In case of problem to create MUDPI/UDP receiver an error message is logged, and component goes to Error state.

[14:20:10:106][ERROR][tel_repub_1] Activity.Initialising: failed, exception: Component tel_repub_1 problem to create MUDPI receiver part
Source file: ../reusableComponents/telRepub/src/telRepubBusinessLogic.cpp
Line no.: 105
Function: CreateMudpiReceivers

The problem could be opening a UDP socket or similar. Maybe IP address does not exist on particular machine, maybe host name is given instead IP address.

If there is no MUDPI receiver that a particular topic wants to use as a source of MUDPI topic data we get an error:

[14:22:21:256][ERROR][tel_repub_1] Activity.Initialising: failed, exception: Component tel_repub_1 Receiver Index out of range: 6 not possible to assign MUDPI topic Id: 0
Source file: ../reusableComponents/telRepub/src/telRepubBusinessLogic.cpp
Line no.: 249
Function: CreateDdsPubs

This would work if we have (at least) 7 receivers defined in Configuration (indexed from 0 to 6), and created.

During the publishing i.e in Operational.Running state error messages can be logged:

If a sample is lost for particular MUDPI topic (TopicId: 3) the Telemetry Republisher reports (and continues) as:

[3] sampleId: 374758624 frameId: 1 jumped samples : 374758625

… and when it resynchronizes we get message like:

[3] sampleId: 374762680 frameId: 1 Synched again.

In case if there is no mapping between MUDPI topic Id and DDS topic such a message is logged.

TopicId: 1234 has no mapping to DDS topic

This message means that topic with Id 1234 has no corresponding mapping. It might be that there is no defined DDS topic in the Configuration that maps to 1234 i.e. has no mudpi_topic data point.

Timeout to send (write) to topic is reported with log:

[08:49:05:104][ERROR][tel_repub_1] [TestTopic2] SampleId: 374896008. DDS write timeout!

The internal telemetry republisher queue overrun and thus samples are lost for a particular topic (TestTopic1) at particular SampleId

[08:49:10:178][DEBUG][tel_repub_1] [TestTopic1] SampleId: 374906361 overrun, so far skipped 1705 samples. Last @: 374906360

In both cases the problem could be slow subscribers, or some other DDS problem.

Limitations and Known Issues

The data wrangling mechanism is not yet implemented.

The payload size of the agnostic topic is limited to 60000 bytes.

Not optimized for performance. Lower data rate with not too many topic. The performance depends also on the machine where the republisher runs.