Overview

This section gives an overview of the components involved in acquiring data, which is performed by each component that provides data to the final data product and is coordinated by OCM and creating the data product to be archived and notifying the Online Archive System, which is performed by DPM. This means that other components in OCS are regularly not included.

Introduction

Stylistic Conventions

Note

Although diagrams sometime follow UML styling they are not created to be formally correct, but to convey information efficiently.

The following visual convention is used for components (or systems), control and data flow:

Fig. 2 Stylistic conventions

The direction of control refers to who the initiator is. The most common case is to use request/reply in which case the diagram shows how requestor initiates the control from A to command B.

Data flow is typically reserved for the out-of-band information that is not carried as part of the control flow. In practice data flow in this manual usually means FITS files being created by a component and that are later transferred for consumption in another component. Strictly speaking the actual transfer then depends on how components are deployed. The data flow can be thought both as the logical information transfer and the physical information transfer, depending on deployment and context.

Note

There are exceptions to using request/reply for e.g. the interaction with OLAS, but in this overview it can be considered conceptually equivalent.

Conceptual Model

A conceptual data model can be useful as a basis for understanding the more detailed documentation. The following diagram show data flow, data entities and multiplicities for how a single Data Product is created for a single Data Acquisition:

• For each Data Acquisition there are 1 to any number of Data Sources providing data.

• Each Data Source may be provide 0 to any number of FITS files (a data source may also only provide FITS keywords in request/reply interface).

• There is one instance of DPM handling a Data Acquisition that transfers all FITS files.

• DPM then produce a single Data Product.

• And can deliver that Data Product to any number of receivers using post-processing recipes (in practice this will almost always be one recipe with one receiver and that is to deliver the Data Product to the Online Archive System).

There is a special case for OCM where it is always included as a data source implicitly. OCM will always deliver a FITS file containing standard and user provided FITS keywords for the final Data Product primary HDU. So there will be at minimum 1 FITS file in a Data Acquisition, even if all user specified data sources do not provide any files.

Important

All configuration that is related to a Data Acquisition is a per-Data Acquisition property. Data Acquisitions are designed to be independent to allow concurrency without surprising side-effects. This also means there is no static data source configuration in OCM. This and other parameters are provided when creating the Data Acqusition.

Note

OCM and/or DPM is not responsible for deleting Data Products that might no longer be useful, after post-processing. This activity falls within the scope of an operational procedure to free disk space of files after confirming they can be removed.

Control and Data Flow

The following sections provides a simplified overview of the control and data flow from Data Acquisition to Data Product delivered to the Online Archive System (OLAS) 1.

Fig. 3 Process overview. Components marked with * are covered by this manual.

Normally it is the Sequencer that is the client when interacting with OCM, but of course any client will function the same. It requests new Data Acquisitions from OCM, specifying the sources to acquire data from and other parameters.

OCM coordinates the Data Acquisition by commanding a number of data sources such as science detectors, function controllers and telescope. In the diagram these are abstracted as the Data Source(s) component. When the Data Acquisition completes OCM commands DPM to create the Data Product from the acquired data.

There are no constraints on number or locality of data sources involved in a Data Acquisition. If a component implements supported interfaces correctly and is reachable over network, OCM can control it to acquire data. Refer to [RD2] for options when it comes to Data Acquisitions that span multiple ICSs.

Once the Data Product is complete it is delivered to the archive system OLAS or which ever post-processing recipe is configured.

1

DPM supports per Data Acquisition configurable post-processing recipes, but the standard, and also default, is to interface with OLAS to archive the Data Product. Refer to section Data Acquisition Process for additional details.

Control Flow

This section provides an overview of the resulting control flow for individual Data Acquisitions. OCM supports any number of concurrent, but independent, Data Acquisitions. For additional details on the Data Acquisition process and how to control it c.f. section Data Acquisition.

Fig. 4 Control flow overview.

Description of the control flow:

1. The client initiates a new Data Acquisition, specifying which data sources to acquire data from using the command StartDaq(). The client continues to be able to control the Data Acquisition using the OCM Data Acquisition MAL interface OcmDaq.

2. OCM coordinates the Data Acquisition by commanding data sources to start, stop or abort, as requested by client.

3. When data has been acquired OCM commands DPM to produce a Data Product from a specification on how to merge the data together.

4. When DPM has created the Data Product it is delivered to OLAS. This done using a special purpose interface and not a normal request/reply MAL control interface.

Data Flow

This section provides additional details on the data flow. To give the full picture of how the Data Products are formed the following diagram also show how Data Product FITS keywords can be provided as part of the control flow from the client.

Fig. 5 Data flow overview.

Description of the data flow:

1. When a new Data Acquisition is initiated FITS keywords can be provided both at the very beginning with the StartDaq() command and after it has started with the UpdateKeywords() command.

2. The individual FITS files created during Acquiring are transferred by DPM to the host where it is deployed, to be merged into the final Data Product. This also include OCM, which provides primary HDU keywords to be merged.

   The individual FITS files follow ESO guidelines and specifications and may contain, apart from mandatory FITS keywords, also ESO hierarchical keywords and/or FITS extensions.

   Note

   Files are transferred explicitly using scp or rsync (TBD) if source files are not reachable on a DPM local mount. Files are transferred implicitly if files are located on a distributed file system, but reachable from the DPM host (i.e. reachable on a locally mounted filesystem).

3. Once Data Product is created by DPM it is delivered to OLAS.

   If DPM is deployed on the same file system where files are delivered to OLAS, then no additional transfer is made. If the destination file system is either different or remote, another Data Product file transfer is made.

Data Product Creation

This section provide an overview of how the final Data Product is created from individual files. The process is fairly simple and mechanical to reduce configuration complexity. In addition the foreseen data volumes makes complicated processing prohibitively expensive. The rule of thumb is that acquired data should be created in the desired format rather than modifying it afterwards.

Given a list of FITS files:

1. One file will be designated the target for subsequent files to be merged into.

Hint

To create a primary HDU without data the FITS file produced by OCM can be used as target.

2. Subsequent FITS files are processed in order:

1. Primary HDU keywords are copied to target primary HDU, using default or user provided keyword filters.

2. FITS extensions are appended to target.

Note

Merging multiple single-HDU FITS files with data is not supported. Data sources should instead be configured to produce FITS file with one or more extensions that can be appended to the same target file.

Supported extension types include:

• Image (XTENSION = 'IMAGE   ')

• Binary Table (XTENSION = 'BINTABLE')

User is able to specify the file order and other aspects such as keyword filters, using Data Product configuration parameters provided with StartDaq() (TBC).