What is PyHDRL?

The High Level Data Reduction Library (HDRL) represents a collection of homogenised, cross-pipeline algorithms implemented in C for pipeline developers. It was developed to decrease the implementation and maintenance efforts of various ESO Pipeline System Group pipelines. The algorithms have been verified against an extensive test suite and include error propagation.

The inclusion of HDRL can accelerate initial pipeline development and brings the ongoing benefit of future bug-fixes and improvements. Its high-level functionalities are complementary to the low-level functionalities of the Common Pipeline Library (CPL) as used by several VLT/VLTI pipelines.

Here we present a new Python library called PyHDRL that provides access to HDRL from Python. It builds upon the successful development of the Python interface to CPL, PyCPL, which is available from here. The two main capabilities of PyCPL are to run existing pipeline recipes from Python and to develop new pipeline recipes in Python. With the addition of PyHDRL, it is now possible to implement Python recipes that seamlessly call both CPL and HDRL routines. In this context, PyHDRL can be viewed as an expansion or companion library to PyCPL.

The PyHDRL library is constructed in a near identical fashion to PyCPL and shares many of its design influences. A key design choice was made to abstract away the hdrl_parameter objects from the user, providing a more pythonic interface to HDRL. There is therefore no equivalent Parameter class that users have to instantiate and pass between different functions. Instead, access to HDRL functionalities are provided via classes whose constructors are instantiated with the parameters required by the ubiquitous _parameter_create functions. Member functions of these classes then allow most functionalities to be executed, with the remainder reserved for static functions. This is demonstrated in the following equivalent HDRL and PyHDRL pseudocode examples from the tests for hdrl.func.Flat:

/* HDRL pseudocode */
hdrl_image * master = NULL;
cpl_image * contrib_map = NULL;
hdrl_parameter * collapse_params = hdrl_collapse_mean_parameter_create();
hdrl_parameter *flat_params = hdrl_flat_parameter_create(filter_size_x, filter_size_y, HDRL_FLAT_FREQ_LOW);
hdrl_flat_compute(imglist, NULL, collapse_params, flat_params, &master, &contrib_map);

# PyHDRL pseudocode
stat_mask = None
collapse = hdrl.func.Collapse.Mean()
flat = hdrl.func.Flat(filter_size_x, filter_size_y, hdrl.func.Flat.Mode.FreqLow)
results = flat.compute(imglist, collapse, stat_mask)
# results is a namedtuple (FlatResult) containing:
#     the master image (hdrl.core.Image) - accessible via results.master
#     the contribution map (cpl.core.Image) - accessible via results.contrib_map

As seen in the PyHDRL example above, named tuples are used for the return values of several functions, e.g. returned HDRL values and in functions that return multiple images.

PyHDRL Features

PyHDRL currently provides:

  • Language bindings for the HDRL C API modules: Effective Airmass, Bad-Pixel (including Cosmic Ray detection), Barycentric Correction, Bias, Object Catalog generation, Dark, Flat, Fringing, Limiting Magnitude, Overscan, Fixed Pattern Noise, Differential Atmospheric Refraction, Image and Cube Resampling, and Strehl. The design of the language bindings is such that they have a natural look and feel for Python developers, i.e. they are pythonic.

  • Language bindings for HDRL Image and ImageList, sharing many similarities with the PyCPL equivalents.

  • As PyHDRL is provided separately to PyCPL, several custom type casters were developed to allow PyHDRL to interface seamlessly with PyCPL objects including cpl.core.Image, cpl.core.ImageList, cpl.core.Mask, cpl.core.PropertyList, cpl.core.Table, cpl.core.Vector and cpl.drs.WCS. A test suite making use of the hdrl.debug module verifies that the translations are correct. The hdrl.debug has no other purpose than facilitating the execution of this test suite.

  • As mentioned above, tuples and namedtuples are used in specific circumstances. The hdrl.func.Collapse module uses namedtuples to return multiple generated images with helpful labels that allow users to identify each component product.

  • A design decision was made to make no bindings for HDRL values (hdrl_value). Instead, HDRL values are made available via a simple custom type caster. Functions expecting HDRL values (e.g. hdrl.core.Image.add_scalar) may take a tuple(double,double), representing the data and error components of the value. Similarly, functions returning an HDRL value (e.g. hdrl.core.Image.get_pixel) return a namedtuple (Value) with components data, error and invalid, where the boolean component invalid may be used to determine e.g. whether a pixel is invalid. For more details, see hdrl.core in API Reference. This tuple and namedtuple approach is closest to the C struct handling of hdrl_value in HDRL where it is common to specify values via e.g. (hdrl_value){2.0,0.1}. An alternative implementation could be to create bindings to a Value class, but this seemed unnecessary, since it is easier for users to specify a tuple via (2.0,0.1) instead of Value(2.0,0.1).

PyHDRL Current Release

The current release is PyHDRL 0.1.0.