Installing PyHDRL

Installation from Pre-Built Packages

Note

Pre-built, binary packages will be made available in the future. This section will be updated with the installation instructions as soon as binary packages of PyHDRL are available.

Installation from Sources

While a system wide installation of PyHDRL is of course supported, the following installation instructions will show an installation using a Python virtual environment. It is expected that this is a more common approach followed by individual users when installing python packages.

It is however assumed that the dependencies needed to build and use PyHDRL can be installed using the system’s package management system or by installing an ESO instrument pipeline source kit.

Software Prerequisites

In order to build, install and use PyHDRL the following software packages, with the given minimum version (if indicated), is needed:

  • C++ compiler supporting C++17 (GCC 9.5, Clang 5)

  • CMake 3.12

  • Python 3.9 (including the development package, i.e. headers and libraries)

  • Python packages:

    • setuptools 45.0.0

    • pybind11 2.8.0

    • PyCPL 1.0.3

    • Sphinx (optional, for generating the API documentation)

    • pytest (optional for running the testsuite)

  • HDRL 1.6

  • CPL 7.2.2

Depending on whether PyHDRL will be installed system wide, local to the user, or in a Python virtual environment (or alike) the required packages must be available in the respective environment. While the compiler, Python itself, CPL and HDRL are generally expected to be provided by the system, or possibly the user’s local environment. CMake and the required Python packages however may be installed into the target virtual environment only, in case a virtual environment approach is chosen.

Getting the Source Code

The most recent version of the PyHDRL source distributions is available here:

Older releases of PyHDRL can be found in the PyHDRL archive.

Users with access to the ESO Gitlab may also clone the PyHDRL repository directly (ESO internal access only):

$ git clone https://gitlab.eso.org/pipelines/pyhdrl

Installing Development Tools

The following shows how to install the standard dependencies for an ESO pipeline source installation. These dependencies are also needed to build PyCPL. These dependencies can be installed using the package manager of the different systems. Please skip to the section that applies to your environment.

Fedora 36 and newer:
$ sudo dnf install gcc gcc-c++ zlib-devel make gzip bzip2 tar perl gawk sed grep coreutils pkgconfig
$ sudo dnf install cmake python3-devel
macOS 12 and newer:
$ sudo port install pkgconfig autoconf automake libtool bzip2
$ sudo port install cmake python312

To set Python 3.12 as the default Python version to use on macOS, run

$ sudo port select --set python python312
$ sudo port select --set python3 python312

Installing CPL

Installation instructions for CPL can be found in the CPL installation section of PyCPL.

Attention

Note that the same CPL installation must be used to build all necessary software components: HDRL, PyCPL, and PyHDRL. Make sure that the desired library search path configuration is set prior to building all components.

Installing HDRL

Currently HDRL must be installed from its source distribution. For historical reasons HDRL was not available as a shared library which is installed into the system or the local user environment as it was always the case for CPL. PyHDRL however requires that HDRL is available as a shared library installation.

The option to build and install HDRL as a shared library is available starting with HDRL 1.6.0. In addition to that building HDRL as a shared library is not the default configuration in order to stay compatible with the traditional way of embedding HDRL into a software project. Thus, building HDRL for use with PyHDRL requires a local, custom build for the time being.

Installing HDRL from Sources

Getting the HDRL 1.6.0 Source Package

Building PyHDRL requires an installed version of HDRL 1.6.0 or newer. Currently only the source package of the HDRL 1.6.0a pre-release version is available for download. It can be downloaded using the wget, or a similar, utility from the shell prompt like:

$ wget https://ftp.eso.org/pub/dfs/pipelines/libraries/hdrl/hdrl-1.6.0a.tar.gz

Alternatively it can also be downloaded using your browser.

Building HDRL

Once the download is finished, unpack the source package into a temporary directory. The example assumes that the source distribution was downloaded into the directory $HOME/Downloads. Unpacking the source distribution will create a subdirectory hdrl-1.6.0a. Change your current working directory to this directory:

$ cd <tmpdir>
$ tar -zxvf $HOME/Downloads/hdrl-1.6.0a.tar.gz
$ cd hdrl-1.6.0a

Decide where the built HDRL library will be installed. If you have a locally built and installed CPL in your user environment that was used to build PyCPL, then it is recommended to install HDRL into the same root directory that was used for CPL. This is not mandatory, but simplifies the setup and keeps the components together in a single directory tree.

The following uses <cpl-root-dir> to indicate the target root directory of the HDRL installation, assuming that CPL is also installed in that directory. This placeholder needs to be replaced by an actual, suitable installation directory.

Because the installation location of HDRL (and also CPL) will be needed later in the process again the two environment variables HDRLDIR and CPLDIR will be used to point to the respective installation directories:

$ export CPLDIR=<cpl-root-dir>
$ export HDRLDIR=$CPLDIR

To build HDRL as a shared library that can be installed and used by PyHDRL an additional command line option, --enable-strandalone, must be passed when configuring the source package prior to building it. A typical command to set up the HDRL sources is then:

$ mkdir build
$ cd build
$ ../configure --prefix=$HDRLDIR --enable-standalone --with-cpl=$CPLDIR --with-gsl=/opt/local

The option --enable-standalone is required. Otherwise a shared library which can be installed will not be created by the following build command. The options --with-cpl and --with-gsl, however, are only needed if CPL and the GSL library are not available in a standard system location. If these libraries are installed in a standard system location, the respective option may be omitted.

With the HDRL source set up properly, the HDRL library can now be build and installed:

$ make
$ make install

Activating the Python Virtual Environment

To prepare for building and installing PyHDRL the use of a Python virtual environment is strongly recommended. Furthermore, the PyCPL package must be available in the same Python environment where PyHDRL will be installed.

While PyEsoRex is not required to build PyHDRL, or just use PyHDRL a module in Python scripts, PyEsoRex needs to be available in the same Python environment where PyCPL and PyHDRL are installed, when PyCPL recipes should be run from the shell prompt is the standard way.

In the following <venv-root-dir> will have to be replaced by the path you have chosen as the installation directory of the virtual environment.

$ . <venv-root-dir>/pycpl/bin/activate

Installing PyHDRL

Once the required dependencies are installed the PyHDRL source distributions can be installed.

Preparing the Environment

The first step is to setup the environment so that PyHDRL uses the correct HDRL (and CPL) installation. In order to point the PyHDRL build system to the appropriate HDRL installation the environment variable HDRLDIR has to be set to the root directory of the HDRL installation. For instance, if the HDRL library is installed in the directory /usr/lib, then HDRLDIR must be set to /usr.

If the environment variables HDRLDIR and CPLDIR are not already set as suggested in Installing HDRL, then it needs to be done now:

$ export CPLDIR=<cpl-root-dir>
$ export HDRLDIR=<hdrl-root-dir>

Note

If in doubt where HDRL is installed, try to find the file libhdrl.so or libhdrl.dylib depending whether you use a Linux system or a macOS system, respectively.

Make sure the libraries from the HDRL and the CPL installation located in $HDRLDIR and $CPLDIR, respectively, are preferred when looking for libraries. This is particularly important if you have more than one HDRL or CPL installation on your system.

This is done by adding the HDRL and CPL library directories to the beginning of the library search path. The environment variable names differ, depending on the OS you are using:

Linux:

$ export LD_LIBRARY_PATH=$HDRLDIR/lib64:$HDRLDIR/lib:$CPLDIR/lib64:$CPLDIR/lib:$LD_LIBRARY_PATH

macOS:

$ export DYLD_LIBRARY_PATH=$HDRLDIR/lib:$CPLDIR/lib:$DYLD_LIBRARY_PATH

Note

If you exit from your shell session and start a new one later you will have to set these previously defined environment variables again before you can continue with the installtion.

The same applies if you are using PyHDRL in a newly started shell environment.

At this point it is recommended to update pip, setuptools, wheel and build, in particular if your virtual environment was created from a somewhat older Python installation:

(pycpl)$ python3 -m pip install --upgrade pip setuptools build wheel

Building PyHDRL

With the environment set PyHDRL can be build and installed from the source distribution packages in the one of the following ways:

Option 1:

The easiest way to install the latest available version of PyHDRL, without the need for downloading the source packages first, you can use the following command:

(pycpl)$ python3 -m pip install --extra-index-url https://ftp.eso.org/pub/dfs/pipelines/libraries pyhdrl
Option 2:

To install PyHDRL after getting the source distributions from the locations given in Getting the Source Code and placing them into a temporary directory <tmpdir>, use the command:

(pycpl)$ python3 -m pip install <tmpdir>/pyhdrl-0.1.0.tar.gz
Option 3:

Installing PyHDRL from a working copy of their respective GitLab repository, or a directory containing the unpacked source distribution package of PyHDRL, the following needs to be done.

(pycpl)$ cd <tmpdir>/pyhdrl
(pycpl)$ python3 -m pip install .

Verifying the Installation

Finally verify that the installation actually succeeded:

(pycpl)$ cd <tmpdir>
(pycpl)$ python3

From the Python interpreter prompt import the hdrl module:

>>> import hdrl

If this succeeds without any messages printed to the terminal then you have successfully installed PyHDRL!