What is the ESO Data Processing System (EDPS)?
EDPS is a system to automatically organise data from ESO instruments for pipeline processing and running the pipeline on these data. It is used for quality control at ESO. The current public release is a beta version without a GUI. A GUI is being developed and the system is meant to eventually replace the older EsoReflex environment. If you make use of EDPS for a scientific publication, we request to reference Freudling, Zampieri, Coccato et al 2023.
EsoReflex is currently the recommended environment for for the reduction of VLT data with ESO pipelines because of its rich GUI, interactive features and detailed documentation including tutorials. However, for some users and in limited circumstances, EDPS is already now the best way to run ESO pipelines. We recommend to use EDPS for users
- who want to learn and experiment with the future environment to run ESO pipelines.
- who want to process large amounts of data in batch mode mostly using standard pipeline setup, in particular if these data are taken over an extended period of time.The advantage of EDPS in this situation is:
- EDPS is written in Python and is much more robust than EsoReflex.
- EDPS takes better advantage of available computational resources. In particular, recipes are run in parallel when possible, taking into account the internal parallelization of the recipes. Depending on circumstances, this might lead to an order of magnitude faster executions.
- Unlike EsoReflex, EDPS takes into account changes in the instrument hardware when organizing the data ("break points").
- who want to (re-)process calibration data independent of science data. EsoReflex is not set up to do this.
The current EDPS version 1.2.6 is a beta release and meant for experimental use.
The basic prerequisites for installing and running EDPS are:
- Python 3.9 or higher
- A working installation of at least one ESO pipeline that supports EDPS workflows. Currently the following pipeline include an EDPS workflow: ESPRESSO, UVES, and KMOS.
- The esorex command to run pipelines needs to be in the path.
EDPS can be installed using the following command in most Linux and MacOS versions:
pip install --extra-index-url https://ftp.eso.org/pub/dfs/pipelines/libraries edps adari_core
Detailed installation instructions are in the EDPS Tutorial.
EDPS is already used for quality control at ESO. For this reason, interfaces and general features are considered stable and well tested. However, we can not garantee at this point that there will not be some changes. In particular, workflows written for the current version of EDPS might need to be modified to run with future version of EDPS.
The general principles, workflow designs and workflow syntax have in Freudling, Zampieri, Coccato et al 2023 described in Documentation for EDPS is still work in progress. We recommend users to first look at this paper. A tutorial on how to install EDPS and run the published workflows should provide all the information to get started and reduce data with EDPS.
Please direct any and all feedback concerning installing or using EDPS to the EDPS support team.