VLT Instrument Pipelines

Latest News:

January 28, 2015: MUSE pipeline recipes v1.0.1 are available!

This is a maintenance release of the MUSE recipe package, addressing a number of Reflex workflow and documentation related issues.
Read more...

Introduction

In collaboration with the various instrument consortia, the Pipeline Systems Department has undertaken to implement data reduction pipelines for the most commonly used VLT/VLTI instrument modes. These data reduction pipelines have three main purposes:

  • Data quality control - pipelines are used to produce the quantitative information necessary to monitor instrument performance (see the ESO Quality Control web pages for more information).
  • Master calibration product creation - pipelines are used to produce master calibration products (e.g. combined bias frames, super-flats, wavelength dispersion solutions).
  • Science product creation - using pipeline-generated master calibration products, science products are produced for supported instrument modes (e.g. combined ISAAC jitter stacks; bias-corrected, flat-fielded FORS images, wavelength-calibrated UVES spectra). The accuracy of the science products can be limited both by the quality of the available master calibration products and by the algorithmic implementation of the pipelines themselves. In particular, adopted reduction strategies may not be suitable for all scientific goals. Therefore, ESO assumes no responsibility for the usefulness of reduced data for any specific scientific project.

Pipelines Running Environments: Reflex, Gasgano and Esorex

Reflex is the recommended way to reduce ESO data. It automatically organizes input files according to their category and runs the entire reduction chain at the push of a button. It supports break points in the reduction sequence in order to inspect and interact with intermediate and final products and rerun the corresponding step if necessary. The supported instrument modes are identified by a blue background color in the pipelines table

Gasgano is a Java-based data file organizer developed and maintained by ESO. It can be used to manage and organize in a systematic way the astronomical data observed and produced by all VLT compliant telescopes. Gasgano is a graphics-based software tool for organising and viewing data files produced by the VLT Control System. Gasgano offers functionalities for data viewing, grouping, sorting, classification, searching, and filtering of data. And, of course, Gasgano will run any requested CPL recipe on the selected data.

Esorex, a command-line utility for running pipeline recipes is also available. It may be embedded by users at their home institute into data reduction scripts for the automation of processing tasks.

Both Gasgano and Esorex are included in the pipeline public release packages.

Reflex-based Pipeline workflows

The pipelines released with a Reflex workflow are identified by the blue background rows in the pipelines table. Follow these steps to install the software (it includes Reflex itself, the pipelines and demo data sets):

  • Check the software prerequisites and installation instructions in the Release Notes of the new Reflex 2.6 workflows
  • Check the "Important Notes and Bug Reports" section of the Release Notes
  • For Apple Mac OS X 10.9 or newer, installation is supported with MacPorts. Refer to the instructions provided here. Otherwise proceed with the following steps:
  • Download the install_esoreflex script.
  • If necessary, make it executable: chmod u+x install_esoreflex
  • Execute the command and follow the instructions: ./install_esoreflex
  • If you make use of any of these workflows in a scientific publication, please consider citing Reflex. You may reference the following paper: Automated Data Reduction Workflows for Astronomy: The ESO Reflex Environment, 2013, Freudling, W., et al., Astron.Astrophys., 559, A96 (Bibtex entry)

Pipeline Releases Based on the most Recent CPL Release

 

The table below lists pipelines delivered with CPL version 6.5.1.They can be installed together using this procedure.

 

Instrument Release Notes Package User Manual Cookbook Additional Documents Additional Datasets Reflex Status
AMBER 2014-04-09 4.3.1 4.3.1
Operational on hold
CRIRES 2014-04-09 2.3.2 1.13
Cookbook
Operational on hold
DETMON 2014-11-08 1.2.7 1.2.7
Operational on hold
FORS 2014-10-01 5.0.0 4.9
Demo Data (29 MB)
Tutorial: 1.6
Demo Data: 0.5
Active
GIRAFFE 2014-11-14 2.12.2 2.12
Cookbook
Standard Calibration Files page
Active
HAWKI 2014-11-17 1.8.14 1.10
Demonstration Package (2,5 GB)
Active
ISAAC 2014-04-09 6.1.4 1.4
Static Calibration Files (50 MB)
End of maintenance
KMOS 2015-04-09 1.3.12 2.16
Tutorial: 1.2
Demo Data: 1.0
Active
MIDI 2014-04-09 2.8.3 2.8.3
End of maintenance
MUSE 2015-01-26 1.0.1 Cookbook
MUSE IFU 6 trace tables
Leagacy MUSE static calibrations
Tutorial: 2.0
Demo Data: 1.1
Active
NACO 2014-09-09 4.4.0 1.1
Operational on hold
SINFONI 2014-11-05 2.5.2 19.1
ADA IV 2006 paper
Calibration Database Example (255 MB)
Demonstration Package (1.2 GB)
Tutorial: 1.1
Demo Data: 0.2
Active
SOFI 2014-04-09 1.5.6 1.2
End of maintenance
UVES 2014-11-05 5.4.3 22.8 (UVES)
18.2 (UVES-FIBRE)
Demonstration Package (2.0 GB)
Tutorial: 6.3 (UVES)
Tutorial: 1.2 (UVES-FIBRE)
Demo Data: 4.3
Operational on hold
VIMOS 2014-11-17 2.9.15 6.10
Demonstration Package (1.7 GB)
Tutorial: 1.2 (VIMOS-MOS)
Tutorial: 2.2 (VIMOS-IFU)
Demo Data: 0.4
Active
VISIR 2014-04-09 3.5.1 1.4
Active
XSHOOTER 2014-11-05 2.5.2 12.4
Additional NIR telluric model calatog (190 MB)
Tutorial: 2.3
Demo Data: 1.1
Operational on hold

Pipelines Status

The pipeline status determines the level of ESO support in fixing the software. For any help with installation and usage of a pipeline, please contact the helpdesk ticketing system (e-mail to usd-help@eso.org or "Ask for help" link in he User Portal homepage). ESO internal (operations) users should request support from the pipeline developers team using JIRA. Here are the possible entries:
  • Active: Pipelines that are affected by a data product upgrade are considered active in the sense that a project team is actively working on them. These pipelines are fully supported and have the highest priority for support. The support consists in prompt evaluation of all requests.
  • Operational on hold: Pipelines that are on hold are not under active development and are of lower priority for support. Pipelines are put into this status when they are considered to be essentially completed, i.e. no enhancements or upgrades are currently under way. The software support consists in porting these pipelines to new CPL versions, carrying out emergency repairs and fixing trivial problems. The User Support Department will evaluate and answer if possible the requests for help received via helpdesk (usd-help@eso.org), but by default there will be no available support from pipeline developers.
  • End of maintenance: Pipelines in this status are the lowest priority for support. The only work performed on these pipelines is to make sure they can still run in the current ESO environment. The User Support Department evaluates and answers if possible the requests for help received via helpdesk (usd-help@eso.org). Tickets reporting problems on the pipeline will not receive any software support.

Installation of Multiple Pipelines

The procedure for a simultaneous installation of multiple pipelines on a single account is:
  • Download the desired pipeline kits from the "Release File" column in the table below into an empty, temporary directory.
  • Download the Pipeline-kit installer into the same directory.
  • If necessary, make it executable: chmod u+x install_pipelinekit
  • Execute the command: ./install_pipelinekit *-kit-*.*
  • After the installation the temporary directory with its contents can be deleted.

 

Additional tools for data visualisation and analysis.

 

Previous Pipeline Releases

The Pipeline Systems Department also makes these data reduction pipelines available to the user community. They are based on older versions of CPL and can in general not be installed together.

Pipeline
With links to the instrument pages
Release Information Release File
Includes CPL,Gasgano,EsoRex,
and static calibration data
Users Manual and Documentation Data Sets
EFOSC 2.2.2 (2011-04-07) 2.2.2 (2011-04-07) EFOSC User Manual 1.0 (2011-02-02) Static Calibration Tables (included in kit) Demo Data (100 MB)
UVES (DISCLAIMER) 2.9.10 (2012-06-13) 2.9.10 (MIDAS based) UVES Pipeline User Manual 2.9.10
FLAMES-UVES Pipeline User Manual 2.9.10
A set of reference and master calibration data is included in the pipeline package.
DISCLAIMER:
MIDAS based release is offered with very limited support. We recommend users to use the Reflex based kit release.

Publications, Reference Documents, and Links


A list of Data Reduction Frequently Asked Questions is available.

The ESO Quality Control Group processes VLT data and monitors the instrument and data quality and also provides a summary of current pipeline status.

A Science Data Product Forum is available to exchange ideas, methods, software…

If you make use of Reflex workflows in a scientific publication, please consider citing Reflex. You may reference the following paper: Automated Data Reduction Workflows for Astronomy: The ESO Reflex Environment, 2013, Freudling, W., et al., Astron.Astrophys., 559, A96 (Bibtex entry)

The latest applicable version of the Data Flow for VLT/VLTI Instruments Deliverable Specifications (VLT-SPE-ESO-19000-1618) is version 3.