VLT Instrument Pipelines

Latest News:

August 14, 2014: An update of the MUSE pipeline recipes is available!

The updated MUSE recipe package now includes the latest available version of the calibration files, and a fix to avoid using wrong reference data when searching for objects in the catalogs.
Read more...

July 04, 2014: An update of the MUSE pipeline recipes is available!

The updated MUSE recipe package now includes a first version of a Reflex workflow, and the cookbook contains a section on how to deal with flat fields taken at low ambient temperature.

June 25, 2014: A pre-release of the MUSE pipeline recipes is available!

In order to support the MUSE Science Verification an early version of the MUSE Data Reduction Software is now available!

June 25, 2014: New release of the X-shooter and UVES pipelines compatible with CPL 6.4.1 and Reflex 2.6

April 10, 2014: New release of all the pipelines compatible with CPL 6.4.1 and Reflex 2.6

April 01, 2014: Two new software tools to correct observations for atmospheric telluric absorption (Molecfit) and subtract sky emission for observations without plain sky information (Skycorr) are released for the first time. The tools have been developed for ESO by a team of astronomers at the Institute for Astro- and Particle Physics at the University of Innsbruck, as part of an in-kind contribution to ESO.

Nov 15, 2013: Reflex 2.5 has been released together with pipelines and workflows FORS 4.11.12, KMOS 1.2.6, SINFONI 2.4.0, UVES 5.3.0, VIMOS 2.9.9 and X-SHOOTER 2.3.0

Sep 19, 2013: KMOS 1.2.4 has been released.

Sep 09, 2013: UVES 5.2.5 has been released.

Introduction

In collaboration with the various instrument consortia, the Pipeline Systems Department has undertaken to implement data reduction pipelines for the most commonly used VLT/VLTI instrument modes. These data reduction pipelines have three main purposes:

  • Data quality control - pipelines are used to produce the quantitative information necessary to monitor instrument performance (see the ESO Quality Control web pages for more information).
  • Master calibration product creation - pipelines are used to produce master calibration products (e.g. combined bias frames, super-flats, wavelength dispersion solutions).
  • Science product creation - using pipeline-generated master calibration products, science products are produced for supported instrument modes (e.g. combined ISAAC jitter stacks; bias-corrected, flat-fielded FORS images, wavelength-calibrated UVES spectra). The accuracy of the science products can be limited both by the quality of the available master calibration products and by the algorithmic implementation of the pipelines themselves. In particular, adopted reduction strategies may not be suitable for all scientific goals. Therefore, ESO assumes no responsibility for the usefulness of reduced data for any specific scientific project.

Pipelines Running Environments: Reflex, Gasgano and Esorex

Reflex is the recommended way to reduce ESO data. It automatically organizes input files according to their category and runs the entire reduction chain at the push of a button. It supports break points in the reduction sequence in order to inspect and interact with intermediate and final products and rerun the corresponding step if necessary. The supported instrument modes are identified by a blue background color in the pipelines table

Gasgano is a Java-based data file organizer developed and maintained by ESO. It can be used to manage and organize in a systematic way the astronomical data observed and produced by all VLT compliant telescopes. Gasgano is a graphics-based software tool for organising and viewing data files produced by the VLT Control System. Gasgano offers functionalities for data viewing, grouping, sorting, classification, searching, and filtering of data. And, of course, Gasgano will run any requested CPL recipe on the selected data.

Esorex, a command-line utility for running pipeline recipes is also available. It may be embedded by users at their home institute into data reduction scripts for the automation of processing tasks.

Both Gasgano and Esorex are included in the pipeline public release packages.

Reflex-based Pipeline workflows

The pipelines released with a Reflex workflow are identified by the blue background rows in the pipelines table. Follow these steps to install the software (it includes Reflex itself, the pipelines and demo data sets):

Pipeline Releases Based on the most Recent CPL Release

 

The table below lists pipelines delivered with CPL 6.4.1, gasgano 2.4.5 and esorex 3.10.2. They can be installed together using this procedure.

 

Instrument Release Notes Package User Manual Cookbook Additional Documents Additional Datasets Reflex
AMBER 2014-04-09 4.3.1 4.3.1
CRIRES 2014-04-09 2.3.2 1.13
Cookbook
DETMON 2014-04-09 1.2.7 1.2.7
FORS 2014-05-19 4.12.12 4.8
Demo Data (29 MB)
Tutorial: 1.5
Demo Data: 0.5
GIRAFFE 2014-04-09 2.12.1 2.12
Cookbook
Standard Calibration Files page
HAWKI 2014-04-09 1.8.14 1.10
Demonstration Package (2,5 GB)
ISAAC 2014-04-09 6.1.4 1.4
Static Calibration Files (50 MB)
KMOS 2014-07-25 1.3.4 2.13
Tutorial: 1.2
Demo Data: 1.0
MIDI 2014-04-09 2.8.3 2.8.3
MUSE 2014-08-04 0.18.5 Cookbook
MUSE IFU 6 trace tables
Tutorial: 0.18.5
Demo Data: 1.0
NACO 2014-04-09 4.3.4 1.1
SINFONI 2014-04-09 2.5.2 19.1
ADA IV 2006 paper
Calibration Database Example (255 MB)
Demonstration Package (1.2 GB)
Tutorial: 1.1
Demo Data: 0.2
SOFI 2014-04-09 1.5.6 1.2
UVES 2014-06-25 5.4.3 22.8 (UVES)
18.2 (UVES-FIBRE)
Demonstration Package (2.0 GB)
Tutorial: 6.3 (UVES)
Tutorial: 1.2 (UVES-FIBRE)
Demo Data: 4.3
VIMOS 2014-05-08 2.9.15 6.10
Demonstration Package (1.7 GB)
Tutorial: 1.2 (VIMOS-MOS)
Tutorial: 2.2 (VIMOS-IFU)
Demo Data: 0.4
VISIR 2014-04-09 3.5.1 1.4
XSHOOTER 2014-07-31 2.5.2 12.4
Additional NIR telluric model calatog (190 MB)
Tutorial: 2.3
Demo Data: 1.1

Installation of Multiple Pipelines

The procedure for a simultaneous installation of multiple pipelines on a single account is:
  • Download the desired pipeline kits from the "Release File" column in the table below into an empty, temporary directory.
  • Download the Pipeline-kit installer into the same directory.
  • If necessary, make it executable: chmod u+x install_pipelinekit
  • Execute the command: ./install_pipelinekit *-kit-*.*
  • After the installation the temporary directory with its contents can be deleted.

 

Additional tools for data visualisation and analysis.

 

Previous Pipeline Releases

The Pipeline Systems Department also makes these data reduction pipelines available to the user community. They are based on older versions of CPL and can in general not be installed together.

Pipeline
With links to the instrument pages
Release Information Release File
Includes CPL,Gasgano,EsoRex,
and static calibration data
Users Manual and Documentation Data Sets
EFOSC 2.2.2 (2011-04-07) 2.2.2 (2011-04-07) EFOSC User Manual 1.0 (2011-02-02) Static Calibration Tables (included in kit) Demo Data (100 MB)
UVES (DISCLAIMER) 2.9.10 (2012-06-13) 2.9.10 (MIDAS based) UVES Pipeline User Manual 2.9.10
FLAMES-UVES Pipeline User Manual 2.9.10
A set of reference and master calibration data is included in the pipeline package.
DISCLAIMER:
MIDAS based release is offered with very limited support. We recommend users to use the Reflex based kit release.

Publications, Reference Documents, and Links


A list of Data Reduction Frequently Asked Questions is available.

The ESO Quality Control Group processes VLT data and monitors the instrument and data quality and also provides a summary of current pipeline status.

A Science Data Product Forum is available to exchange ideas, methods, software…

If you make use of Reflex workflows in a scientific publication, please consider citing Reflex. You may reference the following paper: Automated Data Reduction Workflows for Astronomy: The ESO Reflex Environment, 2013, Freudling, W., et al., Astron.Astrophys., 559, A96 (Bibtex entry)

The latest applicable version of the Data Flow for VLT/VLTI Instruments Deliverable Specifications (VLT-SPE-ESO-19000-1618) is version 3.