A release of the GRAVITY pipeline recipes is now available !

A release of the GRAVITY pipeline recipes is now available.

Together with the data processing modules, we also provide standalone applications for running them:

  • EsoReflex, a GUI which provides an easy and flexible way of runing pipeline execution workflows.
  • Gasgano, a GUI for data browsing and organisation, providing an intuitive and comprehensive interface to pipeline recipes that are based on the ESO Common Pipeline Library (CPL).
  • EsoRex, a command-line driven utility to launch pipeline recipes, which can be used either as an interactive tool, or can be embedded into scripts for automating some of the data reduction tasks.

Changes:

Version 1.6.6 Public Release (01 March 2024)

  • Fix retrieval of latest EOP parameters with recipe gravity_eop.
  • Add metrology demodulation algorithm.
  • Add new PCA calibration to improve differential phase calibration.
  • Add support for IDP products, including full OIFIT2 compliance.
  • Add option to remove unused pixels of FT.
  • Update on the tutorial and pipeline manual.
  • Add option to use separate calibrator for VISPHI.
  • New smoothing algorithm for the metrology faint/bright mode.
  • Improve performance of gravity_vis recipe by ~30%.

Version 1.6.0 Public Release (20 Jan 2023)

  • Add Strehl measurement for science channel in wide field mode (PIPE-9913).
  • Enable extra smoothing of metrology for GRAVITY faint.
  • Improved manual with astrometry tutorial.
  • Corrent calculation of pupil beacon spot angle for Gravity wide.
  • Improve performance of pupil image filtering.
  • Improve verbosity to better detect and deal with 2016 data.
  • Rename parameter phaseref-sc-maxdeg to phase-ref-sc-maxdeg.

Version 1.5.4 Public Release (19 May 2022)

  • Fixed gravity_dark recipe failure when processing data from the archive (2016).
  • Fixed reducing acquisition frames in wide field mode.
  • OCA rules updated to handle INS.MET.MODE.
  • Reflex OCA rules to distinguish between low and high FT detector gain.

Version 1.5.0 Public Release (01 July 2021)

  • Numerous changes to improve support for astrometry mode.
  • Store bad pixel map in P2VM if force-wave-ft-equal=TRUE.
  • Save astrored files when option astro-file is enabled.
  • New version of the pupil tracking algorithm.
  • Fix some failures in the gravity_p2vm recipe due to glitches in the metrology.
  • Smooth the FT metrology over 3 DITs to avoid high residuals when fitting the FT phase metrology.
  • Use by default the median column bias subtraction.
  • Include new roof position after hw intervention.
  • Solved pipeline failure with Strehl=nan.
  • Solved acquisition camera reduction failure when FEED MODE is DUAL-STS.
  • Solved GRAVITY pipeline crash when reducing the acquisition camera.

Version 1.4.1 Public Release (01 November 2020)

  • Fix installation problem with XCode 12 in macOS.

Version 1.4.0 Public Release (02 June 2020)

  • New static calibration files after hardware intervention.
  • Fixed problem loading gravity_viscal workflow with Reflex 2.10.
  • New calibration file GRAVI_OI_WAVE has been added that contains the effective wavelength bandwidth of each pixel. Also the minimum and maximum wavelengths to extract have been added to the static calibration file.
  • Added new QC parameters P2VM_MAXCOHERENCE and P2VM_TRANS_AVG/MIN/MAX to test the goodness of the P2VM.
  • Fix problem with wrong PA calculation of the camera when On-Axis.
  • Fix issue with QC P2VM_MINCOHERENCE_SC13 parameter which was always 0 in low resolution mode.
  • Fix issue with QC P2VM_PHASERMS_AVG_SC parameter which was too high low resolution mode.
  • Improved bias removal by detecting and discarding the illuminated pixels.
  • Ported workflow to Python 3 stack.
  • Added parameter gravity.viscal.smoothing to control TF smoothing.
  • The gravity_vis recipe now averages all accepted frames identifying affected channels and frames by cosmic rays and discarding them.
  • Dark compuation now uses sigma clipping instead of median.
  • The DISP_MODEL static calibration has been updated after the physical change of the grism.
  • PHASE_REF is now fitted against 1/lambda, rather than lambda.
  • The gravi_piezo recipe now handles main delay lines response measurement.
  • The residuals of the DISP fit are now included in the products.
  • For FT data the 6th pixel is now included in the reduction.
  • New QC parameters to judge the quality of the metrology dark: QC DARKRANGE MET and QC DARKRMS MET.
  • Support for reduction of dual-field on-axis data.
  • Synchronisation of the acquisition camera with respect to the metrology values is now supported.
  • OPD_TELFC is now unwrap around median value.
  • gravity_p2vm recipe does not use first pixel in calculation of QC parameters in low res.
  • Metrology data is now smoothed.

Version 1.2.4 Public Release (22 May 2019)

  • IMPORTANT: as of this release the gravity_image is not available using the rpm and macports installation methods. If the recipe is executed with esorex the user is informed that the recipe was compiled without yorick support and therefore is not available. If you need to use the gravity_image recipe then please use the source kit installation and install yorick together with the Yeti and OptimPack modules as described in software prerequisistes. Note that the gravity_image recipe will be deprecated in the near future.
  • Solved viscal workflow crash with python error.
  • Solved several pipeline failres in recipes gravity_wavelamp, gravity_disp, gravity_p2vm.
  • Support in gravity_vis and gravity_wkf workflow for DISP_MODEL static calibration.
  • Fixed issue in gravity_vis option --static-name=true that overwrote sky files.

Version 1.1.2 Public Release (24 April 2018)

  • New way to compute the wavelengths of the LOW spectral resolution mode.
  • Astrometry changes (including metrology) towards dual field observations of the galactic center.
  • Improvements on Strehl ratio computation.

Version 1.0.11 public release (16th November 2017)

  • Support for new DPR keywords.
  • WARNING!! The new pipeline is meant to work with the new DPR keywords present in raw data. If you have old data (before 1st October 2017) it is highly recommended to download the data again from the ESO archive, Care has been taken in the archive to properly update the FITS headers of old data.

Version 1.0.5 public release (26 April 2017)

  • Review the dispersion correction and the format of the calibration.
  • Support for new science beam combiner format after the change of intgrated optics in December.
  • Taking the new laser wavelength into account.
  • New Fringe Tracker data format (2 x 6 pixels).
  • Improve online recipe documentation and pipeline manual.

Pipeline status

In order to further improve the GRAVITY Data Reduction Software we encourage all users, and in particular the PIs of an observing program to provide feedback to the ESO User Support Department mentioning GRAVITY pipeline in the subject.

Known Issues

  • The timestamps of the OI_VIS/2 and OI_T3 columns have 12h artificially added.
  • The error bars are not always properly computed. In same cases they can be equal to zero even if they are valid data points. In some other cases they might contain meaningful values.
  • The spectral calibration between baselines is accurate to 1nm. It corresponds to is half a pixel (thus 1/4 of the spectral resolution element) in HR. This uncertainty can generate biases in the closure phase, which amount to around 3deg when observing at a group-delay of 40 microns. The absolute spectral calibration is accurate to 2nm, which correspond to half a spectral resolution element in HR.

GRAVITY pipeline recipes public version:

The public version of the GRAVITY pipeline recipes is 1.6.6, and it is included in the pipeline distribution kit gravity-kit-1.6.6.tar.gz together with the following packages:

  • the CFITSIO library, version 4.2.0
  • the Common Pipeline Library (CPL), version 7.3.2
  • Gasgano version 2.4.8
  • EsoRex, version 3.13.8
  • Third-party libraries used by CPL (wcslib, fftw)
  • Third-party libraries required by the GRAVITY pipeline

GRAVITY pipeline documentation

User Manual: 1.6.6
EsoReflex Tutorial: 1.5.0

GRAVITY pipeline recipes System Requirements

The GRAVITY pipeline kit version 1.6.6 is verified and supported on the VLT target platform:
  • CentOS 7 (x86_64), using gcc v4.8.5.
It is also verified that the pipeline kit can be built on the following platforms (x86_64, unless specified otherwise):
  • CentOS 7
  • Rocky Linux 8
  • Fedora 31
  • Fedora 34
  • Fedora 35
  • Fedora 36
  • Fedora 37
  • Ubuntu 20.04
  • Ubuntu 22.04
  • Ubuntu 22.10
  • Debian 10
  • Debian 11
  • macOS 10.15
  • macOS 11
  • macOS 12 (Intel & Apple Silicon)
  • macOS 13 (Intel & Apple Silicon)

Installing and running the GRAVITY pipeline recipes 1.6.6

There are several ways to install the pipeline on your machine. The recommended installation procedure depends on whether you are working on a Mac OSX or on Linux.

Installation using RPM repositories

For Fedora 35/36/37, CentOS 7 and Scientific Linux 7, it is recommended to install the pipeline from our RPM repository. The RPM repositories include not only the pipeline recipes but also the library dependencies (CPL, etc..), the EsoReflex workflow, demo data and the EsoReflex tool itself.

Please refer to the instructions provided here.

Installation using MacPorts repositories

For Apple macOS 11/12/13, it is recommended to install the pipeline with MacPorts. The MacPorts repositories include not only the pipeline recipes but also the library dependencies (CPL, etc..), the EsoReflex workflow, demo data and the EsoReflex tool itself.

Please refer to the instructions provided here.

Installation using the EsoReflex installation procedure

Please refer to EsoReflex Software Prerequisites and Installation Instructions for detailed instructions

Installation using the public pipeline kit

To install the GRAVITY pipeline recipes 1.6.6 you will need to retrieve the pipeline distribution kit, unpack and install it.

Prerequisites:
  • Please check the software prerequisites here.

Installation procedure:

  1. Change directory to where you want to retrieve the GRAVITY pipeline recipes 1.6.6 package. It can be any directory of your choice but not:
    • $HOME/gasgano
    • $HOME/.esorex
  2. Download the software and the static calibration data (line catalogs etc.) necessary for running the pipeline recipes:
  3. Unpack using for example the following command:     tar xzf gravity-kit-1.6.6.tar.gz
  4. Install: after moving into the installation directory,     cd gravity-kit-1.6.6 it is recommended to perform the installation using the supplied installer script:     ./install_pipeline (BEWARE: The installation may take several minutes). By default the script will install the GRAVITY recipes, Gasgano, EsoRex, all the necessary libraries, and the static calibration tables, into a directory tree rooted at $HOME. A different path may be specified as soon as the script is run. For instance (user input is boldfaced):
    $ ./install_pipeline
    I am about to install the following software packages:
    • cfitsio4.2.0.tar.gz
    • cpl-7.3.2.tar.gz
    • esorex-3.13.8.tar.gz
    • gasgano-2.4.8.tar.gz
    • gravity-kit-1.6.6.tar.gz
    with the following pipeline calibration file(s):
    • gravity-calib-1.6.6.tar.gz
    The software installation will be organised in bin/, lib/ and include/ directories, while the calibration files will be installed in a calib/ directory. Where should I install the software packages ? [/home/userid] /home/dummy/pipelines
    Where should I install the pipeline calibration files ? [/home/dummy/pipelines] /home/dummy/calibrations
    In this case the software is installed under the directory /home/dummy/pipelines and the static calibration tables under /home/dummy/calibrations/gravity The only exception to all this is the Gasgano tool, that will always be installed under the directory $HOME/gasgano Note that the installer will move an existing $HOME/gasgano directory to $HOME/gasgano.old before the new Gasgano version is installed.
    Important: the installation script would ensure that any existing Gasgano and EsoRex setup would be inherited into the newly installed configuration files (avoiding in this way any conflict with other installed instrument pipelines).
    Please follow the instructions shown at the end of a successful installation script: they will indicate how to set your PATH and CPLDIR variables in your machine in order to run the pipeline.
    Alternatively, it is possible to perform a manual installation (experienced users only): the README file located in the top installation directory contains more detailed information about a step-by-step installation.

Executing the pipeline 1.6.6

Using EsoReflex

  • To run the EsoReflex workflow, please refer to the GRAVITY workflow tutorial available in this table under column Documentation.

Using Gasgano

  1. To run the Gasgano GUI, just add $HOME/gasgano/bin to your PATH environment variable:     export PATH="$HOME/gasgano/bin:"$PATH You should also define an environment variable CPLDIR to point to the same path specified for the installation. Possible files to update are:     $HOME/.bashrc     $HOME/.profile Finally, enter the command     gasgano
  2. If the Gasgano main panel appears, then you have successfully installed Gasgano. Make sure, however, that the expected Gasgano 2.4.8 release is running: the version number is listed at the top of the main Gasgano panel.
  3. Refer to the Gasgano User's Manual to learn how to browse data and run pipeline recipes. Refer to the Pipeline User Manual(s) for detailed information about the recipes usage.

Using EsoRex

  1. The EsoRex executable should be available in the bin directory under your installation path (see above). The PATH environment variable should be upgraded accordingly (similarly as done for Gasgano).
  2. Refer to the EsoRex web page for details about the related features and options.

Documentation

The EsoReflex Users' Manual is availale for download

The GASGANO Users' Manual (~1.2 Mb, 66 pages) is available for download in the Gasgano web page.

On the EsoRex web page some online documentation about EsoRex can be found.

The CPL manuals are available on the CPL web pages.

In case of problems when opening the documents directly from your web browser, the files may be first saved on disk, and then opened with Acrobat Reader.


Bug Reports

If you experience an unexpected behavior of any component of the GRAVITY pipeline recipes package, please, first verify that you are using one of the above mentioned supported platforms and refer to the list of known problems and limitations in the pipeline manual of the current GRAVITY pipeline release.

For any other issues or requests, please, send a report to the ESO User Support Department mentioning GRAVITY pipeline in the subject, describing:        

  • the GRAVITY pipeline version (currently 1.6.6), and the version of other components (e.g., Gasgano, EsoRex, ...) you are using
  • the version of your OS and C compiler.
  • the exact sequence of actions that were performed before the problem occurred
  • what were precisely the symptoms and the possible error message(s)
  • whether the problem is repeatable