The ESO - DMD

NGAS Project

Next Generation Archive System

Introduction

Late in 2000 ESO's Data Management Division started an initiative to face the challenges of the exploding data rate coming from the new instruments of the VLT. The main goal of this initiative is to give recommendations for the design and implementation of archive systems which are able to cope with a data rate in the order of TB per month in mean and some hundreds of GB during one observation night in peaks. Some of the high data rate VLT instruments are (VIMOS), for the VLTI instruments MIDI and AMBER the raw data rate is certainly also very high. In addition there are VST /Omegacam and VISTA under construction and design, respectively and the design of ALMA has also been started. Especially ALMA will produce a higher data rate than the whole fully equipped VLT

On the other hand international activities like Virtual Observatories (AVO/NVO) and AstroGrid have been initiated to enable scientific exploitation across astronomical archives world wide. The ESO Science Archive as one of the biggest astronomical data centers should certainly play a central role in a future AVO, but it has to be prepared to support not only high level query mechanisms but also data reduction on substantial fractions of the whole data holdings. With the current archive design, where only one or two computers are building the gateway to the whole archive, large scale data reduction is unfeasible. Massive parallelism is the most promising approach to address this requirement.

On this background the NGAS (Next Generation Archive System) was initiated April 2001 and has been operating since July 2001. Since then the project has undergone a continues enhancement and development.

NGAS Implementation

The following slide show gives an overview of the NGAS Project: PPS - HTML.

Building up such a system is not so much a hardware problem, but mainly a problem of defining procedures and writing robust software to support the critical day to day archiving of bulk data. Retrieval and processing has to be optimized especially for the topology to enable efficient reduction and distribution of data out of the science archive.

Documentation

NGAS Operations & Troubleshooting Guide

Manual to assist the NGAS Operators in carrying out their daily tasks in connection with NGAS. Contains also information to assist the operator in handling problems with the system.

NGAS Acceptance Test Plan & Hands-On Tutorial

This document is the Acceptance Test Plan for NGAS. It should be applied by the 'customer' receiving a new NGAS System or an upgrade of an existing NGAS System on the site. As the document is a walk-through of many essential features of the system, it can also be used as a Hands-On Tutorial to train NGAS Operators for the tasks to be maintained to operate the system.

NG/AMS User's Manual

This is a comprehensive manual for the NGAS SW, the Next Generation Archive Handling System (NG/AMS). It provides in-depth information about the standard operation of the NG/AMS SW but also information enabling an expert user to extend the system by adding plug-ins to it.