![]() |
The ESO - DMD
|
| |
|
Next Generation Archive SystemIntroductionLate in 2000 ESO's Data Management Division started an initiative to face the challenges of the exploding data rate coming from the new instruments of the VLT. The main goal of this initiative is to give recommendations for the design and implementation of archive systems which are able to cope with a data rate in the order of TB per month in mean and some hundreds of GB during one observation night in peaks. Some of the high data rate VLT instruments are (VIMOS), for the VLTI instruments MIDI and AMBER the raw data rate is certainly also very high. In addition there are VST /Omegacam and VISTA under construction and design, respectively and the design of ALMA has also been started. Especially ALMA will produce a higher data rate than the whole fully equipped VLT On the other hand international activities like Virtual Observatories (AVO/NVO) and AstroGrid have been initiated to enable scientific exploitation across astronomical archives world wide. The ESO Science Archive as one of the biggest astronomical data centers should certainly play a central role in a future AVO, but it has to be prepared to support not only high level query mechanisms but also data reduction on substantial fractions of the whole data holdings. With the current archive design, where only one or two computers are building the gateway to the whole archive, large scale data reduction is unfeasible. Massive parallelism is the most promising approach to address this requirement. On this background the NGAS (Next Generation Archive System) was initiated April 2001 and has been operating since July 2001. Since then the project has undergone a continues enhancement and development. |
NGAS ImplementationThe following slide show gives an overview of the NGAS Project: PPS - HTML. Building up such a system is not so much a hardware problem, but mainly a problem of defining procedures and writing robust software to support the critical day to day archiving of bulk data. Retrieval and processing has to be optimized especially for the topology to enable efficient reduction and distribution of data out of the science archive. |
Documentation
|