Science Archive and the Digital Universe
Science Archive
The Science Archive Operation Group receives and redistributes ESO data and provides front-line archive user support.
About 200 Terabytes (TB) of public data are distributed per year through the ESO archive. As of October 2017, the total archive holding is about 1.01 Petabytes (PB), 44.8 million files, with an ingestion rate of about 131 TB per year. This will soon drastically increase by a factor of 10 or so as the Visible and Infrared Survey Telescope for Astronomy (VISTA) with its near infrared camera will alone produce about 150 TB of data each year.
ALMA science archive comprises about 20.8 million files, 416 TB of data, at an ingestion rate of about 200 TB per year. ALMA also delivers about 200 TB per year to external users.
ESO's enterprise-class database servers are coordinated between Germany and Chile, and their technology and complexity rivals that of major commercial enterprises such as the international banking community.
The Digital Universe
Major breakthroughs in telescope, detector, and computer technology now allow astronomical surveys to produce massive amounts of images, spectra, and catalogues. These datasets cover the sky at all wavelengths from gamma- and X-rays, through optical and infrared, to radio waves.
Astronomers are developing ways to do new science, by making the huge amount of data in this 'digital Universe' easily accessible. These techniques use the GRID paradigm of distributed computing, with seamless and transparent access to the data through 'Virtual Observatories'.
Just as a physical observatory has telescopes, each with unique astronomical instruments, a Virtual Observatory consists of data centres, each with unique collections of astronomical data, software systems and processing capabilities.
This global, community-based initiative is being developed world-wide under the auspices of the International Virtual Observatory Alliance (IVOA) and in Europe under the framework of the EURO-VO project.
Virtual Observatories have already proved their effectiveness, for example by discovering 31 new optically faint, obscured quasar candidates in the existing Great Observatories Origins Deep Survey (GOODS) fields, quadrupling the number previously found. The discovery means that surveys of powerful supermassive black holes have so far underestimated their numbers by at least a factor of two, and possibly by up to a factor of five (ESO Press Release eso0418).