The Beowulf Project
ESO's Data Management & Operations
Division and California Institute of Technology,
CACR are cooperating to investigate
the appropriateness and means of applying Beowulf-class computing
technology to the computing requirements of the ESO Very Large
Telescope Data Flow System.
Beowulf-class computing systems employing cluster of mass market PC
technology subsystems present an important opportunity in scalable
computing system architecture, offering unprecedented price-performance
ratio and system configuration flexibility, reduced vulnerability to
vendor vicissitudes, and stable industry standard software and
programming environments. The California Institute of Technology in
cooperation with NASA has been a leader in Beowulf-class computing
system software and applications throughout its recent emergence and
ESO initiated a contract with the Caltech Centre for Advanced Computing
Research in January 1998. The one year program was to assess the
usefullness of Beowulf technology to the data processing needs of
the European Southern Observatory for the Very Large Telescope (VLT)
project. The VLT is the world's largest optical observatory and is
capable of data rates and data volumes in excess of any esisting ground
or space based astronomical facility.
The objective of the ESO VLT Beowulf project is to establish the
effectiveness and means of applying unprecedented price-performance
Beowulf technology to ESO computing requirements in the near future.
Three major areas of data processing within the VLT framework are
recognized as potential opportunities for Beowulf-class computing systems.
Processing of astronomical data from the VLT archive: Once data from
the telescopes arrive in Europe, they enter the VLT Archive.
This facility will grow to 100 Tbytes within the first 5 years of
VLT operations. Individual data frames can be as large as 0.1 - 0.5
Gbytes and hundreds of frames can be produced each night of observation.
These frames will be processed before distribution to the astronomer
in much the same way the Hubble Space Telescope delivers processed as
well as raw data. Beowulf machines could be used to process these data
files. The processing is "embarrassingly parallel" in that all frames
are operated on by the same algorithm and each frame is essentially
identical.... a good fit to the course-grain parallelism
paradigm of a simple beowulf system. A prototype system has been
assembled at Caltech and tests done on a realistic data set. The
results are promising and if fully successful, a Beowulf system
(initial configuration of approximately 16 PIII PC's) will be
installed at ESO headquarters in Munich during 1999
for operational tests. Based on that system, a second Beowulf will
be installed in 2000 with ~128 nodes.
Many data processing tasks for the VLT involve real-time
feedback of information at the telescope. Detecting sources that vary in
their brightness or move on the sky can be done in real time via applying
Beowulf technology to the processing of raw data frames at the telescope.
Based on the success of the machines at ESO headquarters, Beowulf systems
may be operational at the observatory site some time in 2000
- ESO is also interested in look at Beowulf technology
for data storage and throughput. Using the PCs of the Beowulf as
"smart disks" coupled with fiber-channel technology and 100baseT
connections may allow us to stage large data sets (>100 GByte) for
A one year study of ESO data-flow computing requirements in 1998 has revealed
an important opportunity for advancing those needs through Beowulf clustered
computing technology. The proof-of-concept study performed by Caltech for ESO
determined the suitability of Beowulf in this application domain and demonstrated
the effectiveness of related application codes on a Beowulf, supported by software
tools developed as part of the study. Parallel efficiency of order 70% (35x
speedup using a 50 processor system) was demonstrated using ESO DFS analysis
recipes and astronomical data running on the Caltech Beowulf through project-developed
The second phase of the project will include the delivery and installation
of a prototype Beowulf system at ESO in Q3 1999 the development of a wide field
digital camera data reduction pipeline inside a new DFS pipeline infrastructure
that supports Data Reduction Systems (DRS) running on Beowulf-class resources
a production environment running the prototype Beowulf facility at ESO on astronomical
data from the Wide Field Imager on the ESO/MPI 2.2m telescope at La Silla Observatory
in Q4 1999.
The following links provide more information about the Beowulf project: