Attending:
STScI:Daryl Swade, Joe Pollizzi, Faith Abney, Scott Binegar, Paolo Padovani, Phil Hodge, Bob Hanisch ST/ECF: Alberto Micol, Benoit Pirenne, Rudi Albrecht CADC: Dennis Crabtree, David Schade, Daniel Durand, Severin Gaudet
STIS Exception Tables
Daniel and Severin brought up the question of the Institute's plans vis a vie STIS and its OTFC project. Joe reported that while OTFC is properly calibrating STIS, every STIS observation has effectively become an exception, and that we see no end in sight. This is overburdening our exception tables and, while working, is not a desirable effect. Moreover, the maintenance work required to support the tables in such a fashion is not sustainable.
As a result, the Institute has decided to re-process the data from the POD files on the fly and create completely updated raw data (i.e. updated headers) together with the calibrated data, on demand. We have found the additional overhead is marginal - especially when compared to the calibration steps themselves. An additional advantage of this approach is that we will use a common pipeline both for the realtime downlink processing and OTFC processing. Thus - we will get OTFC essentially for free for ACS, COS, and WF3 with this approach.
We then discussed the issue of how the updated header information could be relayed to the remote distribution sites (STECF, CADC, and NAOJ), since we will not archive the updated raw data. Part of the Institute's plan, now under discussion, is to re-ingest the headers into the database while not actually saving the data. This will not be done synchronously with the OTFC request - but sometime later. The re-ingest will, however, update all the appropriate science header tables in the DB, and these will be replicated to our remote sites. This was considered sufficient for CADC and STECF's purposes. They could then take advantage of the updated tables to drive their own OTFC implementations.
STIS Preview Images
We next discussed the issue of STIS preview images. It was relayed that due to other pressing work, this activity had been put aside at the CADC. However it was also pointed out, that to properly do the previews, they needed the STIS exception tables, so that the proper re-calibration of the data could be applied before the preview was constructed. The Institute replied that the tables should be soon available now that the STIS testing of OTFC was nearly complete.
We then went into a general discussion about previews and the trade-offs between doing them as part of the real-time pipeline (i.e. as the data is received) versus the current approach of waiting. As done currently, the preview process waits for the proprietary period to expire before executing, but this delay also benefits by having significantly improved calibrations from the original versions. One suggestion was that we do both. Once from the pipeline for a timely look at the data (and that could only be accessed or of use by the GOs and/or the Instrument teams supporting them) and then replaced with an improved version as the data becomes public. We didn't come to any definitive conclusions on this topic - other than to say that Institute still wants to consider an in-house approach for generating the previews.
We also briefly discussed what tools we might use to display the spectral preview data.
MO Migration Plan
Faith described the Institute's plan for migration of the DADS archive from optical disk to MO. Development work is going on now and testing is expected to begin in November. Migration will begin next year: new data will be written to MO, and old data will be written as it is requested from DADS by users (e.g., a data request to a given OD will trigger a process to copy the requested data sets and others on the same OD to MO). It was noted that our bulk-CD writer system can be converted to DVD.
Data Distribution Media
Both CADC and ECF have adopted DVD-R as archival media. STScI has acquired a DVD-R/W system for evaluation, primarily in order to support DVD media distribution to GOs and ARs. However, ECF and CADC are also interested in having our data deliveries to them be done on DVD instead of CD. We will do this as time/schedule permit, but probably not until at least six months from now (e.g., after the transition to MO is well in hand). The Institute agreed to create a test (DVD) disk early in the process to verify CADC's and ECF's ability to read the media.
CADC and ECF also briefly confirmed that we're currently in good shape with data synchronization. They are getting complete data in a timely fashion.
Starview II
CADC and ECF agreed to be "friendly user" test sites for SV-II, and ECF is willing to enhance JIPA, e.g., for improved preview capabilities (access to more complex data structures). The quiet collaboration on SV-II, with the contribution of JIPA from ECF, has been very successful!
NGST Archive Development
We discussed the potential roles/tasks that CADC and ST-ECF might play in the development of the NGST archive. Neither site is particularly interested in taking on portions of routine operational system upgrades, nor is STScI particularly interested in trying to farm out work that needs to be tightly integrated into systems at the Institute.
Hanisch, Baum, Postman, and Doxsey recently developed a list of potential project areas in which CADC and ECF could contribute to the NGST Archive effort. This was forwarded to GSFC by Ethan Schreier. The response from GSFC was that all of these activities were above the baseline and could not be used to buy down NASA's development costs.
It is not clear to any of us just how the baseline development costs for the archive were determined and just what they were intended to cover. So, before we give up on this we need to take a new look at the baseline and see if we can get things defined carefully.
Bob suggested that we should try to get an object catalog into the baseline, i.e., an object catalog would become a primary data project for the NGST mission. This could have implications for operations, i.e., we would insist that each observation have some minimum exposure time and at least two or perhaps three filters (or some minimum spectral coverage) in order to facilitate subsequent data mining and exploration capabilities. The group felt that this was an excellent plan and essential. The idea of a special workshop on this topic was suggested, perhaps for next spring. We will need to get community support and buy-in, and need to show how this effort would complement the emerging Virtual Observatory concept from the decadal panel review. In the meantime we will explore the idea of redefining the baseline accordingly, and work on this issue through the NGST operations study team that is working on operations concepts at STScI.