Phase 3 - News and Changes

Please find below any news regarding the Phase 3 process and a record of the past evolution in reverse chronological order.

14 May 2013

Phase 3 validator is ready to verify data submissions from spectroscopic surveys

The configuration of the Phase 3 validator has been updated and is now ready to be used for the verification of 1D spectra and generic images. No user action is needed to activate the new configuration. The Java code itself (validator.jar) remains unchanged. Next time validator.jar is launched, the updated configuration will be taken into account automatically.

Revised version of the Frequently Asked Questions

A revised version of the Questions and Answers is available, with a more detailed description of the metadata for the characterisation of 1D spectra and support for the "ANCILLARY.2DSPECTRUM" and "ANCILLARY.IMAGE" data formats.

Revised version of the Phase 3 policies

A revised version of the Phase 3 policies has been published, which now include specific information on VST, spectroscopic surveys and catalogue information, in addition to VISTA public survey policies.

14 March 2013

Phase 3 validator supports the "ANCILLARY.MASK" data format.

The configuration of the Phase 3 validator has been updated now including support for the "ANCILLARY.MASK" data format, which allows to describe the data quality of image arrays. Next time validator.jar is launched, the updated configuration will be taken into account automatically.

26 February 2013

Phase 3 validator is ready to verify VST OmegaCAM DR1 data.

The Phase 3 validator has been reconfigured and is now ready to be used for the verification of the VST OmegaCAM DR1 data.

No user action is needed to activate the new configuration. Next time validator.jar is launched, the updated configuration will be taken into account automatically. The Java code itself (validator.jar) remains unchanged.

11 January 2013

Phase 3 data format standard for OmegaCAM data products available.

To support the VST Public Surveys in their Phase 3 preparations the format standard for data products to be submitted via Phase 3 has been expanded and now includes OmegaCAM data products (Issue 5, Date 11/01/2013).

15 October 2012

Phase 3 data format standard for spectroscopic data available.

To support ESO spectroscopic surveys in their Phase 3 preparations the format standard for data products to be submitted via Phase 3 has been expanded and now includes spectroscopic data, i.e. extracted, calibrated, and fully characterized one-dimensional spectra.

The new ESO Science Data Products Standard (Issue 4, Date 15/10/2012) is based on the standard definitions, which were previously published in the External Data Products Standard, Issue 3 (25/05/2012). Note that the data format standard for VISTA public survey products (images and catalogues) remains unchanged, meaning that issue 3 still may be used for the preparation of these data.

As the scope of the new standard was enlarged to science data products in general, the document title has been adjusted accordingly. External data providers are not affected by this modification.

ESO Science Data Products Standard (Issue 4, Date 15/10/2012)
Please consult the CHANGE RECORD on p.2 for further details.

02 July 2012

Phase 3 Validator is ready to verify VISTA DR2 data.

DR2 images and source lists require a few modifications with respect to DR1 (see below). To this end the Phase 3 validator has been reconfigured and is now ready to be used for the verification of the VISTA DR2 data.

No user action is needed to activate the new configuration. Next time validator.jar is launched, the updated configuration will be taken into account automatically. The Java code itself (validator.jar) remains unchanged.

Here is a brief summary of changes to the data requirements with respect to DR1:

PROV keys were recommended for image data products (tile, pawprint etc.) submitted in 2011. The upgrade of the Phase 3 infrastructure that took place in May 2012 entails the requirement saying that processing provenance is compulsory metadata for Phase 3 data products delivered thereafter. Now the validator also checks the existence of the PROV keyword for VISTA tiles and pawprints.
RA, DEC keywords are required to point to the actual center of the image. For tiles RA/DEC should be set to the image centre (based on the WCS) instead of the previously tolerated telescope pointing of the first pawprint /offset position. Alternatively, the tile coordinates as defined by SADT can be copied (once converted to decimal degree format).
PRODCATG tokens are upper case, eg. SCIENCE.IMAGE. Although, the previous lower case format is still accepted for DR2, it is recommended to change the software in view to future releases.

Pls. see also: ESO External Data Products Standard (Issue 3, Date 22/05/2012), CHANGE RECORD on p.2. The validation of VISTA DR1 catalogue data products is unaffected by this update.

12 June 2012

Phase 3 FTP server: SSL/TLS encryption requirement dropped.

As of 12 June 2012 any standard FTP client can be used to transfer Phase 3 data including the clients previously suggested like lftp and FileZilla. The requirement to use the SSL/TLS encryption layer for connections to the Phase 3 FTP server at ESO has been dropped. phase3ftp.eso.org now accepts plain FTP connections (port 21) whereas the support for FTP over TLS has been removed.

Normally, the change is fully transparent to the user. If you have previously saved the details of the connection to phase3ftp.eso.org using FileZilla's site manager for instance, you may need to update your setting. In this case encryption needs to be disabled by selecting "Use plain FTP".

25 May 2012

Adding support for catalogue data submission with improved validation of data products

This is a major upgrade of the Phase 3 Infrastructure that entails the following changes:

ESO External Data Products Standard updated (Issue 3, Date 22/05/2012)
- Please consult the CHANGE RECORD on p.2
Guidelines for the Data Release Description updated: template extended; submission as PDF document;
Upgrade of the Phase 3 release validator (version 1.1.1) that now:
- supports the submission of science catalogue data;
- supports the submission of the Phase 3 release description as PDF document;
- improves checks on FITS keywords and their content to identify problems as early as possible in the validation process;
- provides support for PRODCATG tokens in uppercase;
- User documentation/manual updated.
Specific instructions for catalogue data submission added.
The Phase 3 release manager prompts the user for the title, i.e. a brief human-readable content description, when a new Phase 3 data collection is being created. Uniqueness of the data collection name is enforced.