Phase 3 - News and Changes

Please find below any news regarding the Phase 3 process and a record of the past evolution in reverse chronological order.

16 September 2013

Upgrade of the Phase 3 system, release validator version 2.0.0

The Phase 3 system has been upgraded to support the publication of new data products, particularly data to be processed by ESO in the course of data flow operations, so-called internal data products (IDPs). The upgrade includes the following general improvements.

In order to catch problems earlier in the process, the integrity of the processing provenance information provided in the products is verified after data submission to ESO (i.e. when the release has been closed);
The Phase 3 system now fully supports recording the processing provenance of data products within one dedicated FITS binary table extension (cf. ESO SDP Standard, Issue 5, Sect. 2.4.2, for details);
Performance improvements and bugs fixes

Please upgrade to the new version of the Phase 3 release validator (v2.0.0) if you downloaded the Phase 3 release validator before.

20 August 2013

First Data Release of VST Public Survey VPHAS+ Imaging Data

Data products resulting from the first release of the ESO/VST Public Survey VPHAS+ are now available via the dedicated Phase 3 query interface at the ESO Science Archive Facility.

This data release covers the first 9 months of data-taking in which a tenth of the survey footprint was observed to desired quality. These first pointings have favoured the Galactic equator and some well-known open clusters of interest to the Gaia-ESO Survey. Reduced images and unstacked single-band source lists are provided. The total data volume of this data release is 749 GB.

The VPHAS+ public survey data release is accompanied by a comprehensive description.

15 July 2013

First Data Release from the MATISSE/OCA-ESO Project (AMBRE)

Stellar radial velocities, atmospheric parameters (effective temperature, surface gravity, mean metallicity [M/H]) and enrichment in alpha-elements ([alpha/Fe]) resulting from the automatic analysis of 21551 FEROS spectra from the ESO data archive with the MATISSE algorithm have been released. Go to the ESO Catalogue Facility(*) for data download and further information.

(*) Note: requires authentication through the ESO User Portal.

11 July 2013

First Release of VST Public Survey KiDS Imaging Data

Data products resulting from the first release of the ESO/VST Public Survey KiDS are now available via the dedicated Phase 3 query interface at the ESO Science Archive Facility.

This data release consists of the first 50 “tiles” of KiDS, each observed in all four filters (u,g,r,i), during the first year of regular operations (15 October 2011 to 31 September 2012) and during Early Science Time (13 August to 15 October 2011). A total of 800 files is included in this data release. The release consists of astrometrically and photometrically calibrated mosaiced and coadded images (each 1.0 deg²), weight maps, FLAG mask images and associated single band source lists in the different bands of the survey. The total data volume of this data release is 702 GB.

The KiDS public survey data release is accompanied by a comprehensive description.

Deep HAWK-I J and Ks imaging of the X-ray luminous galaxy cluster XMMU J2235.3-2557

Data products resulting from ESO programme ID 60.A-9284(H) as part of the first HAWK-I science verification are now available via the dedicated Phase 3 query interface at the ESO Science Archive Facility.

This data release consists of a mosaic of four pointings in two filters J and Ks, with the center of the X-ray luminous galaxy cluster XMMU J2235.3-2557 imaged in the four extreme corners of the HAWK-I detector mosaic. The deep image covers 13' by 13' on the sky with a pixel scale of 0.1065''. The image quality varies from 0.3'' to 0.7'' depending on filter and location.

The 60.A-9284(H) data release is accompanied by a comprehensive description.

28 June 2013

Second Release of ESO/VISTA Public Survey Imaging Data

New data products resulting from the ESO/VISTA public surveys are now available via the dedicated Phase 3 query interface at the ESO Science Archive Facility. The current release of data from the VVV, VHS, VIKING, and VMC surveys (6.2 TB) covers mostly the period from October 2010 to September 2011 and consists of astrometrically and photometrically calibrated mosaiced and coadded images (each 1.5 deg²), weight maps and associated single band source lists in the different bands of each survey.

The new data largely complement the first release of VISTA public survey data products from 2011 adding up to 12 TB of (compressed) imaging data. Each public survey data release is accompanied by a comprehensive description.

14 May 2013

Phase 3 validator is ready to verify data submissions from spectroscopic surveys

The configuration of the Phase 3 validator has been updated and is now ready to be used for the verification of 1D spectra and generic images. No user action is needed to activate the new configuration. The Java code itself (validator.jar) remains unchanged. Next time validator.jar is launched, the updated configuration will be taken into account automatically.

Revised version of the Frequently Asked Questions

A revised version of the Questions and Answers is available, with a more detailed description of the metadata for the characterisation of 1D spectra and support for the "ANCILLARY.2DSPECTRUM" and "ANCILLARY.IMAGE" data formats.

Revised version of the Phase 3 policies

A revised version of the Phase 3 policies has been published, which now include specific information on VST, spectroscopic surveys and catalogue information, in addition to VISTA public survey policies.

14 March 2013

Phase 3 validator supports the "ANCILLARY.MASK" data format.

The configuration of the Phase 3 validator has been updated now including support for the "ANCILLARY.MASK" data format, which allows to describe the data quality of image arrays. Next time validator.jar is launched, the updated configuration will be taken into account automatically.

26 February 2013

Phase 3 validator is ready to verify VST OmegaCAM DR1 data.

The Phase 3 validator has been reconfigured and is now ready to be used for the verification of the VST OmegaCAM DR1 data.

No user action is needed to activate the new configuration. Next time validator.jar is launched, the updated configuration will be taken into account automatically. The Java code itself (validator.jar) remains unchanged.

11 January 2013

Phase 3 data format standard for OmegaCAM data products available.

To support the VST Public Surveys in their Phase 3 preparations the format standard for data products to be submitted via Phase 3 has been expanded and now includes OmegaCAM data products (Issue 5, Date 11/01/2013).

15 October 2012

Phase 3 data format standard for spectroscopic data available.

To support ESO spectroscopic surveys in their Phase 3 preparations the format standard for data products to be submitted via Phase 3 has been expanded and now includes spectroscopic data, i.e. extracted, calibrated, and fully characterized one-dimensional spectra.

The new ESO Science Data Products Standard (Issue 4, Date 15/10/2012) is based on the standard definitions, which were previously published in the External Data Products Standard, Issue 3 (25/05/2012). Note that the data format standard for VISTA public survey products (images and catalogues) remains unchanged, meaning that issue 3 still may be used for the preparation of these data.

As the scope of the new standard was enlarged to science data products in general, the document title has been adjusted accordingly. External data providers are not affected by this modification.

ESO Science Data Products Standard (Issue 4, Date 15/10/2012)
Please consult the CHANGE RECORD on p.2 for further details.

02 July 2012

Phase 3 Validator is ready to verify VISTA DR2 data.

DR2 images and source lists require a few modifications with respect to DR1 (see below). To this end the Phase 3 validator has been reconfigured and is now ready to be used for the verification of the VISTA DR2 data.

No user action is needed to activate the new configuration. Next time validator.jar is launched, the updated configuration will be taken into account automatically. The Java code itself (validator.jar) remains unchanged.

Here is a brief summary of changes to the data requirements with respect to DR1:

PROV keys were recommended for image data products (tile, pawprint etc.) submitted in 2011. The upgrade of the Phase 3 infrastructure that took place in May 2012 entails the requirement saying that processing provenance is compulsory metadata for Phase 3 data products delivered thereafter. Now the validator also checks the existence of the PROV keyword for VISTA tiles and pawprints.
RA, DEC keywords are required to point to the actual center of the image. For tiles RA/DEC should be set to the image centre (based on the WCS) instead of the previously tolerated telescope pointing of the first pawprint /offset position. Alternatively, the tile coordinates as defined by SADT can be copied (once converted to decimal degree format).
PRODCATG tokens are upper case, eg. SCIENCE.IMAGE. Although, the previous lower case format is still accepted for DR2, it is recommended to change the software in view to future releases.

Pls. see also: ESO External Data Products Standard (Issue 3, Date 22/05/2012), CHANGE RECORD on p.2. The validation of VISTA DR1 catalogue data products is unaffected by this update.

12 June 2012

Phase 3 FTP server: SSL/TLS encryption requirement dropped.

As of 12 June 2012 any standard FTP client can be used to transfer Phase 3 data including the clients previously suggested like lftp and FileZilla. The requirement to use the SSL/TLS encryption layer for connections to the Phase 3 FTP server at ESO has been dropped. phase3ftp.eso.org now accepts plain FTP connections (port 21) whereas the support for FTP over TLS has been removed.

Normally, the change is fully transparent to the user. If you have previously saved the details of the connection to phase3ftp.eso.org using FileZilla's site manager for instance, you may need to update your setting. In this case encryption needs to be disabled by selecting "Use plain FTP".

25 May 2012

Adding support for catalogue data submission with improved validation of data products

This is a major upgrade of the Phase 3 Infrastructure that entails the following changes:

ESO External Data Products Standard updated (Issue 3, Date 22/05/2012)
- Please consult the CHANGE RECORD on p.2
Guidelines for the Data Release Description updated: template extended; submission as PDF document;
Upgrade of the Phase 3 release validator (version 1.1.1) that now:
- supports the submission of science catalogue data;
- supports the submission of the Phase 3 release description as PDF document;
- improves checks on FITS keywords and their content to identify problems as early as possible in the validation process;
- provides support for PRODCATG tokens in uppercase;
- User documentation/manual updated.
Specific instructions for catalogue data submission added.
The Phase 3 release manager prompts the user for the title, i.e. a brief human-readable content description, when a new Phase 3 data collection is being created. Uniqueness of the data collection name is enforced.