The CalSelector service
CalSelector is an archive service that, starting from the results of a query for raw science files, groups together the raw science files that need to be calibrated together (e.g. an infrared jitter sequence) and associates and returns all the raw and static calibrations needed to process the raw science files themselves. Excerpts from the relevant Night Logs and an xml representation of the calibration cascade are added as ancillary files to the request, as well. The file category provided in the xml file is suitable to be provided as input to the ESO pipelines (for more information on the pipelines, please refer to the relevant User Manuals here). A group of raw science files, calibrations and ancillary files is referred to in this context as a dataset.
The set of raw and static calibrations needed to process the raw science files is defined by the calibration plan of that instrument and instrument mode. It is not possible to customize it by, e.g., specifying the type of files needed, nor their number. The calibration plans are distributed as part of the instrument documentation (http://www.eso.org/sci/facilities/paranal/instruments/instrumentName/doc, e.g. http://www.eso.org/sci/facilities/paranal/instruments/hawki/doc).
The time coverage by instrument can be found here. The time coverage is expected to progressivley increase as we are constantly striving to extend it as far back in the past as possible. A list of Frequently Asked Questions (FAQs) is also available.
Important notes the current release:
- The tool only works with the "Instant download" delivery option. The "FTP or media delivery" option is not supported.
- In order to allow for associations to become available as soon as possible after data acquisition at the observatory (typically within two hours), the associations themselves are at first based on uncertified calibrations. They are, then, replaced with associations based on certified calibrations as soon as the calibrations themselves are checked for quality, typically after two working days (the certification status is recorded by the "certified" flag in the top <association> element in the xml file describing the association). In order to access these improved associations you need to re-run the CalSelector by resubmitting the archive request (see below for the detailed steps).
- Under certain circumstances CalSelector returns more science files than actually present in the archive request. This is by design and is intended to provide complete datasets for meaningful processing, e.g. an infrared "on/off-target" sequence where the "off-target" positions may otherwise not be included in the original archive request.
- Under certain circumstances CalSelector does not return an xml file if no association could be established for the input science files (normally it should return an xml file in all cases). The requested input science files are, however, correctly made available for download.
- Under certain circumstances, including a mixture of science and non-science (e.g. acquisition images) files in the same request results in CalSelector behaving erratically. Users are strongly encouraged to include only science files in their query (see below for the details on how to proceed) and let CalSelector associate the other relevant files.
2. In the main query form, if you are interested in a specific programme, e.g. if you are the PI of that programme, just enter its ID and, possibly, a time interval for the nights you are interested in.
In general, in addition to your other search criteria you should also select Science as Category.
3. In the result page, mark the desired files for download and proceed with selecting the "Instant Download" option (CalSelector is not available with the "FTP or Media Delivery" channel):
4. In the Submit Request page, then, tick the CalSelector box:
Pressing the Submit button submits the request to the archive, which in turn creates the datasets composed of the raw science files you have queried for in the previous steps and the associated calibrations.
5. The datasets as created by the CalSelector tool are presented on the next page, where they can be marked for download. The “Select All” button selects for download all files in the current web page. Requests consisting of more than 1,000 files are split among different pages. In this case a “Download Request” button allows downloading the entire request at once, without having to thumb through the different pages.
The content of each dataset can be expanded or collapsed by clicking on the + or - icon next to its name, respectively. Also next to the dataset name, an icon marks whether the association is complete (green tickmark, xml attribute complete="true": all of the intended files were sucessfully associated), incomplete (orange tickmark, xml attribute complete="false": some of the intended files are missing because, e.g., they have not yet been acquired at the telescope) or empty (yellow tickmark: e.g. the input science files are outside of the tool's applicability period). A yellow triangle icon indicates that the tool failed for some technical reason. Typically, thechnical problems are attended to within two working days. Finally, files are also labelled with their category (please note that the category is not assigned if, for whatever reason, an association could not be established).
Calibration files common to more then one dataset will be displayed in all relevant datasets, but will only be downloaded once.
When a dataset is expanded, an icon to the right of each file’s name indicates whether it is accessible to you (green tickmark) or not (e.g. because the file is still under proprietary period of another user; red no access sign). Please note that calibrations associated to science file you do not have access to will not be downloadable, either. A separate specific query will be needed to access them.
The structure and content of the xml file
As mentioned earlier, an xml representation of the calibration cascade is added to the results of the request. This section contains a brief explanation of the structure and content of this xml file (an example of an actual file is here).
Structure: the xml file contains a series of nested <association> elements, where the nesting level reflects the position and role of files in the calibration cascade. Each <association> element, in turn, contains the following elements: <mainFiles> containing, well, the main files for that level of the calibration cascade, <associatedFiles> where the calibrations for the maiFiles are listed and <messages> for additional information on, e.g. the nature of missing files in the case of an incomplete association. The science files are contained in the top <association> element and the cascades ends with the files that do not need any firther calibrations. The names of the actual files are listed in the <file> elements, one file per element.
As for the content, each <association> element has three attributes: a category, a completness flag with possible values "true" or "false" and a type flag with possible values "main" "or "auxiliary" (this latter higlighting files that are pertinent to the science files, but are not stictly needed for processing like, e.g. acquisition images for spectroscopic observations). In addition, the first (main) <association> element contains the "certified" flag, with possible values "true" or "false", to indicate whether quality-certified calibrations were used to build the calibration cascade provided (typically it takes 2-3 working days to certifiy newly acquired calibrations). The <file> elements, each of which contains one file of the calibration cascade, have two attributes: a category, which is suitable to be provided as input to the ESO pipelines, and the name of the file itself (actually, strictly speaking, that is the root name, without the .fits extension). Please note that the category attribute to the <file> element is, in general, different from the category attribute of the <association> element. The former is a property of the individual file, whereas the latter is a property of that branch of the calibration cascade.