Common DFOS tools:
Policy for hiding frames
= Data Flow Operations System, the common tool set for DFO
Guidelines for hiding frames
compiled by Sabine Moehler
The dfos tool hideFrame can be used to hide
frames without information or with misleading information. Here are a few rules about when
a file should be hidden.
- Calibration frames without useful information or with misleading
always be hidden, irrespective of whether they are part of the standard
observatory calibration or part of a PI run.
- We do not commit to 100% completeness of all cases. We hide bad frames if
we encounter them, but it is not required to 'hunt' for them.
- For any hidden data a comment should be entered in raw_comments (within
certifyProducts, this is trivial; outside certifyProducts, use the
comments: [file]" on the raw data report).
Examples for data to be hidden are
- files without useful information
- significantly saturated frames (i.e. not just a few pixels)
- flat fields (e.g. twilight flats, lamp flats)
- standard star spectra (telluric/flux)
- standard star images (only if at least all standard stars are saturated)
- not necessarily saturated arc frames - these might be still useful
- frames with corrupted headers that cannot be re-constructed (this also applies to science data)
- files with misleading information
- frames with no flux at all which should have flux (e.g. flats or arcs without lamp)
- a flat that had also the arclamp on, or other parasitic light
On the other hand, a valid flat that has a wrong header keyword (e.g. DPR.TYPE)
should get this value corrected (also by using the hideFrames dialog but asking
DBCM for the correction), and not be hidden.
Examples for grey areas are
- partly saturated frames, which may still be of some use
(e.g. telluric standard saturated in wavelength ranges far from
telluric absorption bands); add comment.
- data marked 'Not Valid', which are followed by a 'Valid' dataset. In
that case the 'Not Valid' data may be hidden (or have DPR.CATG set to
TEST) without further checks, but may also be checked and kept if
good (also partially).
In the following cases files should not be hidden, but comments should
be provided either for the raw data or the products:
- data that cannot be processed by the current pipeline but may
otherwise be useful (i.e. could be processed by an ideal pipeline), e.g.
- incomplete distortion data (SINFONI)
- standard star images with large errors in their coordinates (FORS2)
- over-/underexposed data within a template that produces good results - the bad data are probably there by intention (e.g. saturated data in some linearity tests)
- calibration data providing a valid description of the current status even if they are out of specs, e.g.
- 'bad' zeropoints/standard stars resulting from bad atmospheric conditions
- bias data with persistent electronic noise pattern
If you decide to hide files *after you have certified them*, it is very important to check
that they are not included in any harvested AB, and that they are not available for further
associations. Otherwise a calSelector user will run into trouble since these download requests
are doomed to fail. Therefore, these checks should be made:
- the files in the hide request must not be contained in the current VCAL directory ($DFO_CAL_DIR/VCAL);
- the files in the hide request must not be contained in any harvested SCIENCE AB (in $DFO_LOG_DIR/CALSELECTOR/<date>).
Note that hideFrame v1.3+ takes care of these requirements.
Master calibrations created from these files must be hidden as well.
| Last update: April 26, 2021 by rhanusch