Include detmon as a generic pipeline

home dfos administration documentation procedures concepts external: QC

  • 2010-12-01: online
  • 2010-12-03: migration put on hold, waiting for new processAB, comments by BW and SM included
  • 2011-02-22: migration continued after upgrade of processAB to v1.10
  • 2011-11-07: step on config.processAB added

For reasons of better and more efficient maintenance of the pipeline recipes, the SDD pipeline group introduces generic pipeline recipes. Generic pipelines collect data processing tasks like detector monitoring which are not specific to an instrument. SDD offers now both lingain recipes, the optical CCD and the NIR array detector version in one generic pipeline. In the future, SDD plans to remove the detmon recipes from the instrument pipelines and will only maintain the generic pipeline version.

This page describes the procedure how to migrate the operational detmon pipeline recipe as part of the instrument pipeline (IP) to the generic detmon pipeline recipe (GP). This page does not describe, how to put the detmon recipe into operations for the first time. The detmon recipe is currently implemented in several instrument pipelines. Either the version for optical CCDs or the version for NIR arrays is included. Both recipes are also available in the new generic detmon pipeline. In the long term it is planned to use the recipe from the generic pipeline instead of the recipe implemented in the instrument specific pipeline. The migration procedure is explained in the detmon pipeline user manual vers 1.0.1, 2010-11-02 (see bottom of that page: detmon pipeline manual ) and is repeated on this page. Additional information is given on the esorex web page. The migration has to be done only once. The procedure does not has to be repeated for generic pipeline upgrades.

IP stands for instrument pipeline (from where the migration starts)
GP stands for generic pipeline (to where the migration follows)

step by step

  1. In case you operate the current IP detmon recipe with a recipe config file in .esorex, (e.g. /home/crires/.esorex/crires_detmon.rc) backup this file. Otherwise this step can be omitted.

  2. To adapt the GP detmon recipe, to the instrument specific needs, the default GP recipe command line parameters have to be overwritten by the instrument/detector specified parameters. Copy the recipe config file into the directory ${HOME}/.esorex of your operational processing machine.

  3. The recipe config values in detmon_...rc files are first guess values provided by SDD and might differ from the operational values.
  4. It is recommented to include .esorex directory in your backup scheme.

  5. Only in case your ${DFO_CONFIG_DIR}/OCA/${DFO_INSTRUMENT}_association.h file contains recipe parameters, check that they are part of the ${HOME}/.esorex/detmon_{opt|ir}_lg_{instrument}.rc.
  6. Type ls /home/quality/pipelines to verify that detmon-1.0.7 pipeline is installed and that there is a soft link from detmon to detmon-1.0.7

  7. Add the detmon pipeline to the instrument specific pipeline:
    in .esorex/esorex.rc add :/home/quality/pipelines/detmon to esorex.caller.recipe-dir={instrument pipeline path}.
    E.g. for VIRCAM the line looks like:

  8. Verify the last step by typing esorex --recipes
    In addition to the recipes of the instrument pipepline, there should occur the following two lines,
               detmon_ir_lg          : Linearity/Gain recipe for the IR domain
               detmon_opt_lg         : Linearity/Gain recipe for the optical domain
    that indicate that both versions of the detmon recipe are available.

  9. You can test the recipe offline by calling

    esorex --recipe-config=${HOME}/.esorex/detmon_{opt|ir}_lg_{instument}.rc detmon_opt_lg SOF

    where SOF can be the set of files extracted from an operational linearity AB. Note that the DO_CLASS values (the raw type tags, which appear after the raw file name in the SOF, which are specified in {instrument}_classification.h are ON_RAW and OFF_RAW and might differ from tags used by the IP version. Note also that the PRO.CATG values of the GP products might differ from the IP version. The --recipe-config=${HOME}/.esorex/detmon_{opt|ir}_lg_{instument}.rc tells esorex to read the instrument specific parameters for recipe detmon_opt_lg (or detmon_ir_lg, whatever version is applicable) from the specified file under ${HOME}/.esorex.

  10. Send your final detmon_{opt|ir}_lg_{instument}.rc to the detmon pipeline developer to reasons of maintenance and to include it in the GP used on Paranal.
  11. Make the generic recipe call operational:
    Change the recipe name in $DFO_CONFIG_DIR/OCA/${DFO_INSTRUMENT}_association.h. The result could look like:
      recipe detmon_ir_lg ;
  12. edit config.processAB accordingly for the new recipe name and the new PRO.CATG (ed. 2011-11-07)
Generic pipeline detmon recipe