ALMA QA2 Packaging Script

The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository.

The contents of the package should be consistent with the guidelines described here:

A pdf document with a more detailed description of the packaging script: documentation.pdf

As of 14 August 2014, the packager has new functionality to either stage or package data products from the pipeline.

As of 1 Nov 2015, the packager has new functionality to either stage or package data products from the SB pipeline.

As of 8 Dec 2015, the packager will apply compression to PB and mask images. It will also pick up the new antennapos.csv file in style "cycle3-pipe1".

Running the script

The script should be run from within casa. It can be run from any directory. The import is done with

from QA2_Packaging_module import *

Packaging of standard script-generator-assisted analysis products

For a standard script-generator-assisted analysis, you can continue to run the script in the following way:

QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='<yourjaomirror>/science/qa2/', style='cycle2-nopipe', mode='fake')

This will just test the packaging (mode = 'fake').

The parameters mean the following:

  • origpath should be the path to the reduction directory for a particular SB. Several subdirectories for EBs could exist within this directory.
  • packpath: the path to the destination folder (which should have the project code as name)
  • paths should not end in /
  • readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
  • PIscript added Nov 2013
  • style: decides on details of the packaging and whether pipeline products are involved (note: the "noms" parameter still exists but is deprecated)
  • mode: The copying mode:
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’ticket’ : Use this to create the '' file to attatch to the SCOPS ticket (see below).
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging for noms=False, but if you use noms=True better use mode="copy" is better because it works across file systems.
  • append (optional): The appending mode (see below).
    • append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
    • append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
    • append = ’member’ : The script appends new member id(s) to the group with highest id (so the other groups, if any, and the other member ids, if any, are not removed).
  • style: alternative (and recommended) way to specify a number of option at once.
    • style = "cycle0-nopipe": corresponds to noms=False
    • style = "cycle1-nopipe": corresponds to noms=True
    • style = "cycle2-nopipe": corresponds to noms=True
    • style = "cycle2-pipe1": corresponds to noms=True and the directive to look for and include pipeline products
    • style = "cycle3-nopipe": same as cycle2-nopipe except for the recognition of the scriptForImagingPrep.Py
    • style = "cycle3-pipe1": same as cycle2-pipe1 except for the recognition of the scriptForImagingPrep.Py and antennapos.csv files
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead

In order to perform the final packaging, run the script with

QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript='<yourjaomirror>/science/qa2/')

Packaging including Pipeline products

If you have run the ALMA Pipeline and produced output which you want to package for further processing elsewhere, you can use the packager to create a "delivery style" package.

Also, if you have received such a "delivery style" package from elsewhere and have done the imaging part of the QA2 work and now want to package everything for final delivery to the PI, the packager enables you to do that as well.

Both use cases are described below:

Case 1:

An SB was processed by the pipeline and is to be packaged for imaging at the ARCs or elsewhere

Then run the packager as follows:

      QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>',
                                      staged_root='<project ID == top dir of output package>',
                                      mode='<hard, copy, fake>',  # default is 'copy'
                                      PIscript="<path to>") # needs to be included in the package

       QA_Pipeline_Stager('2013.100456.S-2014-08-08T123456', '2013.100456.S', mode='copy', PIscript="./")

Case 2:

A package as created in Case 1 above has arrived at the ARC (or a JAO analyst).

The analyst has run the scriptForPI to create the * and verified that the scriptForPI is working in this case.

The analyst proceeded to perform the imaging in the "calibrated" directory which was created by the scriptForPI.

The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory.

The result is QA2_PASS and now the whole data is to be packaged for delivery.

Then run the packager as follows:

       QA_Packager(origpath='<top dir of the analysis dir>',
                               packpath='<project id = top dir of output package>',
                               readme='<path to readme file>',
                               style='cycle2-pipe1', # this style needs to the chosen in this case
                               PIscript="./")  # overwrites the original if needed


See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options.

Packaging of TP data

Since version 1.43 of the packager, handling of Total Power data is supported.

The typical call to package manually calibrated TP data looks like this:

QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2015.1.00501.S', style='cycle2-nopipe', mode='hard')

The packager will detect that the package is a TP dataset from the presence of scripts with names ending in . It will then override other modes and switch to TP mode. The typical output tree is shown here.

Non-standard scripts (i.e. not ending in or not named exactly will also be packaged but a WARNING will be issued about them.

The typical call to stage pipeline-calibrated TP data looks like this:

QA_Pipeline_Stager('2013.1.00911.S_2015_08_07T16_10_05.732', '2013.1.00911.S', mode='copy', PIscript="./")

The typical call to package pipeline-calibrated TP data looks like this:

QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2015.1.00501.S', style='cycle2-pipe1', mode='hard', PIscript="./")
Note: the latter expects fits images in the "calibrated" directory.

Packager Unit Test

In the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working.


     tar xvzf QA2_Packaging_module.unit-test.tgz
     cd QA2_Packaging_module.unit-test
     cp ../ .
     start casapy and run


Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal (Onsala) for continued support.

-- DirkPetry - 8 Dec 2015

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf documentation.pdf r5 r4 r3 r2 r1 manage 73.2 K 2014-09-12 - 14:07 DirkPetry September 2014 version of the packager documentation
Edit | Attach | Watch | Print version | History: r30 < r29 < r28 < r27 < r26 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r30 - 2015-12-08 - DirkPetry
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2016 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding ARC TWiki? Send feedback