Early Science Packaging Script

The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository.

The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database

A pdf document with a more detailed description of the packaging script: documentation.pdf

As of 14 August 2014, the packager has new functionality to either stage or package data products from the pipeline.

Running the script

The script should be run from within casa. It can be run from any directory. The import is done with

from QA2_Packaging_module import *

Packaging of standard script-generator-assisted analysis products

For a standard script-generator-assisted analysis, you can continue to run the script in the following way:

QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='<yourjaomirror>/science/qa2/scriptForPI.py', style='cycle2-nopipe', mode='fake')

This will just test the packaging (mode = 'fake').

The parameters mean the following:

  • origpath should be the path to the reduction directory for a particular SB. Several subdirectories for EBs could exist within this directory.
  • packpath: the path to the destination folder (which should have the project code as name)
  • paths should not end in /
  • readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
  • PIscript added Nov 2013
  • style: decides on details of the packaging and whether pipeline products are involved (note: the "noms" parameter still exists but is deprecated)
  • mode: The copying mode:
    • mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
    • mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
    • mode = ’copy’ : The files are copied in the normal way.
    • mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging for noms=False, but if you use noms=True better use mode="copy" is better because it works across file systems.
  • append (optional): The appending mode (see below).
    • append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
    • append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
    • append = ’member’ : The script appends new member id(s) to the group with highest id (so the other groups, if any, and the other member ids, if any, are not removed).
  • style: alternative (and recommended) way to specify a number of option at once.
    • style = "cycle0-nopipe": corresponds to noms=False
    • style = "cycle1-nopipe": corresponds to noms=True
    • style = "cycle2-nopipe": corresponds to noms=True
    • style = "cycle2-pipe1": corresponds to noms=True and the directive to look for and include pipeline products
    • default = "" (empty): ignore the parameter "style" and use the noms parameter instead

In order to perform the final packaging, run the script with

QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript='<yourjaomirror>/science/qa2/scriptForPI.py')

Packaging including Pipeline products

If you have run the ALMA Pipeline and produced output which you want to package for further processing elsewhere, you can use the packager to create a "delivery style" package.

Also, if you have received such a "delivery style" package from elsewhere and have done the imaging part of the QA2 work and now want to package everything for final delivery to the PI, the packager enables you to do that as well.

Both use cases are described below:

Case 1:
An SB was processed by the pipeline and is to be packaged for imaging at the ARCs or elsewhere

Then run the packager as follows:

      QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>',
                                      staged_root='<project ID == top dir of output package>',
                                      mode='<hard, copy, fake>',  # default is 'copy'
                                      PIscript="<path to scriptForPI.py>") # needs to be included in the package

    Example:
       QA_Pipeline_Stager('2013.100456.S-2014-08-08T123456', '2013.100456.S', mode='copy', PIscript="./scriptForPI.py")

Case 2:
A package as created in Case 1 above has arrived at the ARC (or a JAO analyst).

The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case.

The analyst proceeded to perform the imaging in the "calibrated" directory which was created by the scriptForPI.

The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory.

The result is QA2_PASS and now the whole data is to be packaged for delivery.

Then run the packager as follows:

       QA_Packager(origpath='<top dir of the analysis dir>',
                               packpath='<project id = top dir of output package>',
                               readme='<path to readme file>',
                               mode='<hard,copy,fake,ticket>',
                               gzip_caltables=True,
                               style='cycle2-pipe1', # this style needs to the chosen in this case
                               PIscript="./scriptForPI.py")  # overwrites the original scriptForPI.py if needed

   Example:
       QA_Packager(origpath='./2013.100456.S-analysis',
                               packpath='./2013.100456.S',
                               readme='myREADME.txt',
                               mode='hard',
                               gzip_caltables=True,
                               style='cycle2-pipe1',
                               PIscript="./scriptForPI.py")

Packager Unit Test

In the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working.

Usage:

     tar xvzf QA2_Packaging_module.unit-test.tgz
     cd QA2_Packaging_module.unit-test
     cp ../QA2_Packaging_module.py .
     start casapy and run
         execfile('QA2_Packaging_module.unit-test.py')

Acknowledgements
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script. Thanks to Ivan Marti-Vidal and Adam Ginsburg (ESO) for continued support.


-- DirkPetry - 9 Sept 2014

Topic attachments
I Attachment Action Size Date Who Comment
pdfpdf documentation.pdf manage 73.2 K 12 Sep 2014 - 14:07 DirkPetry September 2014 version of the packager documentation
Topic revision: r24 - 12 Sep 2014 - 14:07:14 - DirkPetry
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding ARC TWiki? Send feedback