ALMA QA2 Packaging Script
The latest version of the packaging script is available via the JAO CVS repository in directory AIV/science/qa2 or via the EU ARC svn repository
The contents of the package should be consistent with the guidelines described here: http://almasw.hq.eso.org/almasw/bin/view/Archive/Cycle01database
A pdf document with a more detailed description of the packaging script: documentation.pdf
As of 14 August 2014, the packager has new functionality to either stage or package data products from the pipeline.
As of 1 Nov 2015, the packager has new functionality to either stage or package data products from the SB pipeline.
As of 8 Dec 2015, the packager will apply compression to PB and mask images. It will also pick up the new antennapos.csv file in style "cycle3-pipe1".
Running the script
The script should be run from within casa. It can be run from any directory. The import is done with
from QA2_Packaging_module import *
Packaging of standard script-generator-assisted analysis products
For a standard script-generator-assisted analysis, you can continue to run the script in the following way:
QA_Packager(origpath='/pathToMySB', readme='./README.header.txt', packpath='./20XX.1.00XXX.S', PIscript='<yourjaomirror>/science/qa2/scriptForPI.py', style='cycle2-nopipe', mode='fake')
This will just test
the packaging (mode = 'fake').
The parameters mean the following:
- origpath should be the path to the reduction directory for a particular SB. Several subdirectories for EBs could exist within this directory.
- packpath: the path to the destination folder (which should have the project code as name)
- paths should not end in /
- readme: The path to an ascii file with the text of the README header (for a template see this link at ESO or find it in the CVS repository here: AIV/science/qa2/README.header.txt, or use the text at the bottom of this page)
- PIscript added Nov 2013
- style: decides on details of the packaging and whether pipeline products are involved (note: the "noms" parameter still exists but is deprecated)
- mode: The copying mode:
- mode = ’fake’ : This is the default. The script creates empty (i.e., dummy) files at the destination folder.
- mode = ’ticket’ : Use this to create the 'ticket.zip' file to attatch to the SCOPS ticket (see below).
- mode = ’copy’ : The files are copied in the normal way.
- mode = ’hard’ : The script generates hard links in the destination folder. This way, the file-pointers at both the origin and the destination folders, refer to the same physical locations (i.e., inodes) in the disk. This is the recommended way of packaging for noms=False, but if you use noms=True better use mode="copy" is better because it works across file systems.
- append (optional): The appending mode (see below).
- append = ’’ : This is the default. It removes any previous data at the destination folder before the packaging.
- append = ’group’ :The script appends a new group id to the destination folder (so the other groups, if any, are not removed).
- append = ’member’ : The script appends new member id(s) to the group with highest id (so the other groups, if any, and the other member ids, if any, are not removed).
- style: alternative (and recommended) way to specify a number of option at once.
- style = "cycle0-nopipe": corresponds to noms=False
- style = "cycle1-nopipe": corresponds to noms=True
- style = "cycle2-nopipe": corresponds to noms=True
- style = "cycle2-pipe1": corresponds to noms=True and the directive to look for and include pipeline products
- style = "cycle3-nopipe": same as cycle2-nopipe except for the recognition of the scriptForImagingPrep.Py
- style = "cycle3-pipe1": same as cycle2-pipe1 except for the recognition of the scriptForImagingPrep.Py and antennapos.csv files
- default = "" (empty): ignore the parameter "style" and use the noms parameter instead
In order to perform the final packaging, run the script with
QA_Packager(origpath='/pathToMySB',readme='./README.header.txt',packpath='./20XX.1.00XXX.S',mode='hard', style='cycle2-nopipe', PIscript='<yourjaomirror>/science/qa2/scriptForPI.py')
Packaging including Pipeline products
If you have run the ALMA Pipeline and produced output which you want to package for further processing elsewhere, you can use the packager to create a "delivery style" package.
Also, if you have received such a "delivery style" package from elsewhere and have done the imaging part of the QA2 work and now want to package everything for final delivery to the PI,
the packager enables you to do that as well.
Both use cases are described below:
An SB was processed by the pipeline and is to be packaged for imaging at the ARCs or elsewhere
Then run the packager as follows:
QA_Pipeline_Stager(pipeline_root='<top dir of pipeline results>',
staged_root='<project ID == top dir of output package>',
mode='<hard, copy, fake>', # default is 'copy'
PIscript="<path to scriptForPI.py>") # needs to be included in the package
QA_Pipeline_Stager('2013.100456.S-2014-08-08T123456', '2013.100456.S', mode='copy', PIscript="./scriptForPI.py")
A package as created in Case 1 above has arrived at the ARC (or a JAO analyst).
The analyst has run the scriptForPI to create the *.ms.split.cal(s) and verified that the scriptForPI is working in this case.
The analyst proceeded to perform the imaging in the "calibrated" directory which was created by the scriptForPI.
The resulting fits files reside in the "calibrated" directory and the scriptForImaging resides there as well or in the "script" directory.
The result is QA2_PASS and now the whole data is to be packaged for delivery.
Then run the packager as follows:
QA_Packager(origpath='<top dir of the analysis dir>',
packpath='<project id = top dir of output package>',
readme='<path to readme file>',
style='cycle2-pipe1', # this style needs to the chosen in this case
PIscript="./scriptForPI.py") # overwrites the original scriptForPI.py if needed
See section "Packaging of standard script-generator-assisted analysis products" for an explanation of the different "mode" options.
Packaging of TP data
Since version 1.43 of the packager, handling of Total Power data is supported.
The typical call to package manually
calibrated TP data looks like this:
QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2015.1.00501.S', style='cycle2-nopipe', mode='hard')
The packager will detect that the package is a TP dataset from the presence of scripts with names ending in scriptForSDCalibration.py .
It will then override other modes and switch to TP mode. The typical output tree is shown here
Non-standard scripts (i.e. not ending in scriptForSDCalibration.py or not named exactly scriptForSDimaging.py) will also be packaged but a WARNING will be issued about them.
The typical call to stage pipeline-calibrated TP data looks like this:
QA_Pipeline_Stager('2013.1.00911.S_2015_08_07T16_10_05.732', '2013.1.00911.S', mode='copy', PIscript="./scriptForPI.py")
The typical call to package pipeline-calibrated TP data looks like this:
QA_Packager(origpath='SB_uid___A001_X196_X40', readme='./README.header.txt', packpath='./2015.1.00501.S', style='cycle2-pipe1', mode='hard', PIscript="./scriptForPI.py")
Note: the latter expects fits images in the "calibrated" directory.
Packager Unit Test
In the same repository directory where you find the packager module, you also find a "unit test" for it. This enables you to confirm that the basic functionality of the packager is working.
tar xvzf QA2_Packaging_module.unit-test.tgz
cp ../QA2_Packaging_module.py .
start casapy and run
Thanks very much to Ivan Marti-Vidal (Onsala) and Anita Richards (Manchester) for producing the first useful version of the script.
Thanks to Ivan Marti-Vidal (Onsala) for continued support.
- 8 Dec 2015