1.3 Partition and Multi-MS support

The partition task is the main task to create a Multi-MS. It takes an input Measurement Set and creates an output Multi-MS based on the data selection parameters. Partition accepts three axis to do separation across: auto, scan or spw. The default auto will first separate the MS in spws, then in scans. Each partitioned MS is referred to as a sub-MS. The user may force the number of sub-MSs in the output MMS by setting the parameter numsubms. By default, the parameter createmms is set to True to create an output MMS.

If set to False, the task will work as the split task and create a normal MS based on the input data selection. Task partition uses two helper classes to handle the parallel jobs; ParallelTaskHelper and ParallelDataHelper. Special care needs to be taken in order to consolidate the sub-tables of the MMS because the spectral window indices in the output are re-indexed in each engine to the same initial index and this needs to be consolidated later. The sub-tables are merged after all engines return for post-processing.

1.3.1 Input Multi-MS Processing

There are essentially two ways of processing an input MMS, these are in parallel using a cluster or as a monolithic MS. Basically every task in CASA can process the MMS as a monolithic MS, because the MMS is made to look like a normal MS. This processing will happen automatically in a transparent way to the user. Examples of such tasks are: listobs, gencal, gaincal, etc.

Other tasks were modified to process the input MMS in parallel such as to speed up the processing, or because they need to modify the input MMS or create a new output MMS. Tasks that only modify the input MMS such as flagdata and applycal may raise NULL MS Selection exceptions depending on the way the MMS was created and the data selection given in the parameters. These exceptions are harmless in these cases and are hidden from the user€™s terminal or logger. Flagdata€™s summary mode does not modify the input, but creates output dictionaries in each parallel engine. These dictionaries are consolidated into one single output dictionary, which is returned to the user.

Tasks that create a new output such as split2, cvel2, hanningsmooth2 and mstransform will process each input sub-MS in parallel whenever possible. In these cases, the output is a Multi- MS with the same separation axis as the input. In some cases, the heuristics are more complicated and it is not possible to process the MMS in parallel or to create an output MMS. These cases are discussed in the following section. The only tasks that can create a Multi-MS from a normal MS are partition and mstransform. These two tasks have a parameter called createmms that controls how to partition the MS. The simple relation between input and output for all tasks is the following:

               input MS  --> output MS
               input MMS --> output MMS
               input MS  --> output MMS (only partition and mstransform)

1.3.2 MSTransform Heuristics

Task mstransform will process an input MMS in parallel whenever possible. Each sub-MS of the MMS will be processed in a separate engine and the results will be post-processed at the end to create an output MMS. The output MMS will have the same separationaxis of the input MMS, which will be written to the table.info file inside the MMS directory.

Naturally, some transformations available in mstransform require more care when the user first partition the MS. If one wants to do a combination of spws by setting the parameter combinespws = True in mstransform, the input MMS needs to contain all the selected spws in each of the sub-MSs or the processing will fail. For this, one may set the initial separationaxis to scan or use the default auto with a proper numsubms set so that each sub- MS in the MMS is self-contained with all the necessary spws for the combination.

The task will check if the sub-MSs contain all the selected spws when combinespws=True and if not, it will issue a warning and process the input MMS as a monolithic MS. In this case, the separation axis of the output MMS will be set to scan, regardless of what the input axis was. This possibility is not available in cvel2.

A similar case happens when the separation axis of the input MMS is per scan and the user asks to do time averaging with time spanning across scans. If the individual sub-MSs are not self-contained of the necessary scans and the duration of the scans is shorter than the given timebin, the spanning will not be possible. In this case, the task will process the input MMS as a monolithic MS and will set the axis of the output MMS to spw. This possibility is not available in split2.

It is important that the user sets the separation axis correctly when first partitioning the MS. See the table below for when it is possible to process the input MMS in parallel or not, using mstransform.

input MMS axis   combinespws=True   nspw > 1   timeaverage=True, timespan='scan'
 -------------------------------------------------------------------------------
 scan                  YES            YES             NO        
 spw                   NO             NO              YES     
 auto                  MAYBE          MAYBE           MAYBE

In the event that the user requests two transformations at the same time: combination of spectral windows and time averaging across scans on an input MMS, similar checks will be applied in order to determine if it is possible to process the input in parallel. First, the task will check if each sub-MS contains the selected spws and only in case of success, it will check if it contains all the scans with proper duration. If the checks are unsuccessful, the input MMS will be processed as a monolithic MS and the output will be in this case a normal monolithic MS.

You can use the task listpartition to verify the contents of an MMS. Listpartition is similar to listobs and can also save the output to a file or return it as a Python dictionary.