FUNCTIONAL TESTS OF CASA TASKS

V3.1

2016-12-9

Sandra Castro

 

 

This document briefly describes how to use the Python unit tests framework (UT) for CASA tasks. The unit tests use PyUnit[1], which is the standard Python unit tests framework. The UT framework is implemented by importing the module unittest. The tests are executed using nose[2], which is implemented by importing the module nose.

 

 

Quick recipe to create new tests

 

  1. Create a new file in trunk/gcwrap/python/scripts/tests and name it test_taskname.py, where taskname is the name of the task you are going to test. Write the test as shown in the examples throughout this document.
  1. Create a directory taskname for the necessary data in the repository under data/regression/unittest/. If the data are used by other tests, link them to this directory instead of making a copy. Make sure to use small data sets that can be easily copied.
  1. Add the test to the file trunk/gcwrap/python/scripts/tests/CMakeLists.txt so that cmake can copy it to the installation directory.
  1. Build gcwrap.
  1. Check if your test works, by running it like this:

            casa  --nologger  --log2term  -c  <path>runUnitTest.py  test_taskname
  1. When the test is complete and running properly, you may add it to the file trunk/gcwrap/python/scripts/tests/unittests_list.txt so that it is part of the automatic list run by runUnitTest.py. Only add the test to this list if it is working fine.

 

 

 

Running the Python unit tests

 

The unit tests should run outside a CASA session, but executing them inside CASA is also possible, although with limited functionality. The main script to run is runUnitTest.py located in the installation directory,, which usually is <casa_install_dir>/python/<PYVER>/regressions/admin. The UT framework will search for tests located in the same directory.

 

 

CASA unit tests framework usage:

 

casa <casa-options> –c runUnitTest.py <options> test_name

 

Options

 

no option

print this message and exit.

-a or --all

run all the tests defined in the file trunk/gcwrap/python/scripts/tests/unittests_list.txt

<testname>

run the specific test. Additional tests are separated by white spaces.

[method,class]

run the list of methods and/or classes from a specific test.

-f or --file <filename>

run the tests defined in and ASCII file, one test per line.

-d or --datadir <dir>

set an environmental variable called TEST_DATADIR, pointing to a directory where the data is found. Tests can be modified apriori to read this variable to access data from an alternate location. This option should be given together with a test_name or list of tests. See the Example1 in this document.

-m or --mem

run the tests and display the number of open files left by the test and the net amount of memory.

-g or --debug

set casalog.filter to DEBUG.

-l or --list

print the list of tests from trunk/gcwrap/python/scripts/tests/unittests_list.txt

-s or --classes

print the classes from a test script. Only prints the classes returned by the suite() function, defined in the script.

-H or --Help

print this message and exit.

 

 

Examples

 

1) Get the usage information.

 

     casa –c runUnitTest.py --Help

 

2) Give any option to casa before the –c argument.

 

     casa –nogui –log2term –c runUnitTest.py --Help

 

3) Run the unit tests of task clean. It will run all test methods from all classes.

 

     casa –c runUnitTest.py test_clean

 

4) Run the unit tests of task clean and set casalog.filter(ÔDEBUGÕ). It will run all test methods from all classes and show any messages that are sent to DEBUG.

 

     casa –c runUnitTest.py –g test_clean

 

5) List  the classes and methods defined in a test script. It will list only the classes returned by the suite() function defined in the script.

 

     casa –c runUnitTest.py --classes test_clean

 

6) Run only one class. Run only class clean_class2 in test_clean, as shown at the end of this document. Use option –s to see which classes are defined in the test script.

 

     casa –c runUnitTest.py test_clean[clean_class2]

 

7) Run only test2 and test23 in test_clean. The names of specific test methods are given as a list after the test name. There is no space between the test name and Ô[Ô.

 

     casa –c runUnitTest.py test_clean[test2,test23]

 

8) Run test_r_r_1 in test_report, test5 in test_clean and all tests in test_plotants.

 

casa –c runUnitTest.py test_report[test_r_r_1] test_clean[test5]  

         test_plotants

 

9) Run all the available unit tests. This option will run all the tests defined in the file trunk/gcwrap/python/scripts/tests/unittests_list.txt.

 

     casa –c runUnitTest.py --all

 

10) Run the tests defined in a text file. The list should contain one test per line.

 

     casa –c runUnitTest.py --file ListofTests.txt

 

11) List all the tests included in the file unittests_list.txt.

 

     casa –c runUnitTest.py --list

 

12) Set an alternate data directory to a test or list of tests. In this example, test_clean is previously modified to read the TEST_DATADIR environmental variable. See Example1 on how to do this.

 

   casa –c runUnitTest.py –datadir /opt/mydata test_clean

 

13) Run a test inside CASA.  Although possible, running the tests inside CASA is not recommended.

 

CASA <1>: runUnitTest.main(['test_clean'])

CASA <2>: runUnitTest.main(['test_clean[test4,test1a]'])

CASA <3>: runUnitTest.main(['test_clean[clean_class2,test5]'])

 

 

The UT framework will report how many tests had errors or failures at the end of the run. In case of any error or failure, a traceback is printed on the screen.

 

The script runUnitTest.py will create a directory called nosedir under the current directory. If the directory already exists, it is removed. The script will save all the input and output of the tests inside nosedir. It creates an XML file for each run under nosedir/xml. The nosedir directory contains the results of the tests, which can be seen using Jenkins.

 

 

Writing new unit tests

 

Create a class from the unittest.TestCase class. Write variables with the names of the input and output files at the beginning of your class. No directory path should be given, only the filename. Write tests as methods of the main class. Each method should be an independent test. The UT framework identifies methods with reserved names and calls them up before and after each test. Before each test method, a method called setUp will run with instructions to copy the input data to the current directory. The tearDown method will clean up the input and optionally the output files after the execution of each test method. See example 1.

 

The setUp and tearDown methods can be omitted if the test does not need any input data.

 

The name of each method should start with the prefix test so that the UT framework can find it. The execution order of the tests is not guaranteed. This means the tests need to be both independent of each other and of the order in which they appear in the class.

 

Test methods do not get any arguments and do not return anything. Any test method can internally call other methods that get and return arguments. In this case, the word test should not be part of the methodÕs name. See example 2.

 

Outside the class, it is mandatory to have a function called suite that returns a list of the class(es) name(s). It is possible to create as many classes as one wants. A good strategy is to create different classes for different data sets. The UT framework also allows choosing which class to run. This is especially useful when creating new tests.

 

Use the CASA task name to name your unit test as in test_taskname.py. Save new tests in the location trunk/gcwrap/python/scripts/tests, and add them to the CMakeList.txt in order to be executed. If creating new data files for unit tests, place them in the svn repository under $CASAPATH.split()[0]+Õ/data/regression/unittest/task_name.

 

 

Example 1: unit tests of task clean, using two classes and two different data sets.

 

import os

import sys

import shutil

from __main__ import default

from tasks import *

from taskinit import *

import unittest

 

datapath = os.environ.get('CASAPATH').split()[0] +\

                            '/data/regression/exportasdm/input/'

 

# Read the data sets from another directory

if os.environ.has_key('TEST_DATADIR'):  

    DATADIR = str(os.environ.get('TEST_DATADIR'))+Õ/clean/Õ

    if os.path.isdir(DATADIR):

        datapath = DATADIR

    else:

        print 'WARN: directory '+DATADIR+' does not exist'

 

print ÔClean tests will use data from '+datapath        

 

class clean_class1(unittest.TestCase):

   

    # Input and output names

    msfile = 'ngc7538_ut1.ms'

    res = None

    img = 'cleantest_im'

 

    def setUp(self):

        self.res = None

        default(clean)

        if (os.path.exists(self.msfile)):

            shutil.rmtree(self.msfile)

           

        shutil.copytree(datapath+self.msfile, self.msfile)

   

    def tearDown(self):

        if (os.path.exists(self.msfile)):

            shutil.rmtree(self.msfile)

        os.system('rm -rf ' + self.img+'*')

 

    def test0(self):

        '''Test 0: Default values'''

        self.res = clean()

        self.assertFalse(self.res)

 

    def test1(self):

        """Test 1: Wrong input should return False"""

        msfile = 'badfilename'

        self.res = clean(vis=msfile, imagename=self.img)

        self.assertFalse(self.res)

       

    def test2(self):

        """Test 2: Good input should return None"""

        self.res = clean(vis=self.msfile,imagename=self.img)

        self.assertEqual(self.res,None)

       

    def test3(self):

        """Test 3: Check if output exists"""

        self.res = clean(vis=self.msfile,imagename=self.img)

        self.assertTrue(os.path.exists(self.img+'.image'))

 

    def test4(self):

        '''Test 4: Non-default imagermode mode mosaicÕÕÕ

        self.res = clean(vis=self.msfile,imagename=self.img,

                         imagermode=ÕmosaicÕ)

        self.assertEqual(self.res, None)

        self.assertTrue(os.path.exists(self.img+'.image'))

 

    def test5(self):

        """Test 5: Non-default field value"""

        self.res = clean(vis=self.msfile,imagename=self.img,

                         field='3~8')

        self.assertEqual(self.res, None)

        self.assertTrue(os.path.exists(self.img+'.image'))

 

         

class clean_class2(unittest.TestCase):

   

    # Input and output names

    msfile = 'split1scan.ms'

    res = None

    img = 'cleantest2'

 

    def setUp(self):

        self.res = None

        default(clean)

        if (os.path.exists(self.msfile)):

            shutil.rmtree(self.msfile)

           

        datapath = os.environ.get('CASAPATH').split()[0] +

                  '/data/regression/unittest/clean/'

        shutil.copytree(datapath+self.msfile, self.msfile)

   

    def tearDown(self):

        if (os.path.exists(self.msfile)):

            shutil.rmtree(self.msfile)

       

    def test1a(self):

        """Clean 1a: Non-default mode velocity"""

        retValue = {'success': True, 'msgs': "", 'error_msgs': '' }           

        res = clean(vis=self.msfile,imagename=self.img,

                  mode='velocity',restfreq='231901MHz')

        if(res != None):

            retValue['success']=False

            retValue['error_msgs']=retValue['error_msgs']\

                     +"\nError: Failed to run in velocity mode."

        if(not os.path.exists(self.img+'.image')):

            retValue['success']=False

            retValue['error_msgs']=retValue['error_msgs']\

                     +"\nError: Failed to create output image."

                        

        # Verify if there are blank planes at the edges

        vals = imval(self.img+'.image')

        size = len(vals['data'])

        if (vals['data'][0]==0.0 or vals['data'][size-1]==0.0):

            retValue['success']=False

            retValue['error_msgs']=retValue['error_msgs']\

                     +"\nError: There are blank planes in the edges

                  of the image."

                    

 

        self.assertTrue(retValue['success'],retValue['error_msgs'])

 

 

 

def suite():

    return [clean_class1,clean_class2]

 

 

Example 2: tests that do not need data and call up a non-test method.

 

import report

import unittest

 

class version_test(unittest.TestCase):

 

    def shortDescription(self):

        return "Unit tests of comparing version strings"

   

    def test_fails(self):

        assert 2 + 2 == 5

 

    def test_execution_failure(self):

        raise Exception("die")

 

    def test_r_r_1(self):

        "test revision vs revision"

        a = "CASA Version 3.0.1 (r10006)"

        b = "CASA Version 3.0.1 (r9933)"

        self.order(b, a)

 

    def order(self, a, b):

        """Verify that the cmp_version function behaves

        as it should, given that a is earlier than b"""

       

        assert report.cmp_version(a, b) < 0

        assert report.cmp_version(b, a) > 0

        assert report.cmp_version(a, a) == 0

        assert report.cmp_version(b, b) == 0

 

def suite():

    return [version_test]

 

 

Note that there is another method called shortDescription, which also has a reserved method name. If present in the script, it will display the short description string for each test, followed by its results. If not present, the string written between 3-quotes in each test will be displayed instead.

 

 

References:

[1] PyUnit documentation, http://pyunit.sourceforge.net/pyunit.html

[2] Nose, http://somethingaboutorange.com/mrl/projects/nose/0.11.1

[3] Look at the available tests in the repository for examples on how to write them in trunk/gcwrap/python/scripts/tests.