Introduction

Testing is part of the XPACC development process. It is very important that we have a consistent, integrated testing in our codes. General guidelines:

We use CMake/CTest/CDash for our automated build and test system (ABATE)
Project unit testing should be centralized/encapsulated in a few constructs
All applications should return 0 to indicate no errors, or passed test, and non-zero otherwise.
Project tests should be integrated through CMake so that the user can use "make test" to run them

Testing with IX

IX offers facilities designed to assist with code and program testing. IX's testing framework supports serial, and parallel tests, platform specific tests, and complex behavior tests. IX's testing facilities come in the form of coding constructs and utility programs. Both are described below.

IX Testing Constructs

IX offers the following code constructs for implementing testing in project codes:

ix::util::TestResults
ix::util::TestingObject

In general, project-specific testing objects can inherit from the ix::util::TestingObject and implement their own tests as member functions of this derived class. The member methods take an object of type ix::util::TestResults (or one that inherits from this class), and populate it with testing results. The results object can be streamed - and so typically these results can be streamed to the screen, to a file, or string for further processing.

The IX testing code constructs are defined in IX/include/Testing.H and an example of their use can be found in the ix::util::TestObject class. This object implements all the tests for the ix::Util namespace, and is driven by the ix::util::UtilTest function. In other words, the ix::util::UtilTest function implements a command line interface for the ix::util::TestObject, which implements all the existing tests for the ix::Util namespace.

More extensive use of the IX code constructs for testing can be found in the example testing objects for PlasCom2, which are implemented in PlasCom2::TestingObject and PlasCom2::ParallelTestingObject. These testing objects use simple test fixture functions from the PlasCom2::TestFixture namespace and are driven by PlasCom2::Test and PlasCom2::ParallelTest, respectively.

Note: A major advantage of encapsulating all tests in a single object is that external entities need only to instantiate your project-specific testing object to get access to the project's tests. This greatly reduces the complexity of rolling integrated software products out of code from multiple projects.

IX Testing Support Utilities

IX also offers a couple of utilities to support the running of tests. These utilities and their documentation are:

runtest (ix::RunTest)
testresults (ix::TestResults)

Running complex tests with runtest.

The runtest utility is designed to be called from the project's CMakeLists.txt cmake configuration file. Its purpose is to run scripted tests where the complexity or platform-dependent nature of the test being run precludes its being run as a simple test. The runtest utility can run a single named executable, a list of test from file, or resolve platform-specific tests.

Examples of how to use the runtest utility can be found in PlasCom2/Testing/CMakeLists.txt, where it is used to run the parallel tests (which must use platform-specific parallel job spawning mechanisms), and other platform-specific, or complex behavior tests.

Checking test results with testresults.

The testresults utility is designed to extract a particular test result from a test results file with one test result per line. If the test's results are such that it has passed, then testresults returns with a zero exit code, and exits with a non-zero error code otherwise.

Examples of how to use the testresults utility can be found in PlasCom2/Testing/CMakeLists.txt, where it is used to extract the results of all the tests.

Putting it all together

All together, the IX testing facilities provide an end-to-end framework for running, collecting, and reporting your project's tests and results to CMake in such a way that CMake's integrated testing facilty, CTest, can be leveraged to integrate the tests into the project's build system, automate the tests, and report the test results to a testing dashboard. The testing dashboard is a web-based facility which collects and reports test results the test histories. XPACC uses CDash for its testing dashboard.

The birds-eye view of the process for using IX's testing facilities with CMake/CTest goes like this:

Create a project-native testing object (PNTO) by inheriting from ix::util::TestingObject.
Implement unit tests as member functions of the PNTO
Write a driver (or a set of drivers) that instantiates your testing object(s), and drives them to produce a ix::util::TestResults object with the restults from the testing.
INCLUDE(CTEST) in your CMakeLists.txt file.
Invoke the test driver(s) with CMake's ADD_TEST() construct and store the results in a composite testing results file. If necessary use runtest to invoke the actual test(s).
If necessary, use testresults to extract the results of the tests from the composite testing results file and exit with a 0 return code for tests that pass, and a non-zero otherwise. This step is also accomplished with CMake's ADD_TEST().
Configure your project (i.e. run cmake), and then run the integrated tests with "make test".

Note: The reason it may or may not be necessary to use runtest and testresults in steps (5) and (6) is that your tests may be directly invoked by CMake's ADD_TEST if the test is a standalone executable that returns 0 if it succeeds, and non-zero otherwise.

Testing in the XPACC Project Template

Direct examples of using IX Testing for several different kinds of tests are provided in the XPACC Project Template. The following sections summarize the main gist of each of these examples.

Serial Test Examples

The XPACC Project Template has both a standalone serial program example, and several serial unit tests. The unit testing is described below, and discussion of the standalone test is deferred to a later section.

For the serial unit tests, the PNTO is called PlasCom2::TestingObject. This object inherits from ix::util::TestingObject as described in Step (1) above. The unit tests are in the PlasCom2::TestingObject's member methods as prescribed in Step (2). The simple tests and the code construct that they test are:

PlasCom2::TestingObject::Test__ExampleFunction tests PlasCom2::ExampleFunction
PlasCom2::TestingObject::Test__TrapezoidQuadrature tests PlasCom2::TrapezoidQuadrature
PlasCom2::TestingObject::Test__MidPointQuadrature tests PlasCom2::MidPointQuadrature

The plascom2_test command-line driver executable, implemented by the PlasCom2::Test function drives the PlasCom2::TestingObject by instantiating it, and calling the PlasCom2::TestingObject::RunTest (if an explicit test name or list was given) or the PlasCom2::TestingObject::Process method to run all tests. This is Step(3).

Step (4) is trivial, and Step (5) is done with the following line from PlasCom2/CMakeLists.txt:

ADD_TEST(RunPlasCom2Tests ${EXECUTABLE_OUTPUT_PATH}/plascom2_test -o plascom2_testresults.txt)

This runs all of the PlasCom2 tests implemented by the PlasCom2::TestingObject, and stores the results in the file plascom2_testresults.txt.

For Step (6), the testresults utility is used to extract the results of each of the tests from plascom2_testresults.txt with the following lines from PlasCom2/CMakeLists.txt:

ADD_TEST(ExampleProgram:Works ${EXECUTABLE_OUTPUT_PATH}/testresults ExampleProgram:Works plascom2_testresults.txt)
ADD_TEST(ExampleFunction:Works ${EXECUTABLE_OUTPUT_PATH}/testresults ExampleFunction:Works plascom2_testresults.txt)
ADD_TEST(TrapezoidQuadrature:Runs ${EXECUTABLE_OUTPUT_PATH}/testresults TrapezoidQuadrature:Runs plascom2_testresults.txt)
ADD_TEST(TrapezoidQuadrature:Accurate ${EXECUTABLE_OUTPUT_PATH}/testresults TrapezoidQuadrature:Accurate plascom2_testresults.txt)
ADD_TEST(TrapezoidQuadrature:Order ${EXECUTABLE_OUTPUT_PATH}/testresults TrapezoidQuadrature:Order2 plascom2_testresults.txt)
ADD_TEST(MidPointQuadrature:Runs ${EXECUTABLE_OUTPUT_PATH}/testresults MidPointQuadrature:Runs plascom2_testresults.txt)
ADD_TEST(MidPointQuadrature:Accurate ${EXECUTABLE_OUTPUT_PATH}/testresults MidPointQuadrature:Accurate plascom2_testresults.txt)
ADD_TEST(MidPointQuadrature:Order ${EXECUTABLE_OUTPUT_PATH}/testresults MidPointQuadrature:Order2 plascom2_testresults.txt)

In Step (7), users configure PlasCom2 and invoke "make test" to run the tests and report the results to stdout.

Parallel Test Examples

The XPACC Project Template has both a standalone parallel application, pepi, and parallel unit tests. The pepi program computes $\pi$ in parallel by doing parallel quadrature, and the unit tests test the parallel quadrature facility. Both will be discussed in this section.

In this test, the PNTO is called PlasCom2::ParallelTestingObject. This object inherits from ix::util::TestingObject as described in Step (1) above. The unit tests are in the PlasCom2::ParallelTestingObject's member methods as prescribed in Step (2). The simple tests and the code construct that they test are:

PlasCom2::ParallelTestingObject::Test__ParallelTrapezoidQuadrature tests PlasCom2::TrapezoidQuadrature in parallel
PlasCom2::ParallelTestingObject::Test__ParallelMidPointQuadrature tests PlasCom2::MidPointQuadrature in parallel

The plascom2_parallel_test command-line driver executable, implemented by the PlasCom2::ParallelTest function drives the PlasCom2::ParallelTestingObject by instantiating it, and calling the PlasCom2::ParallelTestingObject::RunTest (if an explicit test name or list was given) or the PlasCom2::ParallelTestingObject::Process method to run all tests. This is Step(3).

Step (4) is trivial, and Step (5) for this example is more complicated than that of the serial case. Since this is a parallel test, it must be spawned in parallel using something like mpiexec or mpirun. The parallel application spawning mechanism is platform-dependent, and even may need to be done through a batch queueing system.

Due to the platform-specific nature of executing parallel applications, Step (5) must be accomplished using the runtest utility. This is done in the following line from PlasCom2/CMakeLists.txt:

ADD_TEST(RunParallelPlatformTests ${EXECUTABLE_OUTPUT_PATH}/runtest -p ${PROJECT_SOURCE_DIR}/share/Platforms/parallel_platforms -o plascom2_testresults.txt)

This line gives the PlasCom2/share/Platforms/parallel_platforms file as the platform argument to runtest. The parallel_platforms file is line-based and has the following format on each line:

<hostname> <path to platform-specific test list>

If not given on the command line, the runtest utility will determine the hostname and resolve the list of tests from this file. The platform-specific test list should list as many parallel testing scripts as one needs to do on the given platform. For example, see PlasCom2/share/Platforms/parallel_platforms, and PlasCom2/share/Platforms/xpacc_cluster_parallel.list. You will see that the list includes two scripts that invoke the parallel tests:

xpacc_cluster_parallel_test1.csh (runs the parallel unit test driver)
xpacc_cluster_parallel_test2.csh (runs pepi)

On XPACC's cluster (xpacc_cluster), these tests must be conducted through the batch system. These testing scripts submit the tests to the batch system and report the results. These results are then fed into the plascom2_testsresults.txt file.

Essentially, these scripts submit the jobs to the cluster's queue, and then wait on the results before returning to the calling utility, runtest. For further details on how they do this, see PlasCom2/share/Platforms/xpacc_cluster_parallel_test1.csh and PlasCom2/share/Platforms/xpacc_cluster_parallel_test2.csh.

Once the runtest utility has returned, then the plascom2_testresults.txt file has been updated with the results from the parallel tests, and, just like for the serial test Step (6), the testresults utility is used to extract the results of each of the tests from plascom2_testresults.txt with the following lines from PlasCom2/CMakeLists.txt:

ADD_TEST(ParallelExample:Runs ${EXECUTABLE_OUTPUT_PATH}/testresults PEPI:Runs plascom2_testresults.txt)
ADD_TEST(ParallelExample:Works ${EXECUTABLE_OUTPUT_PATH}/testresults PEPI:Works plascom2_testresults.txt)
ADD_TEST(ParallelTrapezoidQuadrature:Runs ${EXECUTABLE_OUTPUT_PATH}/testresults ParallelTrapezoidQuadrature:Runs plascom2_testresults.txt)
ADD_TEST(ParallelTrapezoidQuadrature:Accurate ${EXECUTABLE_OUTPUT_PATH}/testresults ParallelTrapezoidQuadrature:Accurate plascom2_testresults.txt)
ADD_TEST(ParallelTrapezoidQuadrature:Order ${EXECUTABLE_OUTPUT_PATH}/testresults ParallelTrapezoidQuadrature:Order2 plascom2_testresults.txt)
ADD_TEST(ParallelTrapezoidQuadrature:WeakScaling ${EXECUTABLE_OUTPUT_PATH}/testresults ParallelTrapezoidQuadrature:WeakScaling plascom2_testresults.txt)
ADD_TEST(ParallelTrapezoidQuadrature:StrongScaling ${EXECUTABLE_OUTPUT_PATH}/testresults ParallelTrapezoidQuadrature:StrongScaling plascom2_testresults.txt)
ADD_TEST(ParallelMidPointQuadrature:Runs ${EXECUTABLE_OUTPUT_PATH}/testresults ParallelMidPointQuadrature:Runs plascom2_testresults.txt)
ADD_TEST(ParallelMidPointQuadrature:Accurate ${EXECUTABLE_OUTPUT_PATH}/testresults ParallelMidPointQuadrature:Accurate plascom2_testresults.txt)
ADD_TEST(ParallelMidPointQuadrature:Order ${EXECUTABLE_OUTPUT_PATH}/testresults ParallelMidPointQuadrature:Order2 plascom2_testresults.txt)
ADD_TEST(ParallelMidPointQuadrature:WeakScaling ${EXECUTABLE_OUTPUT_PATH}/testresults ParallelMidPointQuadrature:WeakScaling plascom2_testresults.txt)
ADD_TEST(ParallelMidPointQuadrature:StrongScaling ${EXECUTABLE_OUTPUT_PATH}/testresults ParallelMidPointQuadrature:StrongScaling plascom2_testresults.txt)

In Step (7), users configure PlasCom2 and invoke "make test" to run the tests and report the results to stdout.

Direct Test Example

The XPACC Project Template has one serial example program called sep. The sep program simply copies a file. It is a useful example because it can be directly invoked by CTest since sep returns 0 if it succeeds and 1 if not (e.g. the input file did not exist or something).

The following line from the PlasCom2/CMakeLists.txt file invokes sep and evaluates whether it succeeded or failed (based solely on its return code):

ADD_TEST(ExampleProgram:Runs ${EXECUTABLE_OUTPUT_PATH}/sep CMakeCache.txt)

Note: There is no output file, just a name for the test, and then the command it should run.

In order to evaluate whether sep actually did what it was told to do, we need a more complicated facility. In fact, this is done with runtest just like above. The following line from PlasCom2/CMakeLists.txt runs a (number of) script(s) from a list. One of these scripts, namely PlasCom2/share/Testing/test_scripts/serial_tests.csh, actually runs and checks to make sure it copies a file correctly:

ADD_TEST(RunTests ${EXECUTABLE_OUTPUT_PATH}/runtest -l ${PROJECT_SOURCE_DIR}/share/Testing/test_scripts/tests.list -o plascom2_testresults.txt)

Reusing the Examples

It is highly recommended to simply reuse the testing examples provided in the XPACC Project Template when creating your own tests that use this framework.

To reuse the example testing objects, the developer could just remove the existing test functions in the serial testing object PlasCom2::TestingObject and the parallel testing object, PlasCom2::ParallelTestingObject and then implement her own unit tests as member functions of those objects.

The corresponding constructs from the CMakeLists.txt file would need to be removed/added as well - and also the scripts invoking the tests, if necessary. The rest of the framework, including the test object drivers would still be valid and should continue to work without modification.

In order to create stand alone tests that utilize scripts users can copy the scripts located in testing/share/Testing/test_scripts and edit them for their needs. Additionally, they will need to follow the examples shown in testing/CMakeLists.txt for calling the standalone tests and add a call for their test. If users are creating a regression test or a "gold standard" test in which they wish to compare saved data to newly generated data a script and example command have been created to help. More information on this regression script is below.

Creating a "gold standard" test

A script is provided to help users in creating a "gold standard" test. The idea behind a "gold standard" test is to have saved output from a previous run of the software, where the solution data or output is known to be accurate. The test will then run the newly compiled version of the software and compare the generated output against the saved data. Located in /testing/share/Testing/test_scripts is a script titled regression.csh. This script is set up to run a "gold standard" test after a few edits from the user. The places in the file that require editing are marked in the script and are explained below.

1) InputDir=_____ This entry should have the name of the input data directory, which should be created by the user and placed in testing/share/Testing/test_data. This directory should house all the necessary input data for running the user's executable. The regression script will copy this directory, navigate into it, and then execute the given command.

2) Outputs=_____ This entry should contain the names of the generated output files that the user wishes to compare with saved data.

3) OutputsCheck=_____ This entry should contain the names of the saved output data files to compare the new output files against. Note that the files are compared using the diff command. Also, the files in Outputs must have a one to one corresponence with the files in OutputsCheck.

4) TestName=_____ This entry should contain the name the user wishes to use for the test.

5) The command for running the user's executable should be entered at the appropriate place in the script (the loaction is indicated with a comment). The user can also add any other features to the script that may be specific to a test.

In order to run the test and check the results two lines need to be uncommented and one of them edited in testing/CMakeLists.txt. These two lines are present in the testing section and are indicated by RegressionTest as the test name. The first of these must be uncommented. This line calls the runtest executable which in turn calls the regression.csh script. The second instance with RegresionTest calls the testresult executable and verifies the output of the regression.csh script. This second line must be edited to have the name of the user's test used in 4) above. These names must match exactly or the test will indicate failure even if that is not the case. The location to place the test name is indicated in the file. The regression test should then be ready to run with the other tests.

Note that the regression.csh script utilizes an executable called diffdatafiles which is part of IX. This executable works like the Unix diff command but will also compare numbers within a given tolerance. Additionally it can be directed to ignore strings and only compare numbers. Using this command users can compare their numerical output to ensure that the answers are within a certain tolerance and ignore other aspects of a data file that might be unimportant like a time and date stamp. The default written into the script is to compare all output files using only the numbers and comparing within a tolerance of 1.0e-10. Therefore, diffdatafiles will read in each string from the two data files one at a time. If the strings are in fact numbers it will ensure that the two numbers from each file are within 1.0e-10 of one another (strings will not be compared). The usage for the diffdatafiles command is shown below so that users may change its arguments and runtime behavior if desired.

    diffdatafiles [-hnb] [-v [level] -o <filename> -t [tolerance] ] <file1> <file2> 

    -h,--help
            Print out long version of help and exit.

    -v,--verblevel [level]
            Set the verbosity level. (default = 1)

    -o,--output <filename>
            Set the output file to <filename>. (default = stdout)

    -t,--tolerance [tolerance]
            Set the numerical tolerance for comparing numbers to <tolerance>.
            (The default for the tolerance is 1.0e-12.)
            (The default behavior without -t is to compare numbers as strings.)
            (This flag will automatically force the -b flag to be used.)

    -n,--numbers
            Only compare the numbers in the two files.
            (This flag will automatically force the -t flag to be used.)

    -b,--blank
            Ignore blank space between words (or numbers).

    <file1>
            First file to read in for comparison against file2.

    <file2>
            Second file to read in for comparison against file1.

Automated Testing

The XPACC Project Template has a couple of utilities designed to assist in understanding and setting up automated testing through CTest. A sort of "quickstart" set of steps for setting up automated testing is as follows:

Log in to XPACC's CDash instance and create a new project for your project (if it does not already exist).
Make a directory from which to run your automated builds and tests (e.g. ~/AutomatedTesting).
Copy PlasCom2/share/Testing/test_scripts/ctest/{automated_test_script.cmake,run_automated_tests,projects,modules,documentation} into your testing directory.
Edit the projects file to remove the examples and add the projects that you want to test.
Modify the environment module file in modules directory for your projects to reflect the desired build environment.
Modify the documentation file for your project if you want automatic documentation builds
Edit the run_automated_tests script with your customizations.
Test the setup by executing:

./run_automated_tests ./projects Experimental ~/AutomatedTesting
If everything works OK, then add a cron job to invoke run_automated_tests at your desired intervals and modes.

For Step(1), log into CDash and follow the steps to create a new project. Add yourself as an author, and anyone else that should know about the status of the automated builds/tests.

Steps(2) and (3) are obvious.

In Step(4), it should be noted that the projects file is processed line-by-line. Each line should indicate the parameters for a single build and test. The expected format for each functional line of projects file is as follows:

<Project Name>|<Branch Name>|<Branch Path>|<Repository Type>|<Documentation Target>

Based on the line from the projects file, the testing utilities will automatically try to check out the following branch from either GIT or SVN with the following command:

svn:

svn co <Branch Path> <Project Name>_<Branch Name>

git:

git clone <Branch Path> <Project Name>_<Branch Name>

If the <Project Name>_<Branch Name> directory already exists, then CTest will simply update from SVN if there are changes in the repository. On fresh check-outs or updates, CTest will (re)configure and (re)build the project and run the tests.

In Step(5), the script automatically loads (and subsequently unloads) an environment module file with the project name (if it exists). The module files should setup up the required build environment and reduce the need for hand editing of the run_automated_tests script.

In Step(6), the testing framework will build documentation for a project if it is enabled through the CMake option BUILD_DOCUMENTATION. The documentation can be installed in either a specified directory to which the user has access or uploaded to a git branch. The latter effectively publishes our documentation to the internet, complete with source code (depending on the Doxygen options enabled). The user should add a file with the name of the project in the documentation directory. The expected format of each functional line is as follows:

<destination>|<path>|<doc_src>

<destination> is either local or git <path> is either a local path accessible to the user, or a git branch name (typically gh-pages) <doc_src> is the directory for the documentation to be copied to.

The presence of a project specific file will enable building of the documentation specified by the build target in the projects file. Note that the documentation build can be disabled if the user specifies "none" for the documentation target.

To add documentation to github, the user should create a branch named gh-pages containing the contents of the html directory created by doxygen.

In Step(7), pay particular attention to the settings of the shell, and to the two optional environment settings, CMAKE_PREFIX_PATH and PROJECT_CONFIGURE_OPTIONS. These options are passed to CMake when CTest configures your project(s) for automated build and testing. Ideally, any environment specific options should be handled by the module files to keep the build environment clean for subsequent projects.

For Step(8), make sure to replace "~/AutomatedTesting" with the directory that you created in Step(2). The usage for the run_automated_tests utility is as follows:

run_automated_tests <projects file> <mode> <running directory>

Valid default modes are Experimental, Nightly, and Continuous. The only real difference between Experimental and Nightly is how they are denoted in CDash. Experimental builds can happen at any time, whereas nightly tests are expected to be, well... nightly. Continuous builds are useful for things such as Test Driven Development. Continuous builds do not require cron jobs - but automatically update and build your project according to an interval that you can specify, say every 2 minutes or every 10 minutes for example.

In Step(9), your cron job will (presumably) invoke the nightly build/tests. Assuming you want to to run nightly build and test every night at midnight, say, then your crontab would be:

00 00 * * * /home/my/AutomatedTesting/run_automated_tests /home/my/AutomatedTesting/projects Nightly /home/my/AutomatedTesting

In general, the format for the crontab is:

MM HH DD NN W Command

Where MM is the two digit minute, HH the two digit hour (in 24 hour format), DD is date of the month, NN is the month, and W is the day of the week (0-6). "Command" is the command to be executed.

Once you have all of this working, the building and testing can be more easily customized. The CTest Manual is useful in learning about the ways this process can be customized further. Builds and tests can also be customized or made more specific by editing/tweaking the run_automated_tests script and cron job(s).