MyraTest
Tutorial for writing tests

Overview

Our first experience with MyraTest will be simple: a single function checked by a single test, all within a single file. After some experimentation here with the basic API and the break/test/fix development cycle, we'll study a larger example that introduces a more realistic physical design with separate compilation and shared linkage between tests and the code under test. This example explores more advanced features of MyraTest: (i) adding tags to tests to group them, (ii) how MyraMath handles exceptions, and (iii) how to branch within tests to extract repetitive setup code.

Below are links to the sections of this page:

Our first test

For motivation, consider the function below, pow2(k), that computes integer powers of two (2k) recursively:

1 int pow2(int k)
2  { return (k > 1) ? 2*pow2(k-1) : 2; }

It looks correct enough, but you can never be too careful. Let's write some tests for it using MyraTest, to verify that it correctly computes 23=8 and 24=16. For now, just squash everything into one .cpp file (translation unit) to keep things self contained. See tutorial/tutorial1.cpp, listed below:

1 #define MYRATEST_MAIN
2 #include <myratest.h>
3 
4 int pow2(int k)
5  { return (k > 1) ? 2*pow2(k-1) : 2; }
6 
7 ADD_TEST("test_pow2","")
8  {
9  REQUIRE( pow2(3) == 8 );
10  REQUIRE( pow2(4) == 16 );
11  }
12 
13 // g++ tutorial1.cpp -o tutorial1.out -I/path/to/myratest.h

Including myratest.h provides the ADD_TEST() macro, which is a way to group/lexically block your testing code and give it a unique name ("test_pow2", in this instance). Within a test, you can use the REQUIRE() macro to assert a (boolean) expression. The #define MYRATEST_MAIN macro is special, it informs MyraTest to expand an int main() into this translation unit, which implements the test runner. You can run the test from the command line by writing ./tutorial1.out -run -all, which yields:

$ ./tutorial1.out -run -all Collected 1 tests: (1/1) test_pow2: pass (0.000000 sec) Summary: all tests passed (total of 2 assertions in 1 test cases) (0.000000 sec)

Let's improve the coverage, making sure that pow2(k) works for more choices of k. See tutorial/tutorial2.cpp below:

1 #define MYRATEST_MAIN
2 #include <myratest.h>
3 
4 int pow2(int k)
5  { return (k > 1) ? 2*pow2(k-1) : 2; }
6 
7 ADD_TEST("test_pow2","")
8  {
9  REQUIRE( pow2(0) == 1 );
10  REQUIRE( pow2(1) == 2 );
11  REQUIRE( pow2(2) == 4 );
12  REQUIRE( pow2(3) == 8 );
13  REQUIRE( pow2(4) == 16 );
14  REQUIRE( pow2(5) == 32 );
15  REQUIRE( pow2(6) == 64 );
16  REQUIRE( pow2(7) == 128 );
17  REQUIRE( pow2(8) == 256 );
18  REQUIRE( pow2(9) == 512 );
19  REQUIRE( pow2(10) == 1024 );
20  }
21 
22 // g++ tutorial2.cpp -o tutorial2.out -I/path/to/myratest.h

This time, the summary indicates a failure:

$ ./tutorial2.out -run -all Collected 1 tests: (1/1) test_pow2: fail (0.000000 sec) Summary: 0 pass, 1 fail (total of 1 assertions in 1 test cases).

By default, MyraTest says little. Adding the -out=cout argument provides more detail on the standard output:

$ ./tutorial2.out -run -all -out=cout Collected 1 tests: [FAIL] REQUIRE() failure in TEST = "test_pow2", located within ./tutorial/tutorial2.cpp at line 9 Note: the expression "pow2(0) == 1" evaluated into "2 == 1" (1/1) test_pow2: fail (0.000000 sec) Summary: 0 pass, 1 fail (total of 1 assertions in 1 test cases).

MyraTest indicates the name and file of the failing test, as well as the line number of the REQUIRE() in question. When possible, it expands both sides of the conditional expression into actual values. The defect here is the choice of base case, 20=1, not 2. The fix is given by tutorial/tutorial3.cpp, which also splits off k=0 into an independent test.

1 #define MYRATEST_MAIN
2 #include <myratest.h>
3 
4 int pow2(int k)
5  { return (k > 0) ? 2*pow2(k-1) : 1; }
6 
7 ADD_TEST("test_pow2(0)","")
8  {
9  REQUIRE( pow2(0) == 1 );
10  }
11 
12 ADD_TEST("test_pow2(k)","")
13  {
14  REQUIRE( pow2(1) == 2 );
15  REQUIRE( pow2(2) == 4 );
16  REQUIRE( pow2(3) == 8 );
17  REQUIRE( pow2(4) == 16 );
18  REQUIRE( pow2(5) == 32 );
19  REQUIRE( pow2(6) == 64 );
20  REQUIRE( pow2(7) == 128 );
21  REQUIRE( pow2(8) == 256 );
22  REQUIRE( pow2(9) == 512 );
23  REQUIRE( pow2(10) == 1024 );
24  }
25 
26 // g++ tutorial3.cpp -o tutorial3.out -I/path/to/myratest.h

This time, the summary shows the defect has been corrected:

$ ./tutorial3.out -run -all Collected 2 tests: (1/2) test_pow2(0): pass (0.000000 sec) (2/2) test_pow2(k): pass (0.000000 sec) Summary: all tests passed (total of 11 assertions in 2 test cases) (0.000000 sec)

A larger example

Placing all your tests and code under test within a single file doesn't scale very well. For larger programs/libraries, physical design (separating functionality across files/folders) is an important consideration. Although some unit testing frameworks encourage tests to be "embedded" alongside the code under test, MyraTest is designed to "bolt-onto" existing/legacy libraries (.so's or .dll's), by writing new test code that calls the public API of the library just like any other client would. Test cases can be split across multiple translation units (.cpp's), but only one of them should contain the #define MYRATEST_MAIN macro (typically, this will be its own .cpp file that contains nothing else).

The folder structure of an example project is shown below:

o
|
|
+--example ]
| | ]
| +---math ]
| | factorial.cpp ] ]
| | factorial.h ] ]
| | fibonacci.cpp ] ]
| | fibonacci.h ] ]
| | pow2.cpp ] ]-- Compiled/gathered into a shared
| | pow2.h ] ] library (example.so/example.dll) <-----.
| | ] |
| \---str ] |
| str2int.cpp ] ] |
| str2int.h ] ] Linked as a shared library, |
| strfind.cpp ] ] just like other client code. |
| strfind.h ] ] |
| |
| Code under test. |
========================== =====
| Testing code. |
| |
\--tests |
| myratest.h Has #define MYRATEST_MAIN, ] |
| example_tests.cpp ]-- expands into "int main()" ] |
| for the test runner. ] |
+---internal ] |
| arguments.cpp ] ] |
| branch.cpp ] ] |
| branch_fail.cpp ] ] |
| evaluation.cpp ] ] |
| exception0.cpp ] ] |
| exception1.cpp ] Includes <myratest.h>, has ] |
| exception2.cpp ]-- ADD_TEST() cases, but they ] |
| exception3.cpp ] only test MyraTest itself. ] |
| exception4.cpp ] ]
| exception5.cpp ] ] Compiled into example_tests
| fail.cpp ] ] executable (.out/.exe), can
| output.cpp ] ]-- be launched from console to
| pass.cpp ] ] select and run tests.
| ]
| ]
+---math Includes <myratest.h> and ]
| factorial.cpp ] headers from example/math, ]
| fibonacci.cpp ]-- contains ADD_TEST() cases ]
| pow2.cpp ] for example/math functions. ]
| ]
| ]
+---str Includes <myratest.h> and ]
| str2int.cpp ] headers from example/str, ]
| strfind.cpp ]-- contains ADD_TEST() cases ]
| for example/str functions. ]
| ]
| ]
\---integration Includes <myratest.h> and ]
strfactorial.cpp ] headers from example/*, ]
strfibonacci.cpp ]-- contains ADD_TEST() cases ]
strpow2.cpp ] across multiple functions. ]

All of these tests are compiled into the same test runner executable, so you can run them all together:

$ ./example_tests.out -run -all Collected 14 tests: (1/14) strfibonacci: pass (0.000000 sec) (2/14) strfactorial: pass (0.000000 sec) (3/14) strpow2: pass (0.000000 sec) (4/14) str2int: pass (0.000000 sec) (5/14) strfind: pass (0.000000 sec) (6/14) fibonacci: pass (0.115006 sec) (7/14) factorial: pass (0.000000 sec) (8/14) pow2: pass (0.000000 sec) (9/14) exception5: pass (0.000000 sec) (10/14) arguments: pass (0.000000 sec) (11/14) pass: pass (0.000000 sec) (12/14) exception3: pass (0.000000 sec) (13/14) output: pass (0.000000 sec) (14/14) branch: pass (0.000000 sec) Summary: all tests passed (total of 90 assertions in 14 test cases) (0.115006 sec)

Although running all tests provides a useful "thumbs-up/thumbs-down" for the library as a whole, sometimes it's helpful to be able to run your tests with finer granularity. If you're only changing one feature, the "edit/compile/test" cycle could be accelerated by running only the subset of tests that are known to depend upon that feature. MyraTest's "tagging" mechanism can help here.

Adding tags to tests

The ADD_TEST() macro takes two C-string arguments, ADD_TEST("name","[tags]"). The first is a name for the test, a unique identifier that the test runner uses for report generation. The second is an unordered collection of tags, which should be [delimited][with][brackets]. The tests below (tests/str/str2int.cpp, tests/math/factorial.cpp, and tests/integration/strfactorial.cpp) have all been annotated with tags:

1 
6 #include <example/str/str2int.h>
7 
8 #include <myratest.h>
9 
10 ADD_TEST("str2int","[str]")
11  {
12  using namespace example;
13  REQUIRE(str2int("7") == 7);
14  REQUIRE(str2int("13") == 13);
15  REQUIRE(str2int("546") == 546);
16  REQUIRE(str2int("2980") == 2980);
17  }
1 
6 #include <example/math/factorial.h>
7 
8 #include <myratest.h>
9 
10 ADD_TEST("factorial","[math]")
11  {
12  using namespace example;
13  REQUIRE(factorial(0) == 1);
14  REQUIRE(factorial(1) == 1);
15  REQUIRE(factorial(2) == 2);
16  REQUIRE(factorial(3) == 6);
17  REQUIRE(factorial(4) == 24);
18  REQUIRE(factorial(5) == 120);
19  REQUIRE(factorial(6) == 720);
20  REQUIRE(factorial(7) == 5040);
21  REQUIRE(factorial(8) == 40320);
22  REQUIRE(factorial(9) == 362880);
23  }
24 
1 
6 #include <example/str/str2int.h>
7 #include <example/math/factorial.h>
8 
9 #include <myratest.h>
10 
11 ADD_TEST("strfactorial","[str][math]")
12  {
13  using namespace example;
14  REQUIRE(factorial(str2int("4")) == str2int("24"));
15  REQUIRE(factorial(str2int("5")) == str2int("120"));
16  }

These tags denote ad-hoc groups of tests that can be run independently within example_tests.out by choosing different command line arguments. For instance, try replacing -run -all with -run -add=[str] or -run -add=[math]:

$ ./example_tests.out -run -add=[str] Collected 5 tests: (1/5) strfibonacci: pass (0.000000 sec) (2/5) strfactorial: pass (0.000000 sec) (3/5) strpow2: pass (0.000000 sec) (4/5) str2int: pass (0.000000 sec) (5/5) strfind: pass (0.000000 sec) Summary: all tests passed (total of 25 assertions in 5 test cases) (0.000000 sec)
$ ./example_tests.out -run -add=[math] Collected 6 tests: (1/6) strfibonacci: pass (0.000000 sec) (2/6) strfactorial: pass (0.000000 sec) (3/6) strpow2: pass (0.000000 sec) (4/6) fibonacci: pass (0.117006 sec) (5/6) factorial: pass (0.000000 sec) (6/6) pow2: pass (0.000000 sec) Summary: all tests passed (total of 68 assertions in 6 test cases) (0.117006 sec)

The str2int test was only run the first time, while the factorial test was only run the second. The strfactorial test was run both times, because it carries both the [str] and [math] tags.

On exceptions

There are multiple strategies for handling errors in C++, the two most common ones are (i) propagating error codes within return values, and (ii) throwing exceptions. Dealing with (i) is straightforward, just capture the error code and test it using a REQUIRE() macro as usual. Dealing with (ii) can be trickier because of the way throw statements alter the flow of control. See tests/internal/exception1.cpp and tests/internal/exception2.cpp, below, for example tests that throw exceptions:

1 
6 #include <myratest.h>
7 
8 ADD_TEST("exception1","[.][internal]")
9  {
10  throw int(5);
11  }
1 
6 #include <myratest.h>
7 #include <stdexcept>
8 
9 ADD_TEST("exception2","[.][internal]")
10  {
11  throw std::runtime_error("Hello, I am a std::exception.\"");
12  }

When the code under test throw's, MyraTest will catch and mark the test as failing. If the exception happens to inherit from std::exception (recommended practice), MyraTest will also print the .what() message:

$ ./example_tests.out -run -add=exception1 -out=cout Collected 1 tests: [FAIL] Uncaught exception (of unknown type) in TEST = "exception1", located within tests/internal/exception1.cpp near line 8 (1/1) exception1: fail (0.000000 sec) Summary: 0 pass, 1 fail (total of 0 assertions in 1 test cases).
$ ./example_tests.out -run -add=exception2 -out=cout Collected 1 tests: [FAIL] Uncaught std::exception in TEST = "exception2", located within tests/internal/exception2.cpp near line 9 Note: e.what() = Hello, I am a std::exception." (1/1) exception2: fail (0.000000 sec) Summary: 0 pass, 1 fail (total of 0 assertions in 1 test cases).

Of course, sometimes throwing an exception is the expected/desired behavior, like when calling a bounds-checked accessor with an invalid index. In this case, we'd like to test/enforce that invalid code really does throw. Use the REQUIRE_EXCEPTION() macro for this. The test below, tests/internal/exception3.cpp shows how to use it:

1 
6 #include <myratest.h>
7 #include <vector>
8 
9 ADD_TEST("exception3","[internal]")
10  {
11  std::vector<int> v(3,0); // [ 0 0 0 ]
12  REQUIRE( v.at(0) == 0 );
13  REQUIRE( v.at(1) == 0 );
14  REQUIRE( v.at(2) == 0 );
15  REQUIRE_EXCEPTION( int i = v.at(10); ); // out of bounds
16  }

The statement int i = v.at(10); throws an exception due to out of bounds access. But because it is expected, and inside a REQUIRE_EXCEPTION() macro, MyraTest catches the exception and the test passes:

$ ./example_tests.out -run -add=exception3 Collected 1 tests: (1/1) exception3: pass (0.000000 sec) Summary: all tests passed (total of 4 assertions in 1 test cases) (0.000000 sec)

On the other hand, if a statement within a REQUIRE_EXCEPTION() macro doesn't throw anything, MyraTest will mark the test as failing. See tests/internal/exception4.cpp, below:

1 
6 #include <myratest.h>
7 #include <vector>
8 
9 ADD_TEST("exception4","[.][internal]")
10  {
11  std::vector<int> v(3,0); // [ 0 0 0 ]
12  REQUIRE( v.at(0) == 0 );
13  REQUIRE( v.at(1) == 0 );
14  REQUIRE( v.at(2) == 0 );
15  REQUIRE_EXCEPTION( int i = v.at(1); ); // in bounds
16  }

The statement int i = v.at(1); is within the bounds [0,3). The underlying code doesn't throw any exception, but REQUIRE_EXCEPTION() is expecting one, so the test is marked as failing:

$ ./example_tests.out -run -add=exception4 -out=cout Collected 1 tests: [FAIL] REQUIRE_EXCEPTION() failure in TEST = "exception4", located within tests/internal/exception4.cpp at line 15 Note: the statement "int i = v.at(1);" was expected to throw an exception, but it didn't. (1/1) exception4: fail (0.000000 sec) Summary: 0 pass, 1 fail (total of 4 assertions in 1 test cases).

Test branches

Occasionally, multiple tests will contain common setup ("fixture") code to set up the preconditions/environment for the tests themselves. MyraTest's mechanism for reusing setup code is the ADD_BRANCH() macro. The example below, tests/internal/branch.cpp, demonstrates the syntax:

1 
6 #include <myratest.h>
7 
8 ADD_TEST("branch","[internal]")
9  {
10  int i = 1;
11  ADD_BRANCH("branch1")
12  {
13  myra::out() << "visiting branch1" << std::endl;
14  ++i;
15  }
16  ADD_BRANCH("branch2")
17  {
18  myra::out() << "visiting branch2" << std::endl;
19  ++i;
20  }
21  ADD_BRANCH("branch3")
22  {
23  myra::out() << "visiting branch3" << std::endl;
24  ++i;
25  }
26  REQUIRE(i==2);
27  }

When multiple branches are introduced into the body of a test, it will be run multiple times. During each pass, the same setup code (located at the top of the test, above the branches) will be rerun, but then a different/unique branch will be chosen. In this example, we see each of the visiting ... messages in sequence. But because only one branch is executed on each pass, the final value of i should always be just 2, which is verified by the REQUIRE() macro:

$ ./example_tests.out -run -add=branch -out=cout Collected 1 tests: visiting branch1 visiting branch2 visiting branch3 (1/1) branch: pass (0.000000 sec) Summary: all tests passed (total of 3 assertions in 1 test cases) (0.001000 sec)

Each ADD_BRANCH() macro requires a C-string argument for a name, which will appear in the error message when a failure occurs. See tests/internal/branch_fail.cpp, below:

1 
6 #include <myratest.h>
7 
8 ADD_TEST("branch_fail","[.][internal]")
9  {
10  int i = 1;
11  ADD_BRANCH("good1")
12  {
13  myra::out() << "visiting good1" << std::endl;
14  REQUIRE(true);
15  ++i;
16  }
17  ADD_BRANCH("bad")
18  {
19  myra::out() << "visiting bad" << std::endl;
20  REQUIRE(false);
21  ++i;
22  }
23  ADD_BRANCH("good2")
24  {
25  myra::out() << "visiting good2" << std::endl;
26  REQUIRE(true);
27  ++i;
28  }
29  REQUIRE(i==2);
30  }

Although a failure within an earlier branch doesn't prevent later branches from running, MyraTest will mark the encapsulating test as a failure:

$ ./example_tests.out -run -add=branch_fail -out=cout Collected 1 tests: visiting good1 visiting bad [FAIL] REQUIRE() failure in TEST = "branch_fail", BRANCH = "bad", located within tests/internal/branch_fail.cpp at line 20 Note: the expression "false" evaluated into "false" visiting good2 (1/1) branch_fail: fail (0.000000 sec) Summary: 0 pass, 1 fail (total of 5 assertions in 1 test cases).

Summary

Writing unit tests using MyraTest entails the following steps:

The next tutorial will examine the test runner in greater detail.

Continue to Tutorial for running tests, or go back to API Tutorials

Generated on Sun Aug 4 2024 11:37:33 for MyraTest by   doxygen 1.8.13