Overview

MyraTest packages all tests into a test runner executable that runs from the command line. You can use it to -show -all tests or -run -all tests, or you can work with just a subset by collecting tests by name or tags. There are also a handful of special arguments that control how the test runner receives input and generates output.

Below are links to the sections of this page:

Commands

The test runner has a rich command line interface, but the first argument (argv[1]), plays a special role as the "command". The most useful command is -help, which displays a quick reference. It displays (among other things) a complete list of the other available commands:

$ ./example_tests.out -help Usage: ./example_tests.out -command -options -command choices: -help Show this screen. -show Show tests, on stdout. -showtags Show the [tag]'s of tests, on stdout. -run Run tests. -doxygen Generate test summary for doxygen. (Note: this -command argument must come first) ...

We're already familiar with -run from the previous tutorial, it just runs tests. An equally useful one is -show, which lists all the tests compiled into the test runner. Try it out:

$ ./example_tests.out -show Collected 23 tests. strfibonacci : [str], [math] strfactorial : [str], [math] strpow2 : [str], [math] str2int : [str] strfind : [str] fibonacci : [math] factorial : [math] pow2 : [math] //fail : [.], [internal] exception5 : [internal] //exception2 : [.], [internal] //evaluation1 : [.], [internal] //evaluation2 : [.], [internal] //evaluation3 : [.], [internal] //exception6 : [.], [internal] arguments : [internal] pass : [internal] exception3 : [internal] output : [internal] //branch_fail : [.], [internal] //exception4 : [.], [internal] //exception1 : [.], [internal] branch : [internal]

The -show command prints the name of every test and then lists their associated tags. You can also "flip" this view around using the -showtags command, which prints every tag and then lists their associated tests:

$ ./example_tests.out -showtags Collected 23 tests. Collected 4 tags. //[.] : fail, exception2, evaluation1, evaluation2, evaluation3, exception6, branch_fail, exception4, exception1 [internal] : fail, exception5, exception2, evaluation1, evaluation2, evaluation3, exception6, arguments, pass, exception3, output, branch_fail, exception4, exception1, branch [math] : strfibonacci, strfactorial, strpow2, fibonacci, factorial, pow2 [str] : strfibonacci, strfactorial, strpow2, str2int, strfind

Note that a single test can have multiple tags, so it might appear more than once when using -showtags. The main purpose of -show and -showtags is to help you select tests more judiciously before you -run them. When working on a small feature of a large project, running the entire suite of tests might be too time consuming. Judicious use of -show can help you collect just the right tests (using names, wildcards and tags), then at the end just change -show to -run.

Collecting tests by name

If you'd like to run a small handful of tests, the easiest way to pick them out is by their individual name. For instance, to run just the fibonacci and factorial tests, just use -run -add=fibonacci -add=factorial:

$ ./example_tests.out -run -add=factorial -add=fibonacci Collected 2 tests: (1/2) fibonacci: pass (0.115006 sec) (2/2) factorial: pass (0.000000 sec) Summary: all tests passed (total of 45 assertions in 2 test cases) (0.115006 sec)

You can also collect test names that share a common prefix, by using a wildcard (prefix*). Let's collect all tests that start with f, using -add=f*. Before running anything, it's wise to use -show to find out exactly what your wildcard has collected:

$ ./example_tests.out -show -add=f* Collected 3 tests. factorial : [math] //fail : [.], [internal] fibonacci : [math]

We see that the -add=f* argument also collected a test called fail, but this is not one we wish to run. We can take it back out by following up our -add argument with an -omit argument. All these test-collecting arguments are applied left to right, so we want -add=f* -omit=fail:

$ ./example_tests.out -show -add=f* -omit=fail Collected 2 tests. factorial : [math] fibonacci : [math]

You can also -add and -omit tests by common suffix using a *suffix wildcard, or by common infix using a *infix* wildcard. These wildcards can be useful for collecting groups of tests, but requires rather careful choice of names to be most effective. For a more general alternative, consider collecting your tests by annotating them with tags.

Collecting tests by tag

Tagging tests is a useful way to group them in a way that's orthogonal to their names. When defining test cases, the ADD_TEST() macro takes two C-string arguments, ADD_TEST("name","[tag1][tag2][tag3]"). The -showtags command can list all the tags and their associated tests, then the -add=[tag] and -omit=[tag] arguments can use these tags to collect tests. For example, -add=[math] will collect all tests that have been tagged with [math]:

$ ./example_tests.out -show -add=[math] Collected 6 tests. strfibonacci : [str], [math] strfactorial : [str], [math] strpow2 : [str], [math] fibonacci : [math] factorial : [math] pow2 : [math]

If you provide multiple tags to -add, it works like an "and" operation. For instance, -add=[str][math] will only collect tests that have been tagged with both [str] and [math]:

$ ./example_tests.out -show -add=[str][math] Collected 3 tests. strfibonacci : [str], [math] strfactorial : [str], [math] strpow2 : [str], [math]

If you want an "or" operation, just use multiple arguments, like -add=[str] -add=[math]:

$ ./example_tests.out -show -add=[str] -add=[math] Collected 8 tests. strfibonacci : [str], [math] strfactorial : [str], [math] strpow2 : [str], [math] str2int : [str] strfind : [str] fibonacci : [math] factorial : [math] pow2 : [math]

The dot tag ([.]) is special and indicates a hidden test that cannot be collected using -all. You can always -add a hidden test by name to run it on-demand, or -add=[.] to get all of them. The example library has many hidden tests tagged with [.], most of them are coverage tests for MyraTest itself. Hiding a test by tagging it with [.] is a good way to suppress a questionable test that fails too often or takes too long, yet you still want it present in the suite.

Forwarding arguments to tests

To encourage reproducibility, tests should be written in such a way that they do not depend upon their environment (minimize "hidden" dependencies upon the filesystem/network, etc). But sometimes you don't have a choice, so MyraTest offers a way to forward command line arguments into a test using the special arguments -args_begin and -args_end. Anything in between them will be forwarded to the test and can be accessed using int myra::argc(void); and const char* myra::argv(int);. The example below, tests/internal/arguments.cpp shows how to use these functions, it just prints every forwarded argument:

 
 #include <myratest.h>
 
 ADD_TEST("arguments","[internal]")
   {
   int N = myra::argc();
   myra::out() << "::myra::argc() = " << N << std::endl;
   for (int n = 0; n < N; ++n)
     myra::out() << "  ::myra::argv(" << n << ") = " << myra::argv(n) << std::endl;
   }

Run this example using -run -add=arguments -args_begin 1 two 3.0 -args_end -out=cout:

$ ./example_tests.out -run -add=arguments -args_begin 1 two 3.0 -args_end -out=cout Collected 1 tests: ::myra::argc() = 4 ::myra::argv(0) = ./example_tests.out ::myra::argv(1) = 1 ::myra::argv(2) = two ::myra::argv(3) = 3.0 (1/1) arguments: pass (0.000000 sec) Summary: all tests passed (total of 0 assertions in 1 test cases) (0.000000 sec)

Note that the forwarded arguments begin with myra::argv(1). For historical reasons, myra::argv(0) always carries the executable name.

Redirecting the output

The recommended way to log output from a test is using myra::out(). It is a function that returns a std::ostream&, so you can use it just like std::cout (write to it using operator <<, use I/O manipulators, flush it using std::endl, and so forth). Depending upon the -out arguments given to the test runner, myra::out() can be connected to any of these things:

-out=none: connects myra::out() to nothing, the output is silenced. This is the default.
-out=cout: connects myra::out() to std::cout, the standard console output.
-out=file: connects myra::out() to a std::ofstream, using a unique file for each test.
-out=file1: connects myra::out() to a std::ofstream, appending all tests to a single file.

The example below, tests/internal/output.cpp demonstrates how to use myra::out():

 
 #include <myratest.h>
 
 #include <cmath>
 #include <iomanip>
 
 ADD_TEST("output","[internal]")
   {
   double pi = std::acos(-1.0);
   myra::out() << "Digits of pi:" << std::endl;
   for (int d = 1; d < 10; ++d)
     myra::out() << "  " << std::setprecision(d) << pi << std::endl;
   double epsilon = 1.0e-14;
   REQUIRE( std::fabs(std::cos(pi)+1.0) < epsilon );
   REQUIRE( std::fabs(std::sin(pi)) < epsilon );  
   }

By default, the test is silent when run:

$ ./example_tests.out -run -add=output Collected 1 tests: (1/1) output: pass (0.001000 sec) Summary: all tests passed (total of 2 assertions in 1 test cases) (0.001000 sec)

But with -out=cout it will print various digits of pi, using std::setprecision() from <iomanip>:

$ ./example_tests.out -run -add=output -out=cout Collected 1 tests: Digits of pi: 3 3.1 3.14 3.142 3.1416 3.14159 3.141593 3.1415927 3.14159265 (1/1) output: pass (0.000000 sec) Summary: all tests passed (total of 2 assertions in 1 test cases) (0.000000 sec)

Of course, myra::out() is opt-in, nothing prevents a test from using std::cout directly. But such output always appears on the console and tends to clutter the test summary.

When using -out=file or -out=file1, another useful option is -path=/to/some/folder, which redirects the created files to a user-specified location. By default, the present working directory is used.

Appendix: Coloring the output

To add visual emphasis to certain items, the MyraTest test runner adds a modest amount of color to its output (green/red for passing/failing tests, blue for hidden tests, etc). It uses standardized ANSI escape sequences (reference), which are understood by the common linux shells (bash, etc), cygwin, and the Windows10 console. Unfortunately the Windows7 console does not understand these codes, it just passes them verbatim to the display (see the ←[31;1m and similar fragments):

Click here to view fullsize image.

MyraTest provides a utility, win32colorizer.exe, that can be used to add color back to Windows7. When the output from the test runner is piped through this utility, it removes the ANSI color codes and replaces them with calls to SetConsoleTextAttribute() from the Windows.h API:

Click here to view fullsize image.

If this seems too much fuss, ANSI color is an opt-in feature that can be suppressed on any platform using -nocolor:

Click here to view fullsize image.

Appendix: Doxygen integration

MyraTest can generate Doxygen-compatible documentation for your test suite using the -doxygen command. MyraTest will run every test, capture the output to a -path=of/your/choice, and then create a summary page, tests.dox, that links it all together. To help Doxygen find all this content, add the following items to your Doxyfile:

Append the path/to/tests.dox onto Doxygen's INPUT variable.
Add the path to the root your test tree onto Doxygen's EXAMPLE_PATH variable, and set EXAMPLE_RECURSIVE=YES
Add a link to the testing page from your existing documentation, using \ref tests_tutorial.
If you annotate test files with a \brief Description of what they do, it will appear on the summary page.

The -doxygen documentation for this example can be found here: tests_tutorial