enhance test_output.h to allow tests to be dynamically enabled/disabled

Bug #933717 reported by James Hunt on 2012-02-16
This bug affects 1 person
Affects Status Importance Assigned to Milestone

Bug Description

The test_output.h macros are used extensively for both NIH's own tests, and also for Upstarts.

Some of the Upstart test functions are getting very big with large numbers of TEST_FEATUREs in each. This makes developing new tests or attempting to debug existing tests awkward since all the tests have to be run to get to the failing test. Commenting out tests also isn't that trivial since with gcc's maximum warning levels, it barfs about not just _uninitialized_ variables, but also _unused_ ones now! To resolve that, you either need to fiddle with the compiler flags, or put in some
crude code temporarily to initialize+use each variable used by the commented out tests (not ideal). And yet, commenting out some tests is essential though in certain cases to allow other tests to be debugged via gdb (since some tests invoke ptrace directly).

I've come up with a minor interface change to TEST_FEATURE() et al - rather than just printing the test messages, it now accepts a block of code. This is highly advantageous since:

- with some helper code, tests could be individually enabled/disabled.
- a list of tests can be obtained without actually running hem.
- it could in principle check and enforce the environment between tests is clean (in
  a known state).

Overall this would make developing and debugging tests a lot easier.

The attached proof-of-concept shows the direction I'm looking at. The plan is that test_output.h will work as it does currently unless the test code defines a special value:

  #define NIH_TEST_VERSION_2

Once that symbol is defined, TEST_FEATURE() will expect a code block.

Tests could optionally call nih_test_init() and nih_test_teardown(). The advantage of doing so is that the test program will then automagically have command-line arguments created for it.

For example, if you compile the attached test_feature_macro_improved.c as 'foo', you can then run it like this:

 foo # Run all tests.
 foo --help # Usage.
 foo --list-tests # List all available tests, but don't execute them.
 foo -t 'foo feature 1' # Disable test called 'foo feature 1'.
 foo -t 'foo feature 1' --disable # Same as above.
 foo -t 'foo feature 1' --enable # Only run test 'foo feature 1'.
 foo -t 'foo feature 1' -e --quiet # Only run test 'foo feature 1' and suppress
                                   # warnings about other tests being disabled.

Planed extensions are to:

- allow a group of tests to be enabled/disabled
- allow a functions tests to be enabled/disabled
- read a list of tests/groups/functions to enable/disable from a file.
- read a list of regex's representing a list of tests/groups/functions to enable/disable from a file.
- ability to register functions before each test is run to say check the
  environment is clean, or maybe forcibly make it clean.
- ability to set tags on tests like "ptrace" or "fds" to denote that
  tests make use of ptrace directly. This would allow the ability to say
  "run all tests that don't use ptrace directly".

James Hunt (jamesodhunt) wrote :
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers