enhance test_output.h to allow tests to be dynamically enabled/disabled
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
libnih |
New
|
Undecided
|
Unassigned |
Bug Description
The test_output.h macros are used extensively for both NIH's own tests, and also for Upstarts.
Some of the Upstart test functions are getting very big with large numbers of TEST_FEATUREs in each. This makes developing new tests or attempting to debug existing tests awkward since all the tests have to be run to get to the failing test. Commenting out tests also isn't that trivial since with gcc's maximum warning levels, it barfs about not just _uninitialized_ variables, but also _unused_ ones now! To resolve that, you either need to fiddle with the compiler flags, or put in some
crude code temporarily to initialize+use each variable used by the commented out tests (not ideal). And yet, commenting out some tests is essential though in certain cases to allow other tests to be debugged via gdb (since some tests invoke ptrace directly).
I've come up with a minor interface change to TEST_FEATURE() et al - rather than just printing the test messages, it now accepts a block of code. This is highly advantageous since:
- with some helper code, tests could be individually enabled/disabled.
- a list of tests can be obtained without actually running hem.
- it could in principle check and enforce the environment between tests is clean (in
a known state).
Overall this would make developing and debugging tests a lot easier.
The attached proof-of-concept shows the direction I'm looking at. The plan is that test_output.h will work as it does currently unless the test code defines a special value:
#define NIH_TEST_VERSION_2
Once that symbol is defined, TEST_FEATURE() will expect a code block.
Tests could optionally call nih_test_init() and nih_test_
For example, if you compile the attached test_feature_
foo # Run all tests.
foo --help # Usage.
foo --list-tests # List all available tests, but don't execute them.
foo -t 'foo feature 1' # Disable test called 'foo feature 1'.
foo -t 'foo feature 1' --disable # Same as above.
foo -t 'foo feature 1' --enable # Only run test 'foo feature 1'.
foo -t 'foo feature 1' -e --quiet # Only run test 'foo feature 1' and suppress
Planed extensions are to:
- allow a group of tests to be enabled/disabled
- allow a functions tests to be enabled/disabled
- read a list of tests/groups/
- read a list of regex's representing a list of tests/groups/
- ability to register functions before each test is run to say check the
environment is clean, or maybe forcibly make it clean.
- ability to set tags on tests like "ptrace" or "fds" to denote that
tests make use of ptrace directly. This would allow the ability to say
"run all tests that don't use ptrace directly".