testr has no way to exclude tests other than with regexp

[David Kranz]

The use case prompting this is where there a bunch of tests that you want to include based on a match in the test name, but some of them should be excluded based of matching other parts of the name. This was especially desired because testr makes tags on test cases part of the name. As is, constructing regexps out of a combination of includes and excludes is hard for those who are not true regexp wizards.

Here is a proposal:

Provide a --exclude REGEXP argument that can appear any number of times. After the list of included tests is constructed, each test whose name matches one of the excludes is removed.

[Robert Collins (lifeless)]

So this basically builds a language for including and excluding test, which is exactly what a regex is. What happens if someone includes two different patterns and excludes two different patterns? Is it evaluated left to right or union-adds then subtract all excludes? What if someone can't exclude precisely and wants to exclude then include?

I'm open to the idea of having a meta language rather than just regexp, but it needs to be designed, with care, or it just becomes a wart.

[Matthew Treinish (treinish)]

So back during the OpenStack Summit in HK the idea was brought up that we should just specify using the same grammar as rsync for doing this. See these links for examples of this:

http://www.thegeekstuff.com/2011/01/rsync-exclude-files-and-folders/

and the man page:

http://linux.die.net/man/1/rsync

I think this section is probably the relevant portion(at least part of it):

-F same as --filter='dir-merge /.rsync-filter'
                             repeated: --filter='- .rsync-filter'
     --exclude=PATTERN exclude files matching PATTERN
     --exclude-from=FILE read exclude patterns from FILE
     --include=PATTERN don't exclude files matching PATTERN
     --include-from=FILE read include patterns from FILE
     --files-from=FILE read list of source-file names from FILE

[Bob Ball (bob-ball)]

See also https://review.openstack.org/#/c/81565/.

This is crucial functionality that's needed by several 3rd party CI systems; for Nova each 3rd party CI system has it's own method for excluding tests.

Is Matthew's suggestion above the way everyone would like to see this go, or are there alternatives?

[Matthew Thode (prometheanfire)]

I would like to be able to exclude specific tests while allowing others (like restricting test_get_system_ca_file in bug 1313257 ). At the moment I have to restrict testing entirely.