refstack-client - Inconsistency between test list supplied in argument to test list executed

[David Paterson]

Something seems wrong with how refstack-client is building the list of tests to run.

I passed in a file with 222 tests but it looks like 791 tests were run.

Command Run
===================================================
./refstack-client test -c tempest.conf -v --test-list "/home/stack/refstack-client/test_cases.txt"

Output
==================================================
2016-10-14 16:15:20,312 refstack_client.list_parser:139 DEBUG Number of tests: 222
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpeCEgLb
....
======
Totals
======
Ran: 791 tests in 7479.0000 sec.

=================================
Why 791 tests run instead of 222?

Attached is file containing both lists, test_cases.txt and the contents of /tmp/tmpeCEgLb which was passed to subunit.

[Catherine Diep (cdiep)]

Hi David,
There was a bug in RefStack that used IDs instead of fully qualified test names to generate the whitelist used by ostestr. This patch [1] fixes that. Could you please check to see whether this will help! Thanks a lot!

[1] https://review.openstack.org/#/c/381232/

[Catherine Diep (cdiep)]

https://review.openstack.org/#/c/381232/ merge Oct 10, 2016