list ignores whitelist-file argument

[Trevor McCasland]

Steps to reproduce

1. clone tempest
2. install requirements
3. ostestr --list --whitelist-file whitelist.txt # lists all tests
4. stestr list --whitelist-file whitelist.txt # lists whitelisted tests

Given a whitelist of the following format
$ cat whitelist.txt
tempest.api.network.admin.test_floating_ips_admin_actions.FloatingIPAdminTestJSON.test_create_list_show_floating_ip_with_tenant_id_by_admin
tempest.api.network.admin.test_floating_ips_admin_actions.FloatingIPAdminTestJSON.test_list_floating_ips_from_admin_and_nonadmin
tempest.api.network.admin.test_negative_quotas.QuotasNegativeTest.test_network_quota_exceeding
tempest.api.network.test_networks.NetworksIpV6Test.test_external_network_visibility
tempest.api.network.test_networks.NetworksTest.test_external_network_visibility

ostestr --whitelist-file whitelist.txt --list

will not match the list provided.

Perhaps it couldn't find any tests because it does not use regex when listing files from a whitelist and defaults to listing all tests if the result set is empty but I would like to have the output to include the test metadata too, like
tempest.scenario.test_volume_migrate_attached.TestVolumeMigrateRetypeAttached.test_volume_migrate_attached[compute,id-deadd2c2-beef-4dce-98be-f86765ff311b,slow,volume]

When running with a whitelist it does run the correct files (I don't have services running at the moment).

$ ostestr --whitelist-file whitelist.txt
2017-10-26 12:32:35.648 10441 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf
2017-10-26 12:32:35.676 10443 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf
2017-10-26 12:32:35.696 10445 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf
2017-10-26 12:32:35.699 10447 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf
{0} setUpClass (tempest.api.network.test_networks.NetworksIpV6Test) ... SKIPPED: Neutron support is required
{1} setUpClass (tempest.api.network.admin.test_floating_ips_admin_actions.FloatingIPAdminTestJSON) ... SKIPPED: Neutron support is required
{3} setUpClass (tempest.api.network.admin.test_negative_quotas.QuotasNegativeTest) ... SKIPPED: Neutron support is required
{2} setUpClass (tempest.api.network.test_networks.NetworksTest) ... SKIPPED: Neutron support is required

as advertised, the list option states:
 --list, -l List all the tests which will be run.

but we can clearly see the --list option print all the tests, I did not include them for brevity but you can try it out.

stestr works fine though so we can skip this if we plan on replacing ostestr with stestr :)
$ stestr list --whitelist-file whitelist.txt
tempest.api.network.admin.test_floating_ips_admin_actions.FloatingIPAdminTestJSON.test_list_floating_ips_from_admin_and_nonadmin[id-64f2100b-5471-4ded-b46c-ddeeeb4f231b]
tempest.api.network.test_networks.NetworksTest.test_external_network_visibility[id-af774677-42a9-4e4b-bb58-16fe6a5bc1ec,smoke]
tempest.api.network.test_networks.NetworksIpV6Test.test_external_network_visibility[id-af774677-42a9-4e4b-bb58-16fe6a5bc1ec,smoke]
tempest.api.network.admin.test_negative_quotas.QuotasNegativeTest.test_network_quota_exceeding[id-644f4e1b-1bf9-4af0-9fd8-eb56ac0f51cf,negative]
tempest.api.network.admin.test_floating_ips_admin_actions.FloatingIPAdminTestJSON.test_create_list_show_floating_ip_with_tenant_id_by_admin[id-32727cc3-abe2-4485-a16e-48f2d54c14f2]

[Matthew Treinish (treinish)]

I'm just curious is os-testr installed in the same python environment as stestr 1.1.0? There was a bug in stestr where the whitelist argument was being ignored which should be fixed in stestr 1.1.0. We do a straight copy of the whitelist file argument to stestr's run command:

https://git.openstack.org/cgit/openstack/os-testr/tree/os_testr/ostestr.py#n168

so if there's a bug it's in stestr. But I was pretty sure we got all the filter stuff working in stestr with:

https://github.com/mtreinish/stestr/commit/978fc905bcc4bfeb8be96e7e3b85685cdde906ab

[Trevor McCasland (twm2016)]

yes it is, the versions are:
os-testr==1.0.0
stestr==1.1.0

stestr works, os-testr does not.

stestr list --whitelist-file whitelist.txt # output is as expected

ostestr --whitelist-file whitelist.txt --list # will not match the whitelist provided

[Trevor McCasland (twm2016)]

We were able to get around issues with blacklists and whitelists by downgrading to 0.8.2