Refstack client runs all tests ignoring the test-list option

[Rama Krishna Bhupathi]

I am running refstack_client passing the parameter test-list (whch contains only 2 tests) , but goes ahead and runs all the 1,300 odd tests , ignoring the test-list parameter I set. In this case my temp.list has only two tests (as reported in the debugging statement below) bu goes ahead and runs all the tests. I saw this behavior multiple times.

(.venv) stack@hlm003-cp1-c1-m1-mgmt:~/refstack-client$ ./refstack-client test -c ./tempest_region1.conf -v --test-list temp.list
2017-10-16 04:53:24,178 refstack_client:139 DEBUG The .testrepository/next-stream file was not found. Assuming subunit results will be stored in file 0.
2017-10-16 04:53:24,178 refstack_client:192 WARNING Using identity section of tempest config file to specify user credentials is deprecated and won't be supported soon. User credentials should be defined in the accounts file as described in the Tempest configuration guide (http://docs.openstack.org/developer/tempest/configuration.html).
2017-10-16 04:53:24,205 refstack_client:315 WARNING Problems retrieving CPID from Keystone using v3 endpoint (https://myhelion.test:5000/v3/auth/tokens) with error (HTTPSConnectionPool(host='myhelion.test', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)'),)))
2017-10-16 04:53:24,205 refstack_client:321 INFO Creating hash from endpoint to use as CPID.
2017-10-16 04:53:24,206 refstack_client:421 INFO Starting Tempest test...
2017-10-16 04:53:24,206 refstack_client:431 INFO Normalizing test list...
2017-10-16 04:53:26,914 refstack_client.list_parser:142 DEBUG Number of tests: 2
/home/stack/refstack-client/.tempest/.venv/local/lib/python2.7/site-packages/os_testr/ostestr.py:120: UserWarning: No .stestr.conf file found in the CWD. Please create one to to replace the .testr.conf. You can find a script to do this in the stestr git repository.
  warnings.warn(msg)
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent [0.977857s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent [1.978011s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents [1.912597s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter [3.459640s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent [1.648955s] ... ok

[Paul Van Eck (pvaneck-z)]

Hmm, there was recently a switch in testing frameworks used by tempest/ostestr. I just merged something that should hopefully fix the issue (https://review.openstack.org/#/c/503209/). So once that is in, try pulling the latest and reinstalling.

[Rama Krishna Bhupathi (ramakris)]

Ok. The patch seems to be working fine.
BTW .. I see these messages when I running the Defcore certified tests. These are WARNING messages...but trying to understand what these mean?

st_update_volume_with_empty_volume_id not found in Tempest list.
2017-10-17 01:49:23,342 refstack_client.list_parser:141 WARNING Test tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_volume_from_snapshot not found in Tempest list.
2017-10-17 01:49:23,342 refstack_client.list_parser:141 WARNING Test tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_update_volume_metadata_item not found in Tempest list.
2017-10-17 01:49:23,342 refstack_client.list_parser:141 WARNING Test tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_with_params not found in Tempest list.
2017-10-17 01:49:23,342 refstack_client.list_parser:141 WARNING Test tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_host_routes_and_dns_nameservers not found in Tempest list.
2017-10-17 01:49:23,342 refstack_client.list_parser:141 WARNING Test tempest.api.volume.test_availability_zone.AvailabilityZoneV2TestJSON.test_get_availability_zone_list not found in Tempest list.
2017-10-17 01:49:23,342 refstack_client.list_parser:141 WARNING Test tempest.api.network.test_networks.NetworksTestJSON.test_list_subnets not found in Tempest list.
2017-10-17 01:49:23,342 refstack_client.list_parser:141 WARNING Test tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_with_invalid_status not found in Tempest list.
2017-10-17 01:49:23,342 refstack_client.list_parser:141 WARNING Test tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_update_snapshot_metadata_item not found in Tempest list.
2017

[Catherine Diep (cdiep)]

The test names with warning messages are the tests with aliases due to test being relocated (patch changed) or renamed. Depending on the version of Tempest that you are running. Your test run should only locate and run one of the tests in the alias list.

For example:

For tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_volume_from_snapshot, there are 2 test names in
https://github.com/openstack/interop/blob/master/2017.01.json#L2788-L2791 .
Only one test ( in this case tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_volume_from_snapshot ) will be located in your tempest test list. The other will be logged with warning message.

[Martin Kopec (mkopec)]

Fixed as https://review.openstack.org/#/c/503209/ is merged.