refstack-client removes tests which could be found by id (renamed tests)

[Neal Oakey]

refstack-client's list_parser will filter the tests by comparing the name of the tests with the `testr list` from tempest.

If tempest is updated these names will change, but the [id-...] attr will stay the same.
So why are the Tests selected by the test name not by their id?

As a fancy side effect one could let (if set with a parameter) the refstack-client fail if there are tests missing.
So you can directly see if you will complete the Interop or not,
currently you will have to compare the number of tests or upload the result.

[Catherine Diep (cdiep)]

idempotent_id will stay the same for a test method but not for a test case which is defined using a fully qualified test name.

For example, test case [1] and [2] perform the same test which is test_list_networks_fields with id-6ae6d24f-9194-4869-9c85-c313cb20e080, but they are two different test cases. One tests for IPv4 feature while the other for IPv6. Until we have a unique way to identify test case, using a fully qualified test name is the choice for now. Please see more information in:

https://bugs.launchpad.net/tempest/+bug/1433700
https://bugs.launchpad.net/tempest/+bug/1470711

[1] tempest.api.network.test_networks.NetworksIpV6Test.test_list_networks_fields[id-6ae6d24f-9194-4869-9c85-c313cb20e080]
[2] tempest.api.network.test_networks.NetworksTest.test_list_networks_fields[id-6ae6d24f-9194-4869-9c85-c313cb20e080]

[Martin Kopec (mkopec)]

Based on the info provided in the comments and the age of the bug report, I'm gonna mark this as Invalid. In case you feel otherwise, feel free to create a new bug report in the new bug tracker system:
https://storyboard.openstack.org/#!/project/openinfra/refstack-client