Tempest Idempotent IDs are not unique under inheritance

[Yair Fried]

When inhering test classes, "test_*" methods are inherited along with their idempotent_id which makes it no longer unique:

tempest/api/volume/admin/test_volumes_actions.py
tempest.api.volume.test_volumes_negative.VolumesV1NegativeTest.test_volume_extend_with_nonexistent_volume_id[id-8f05a943-013c-4063-ac71-7baf561e82eb,negative]
tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_extend_with_nonexistent_volume_id[id-8f05a943-013c-4063-ac71-7baf561e82eb,negative


Is this by design or should inherited test methods have their own UUID?

[Yaroslav Lobankov (ylobankov)]

Hello Yair. This is purposely designed in such a way. We have two tests (for V1 and V2), but one test case. That is why two tests have the same IDs.

[Russell Sim (russell-sim)]

Hey Yaroslav,

I thought that the point of the UUIDs was to make sure that the tests could be tracked even if the tests are refactored.

In this case is the volume V1 and the V2 API considered to be the same functionality? Because if it's not then you are defeating the purpose of the UUID system, since to track any of these tests both the name and the UUID are required. What I mean, is unless both v1 and v2 are the same functionality, then how am I supposed to track the tests after a refactoring?

I might also point out that the documentation also states that these should be unique. http://docs.openstack.org/developer/tempest/HACKING.html#test-identification-with-idempotent-id

Cheers,
Russell

[Russell Sim (russell-sim)]

From what i can see there are currently 538 tests that share ids. There are up to 3 tests sharing a single id. The triplets are all coming from the neutron tests, all due to the IPv4 vs IPv6 vs IPv6-Attr tests

e.g.
tempest.api.network.test_networks.NetworksIpV6Test.test_list_networks[id-f7ffdeda-e200-4a7a-bcbe-05716e86bf43,smoke]
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_list_networks[id-f7ffdeda-e200-4a7a-bcbe-05716e86bf43,smoke]
tempest.api.network.test_networks.NetworksTest.test_list_networks[id-f7ffdeda-e200-4a7a-bcbe-05716e86bf43,smoke]

Other examples from heat,

tempest.api.orchestration.stacks.test_templates.TemplateAWSTestJSON.test_validate_template[id-ed53debe-8727-46c5-ab58-eba6090ec4de]
tempest.api.orchestration.stacks.test_templates.TemplateYAMLTestJSON.test_validate_template[id-ed53debe-8727-46c5-ab58-eba6090ec4de]

And another example from nova,

tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_duplicate_network_nics[id-1678d144-ed74-43f8-8e57-ab10dbf9b3c2]
tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_duplicate_network_nics[id-1678d144-ed74-43f8-8e57-ab10dbf9b3c2]

Anyway, from what I can see the only way to fix this is to manually override every inherited test case with a wrapper that creates a new distinct ID.

e.g.

 class AvailabilityZoneV1TestJSON(AvailabilityZoneV2TestJSON):
     _api_version = 1

     @test.idempotent_id('01f1ae88-eba9-4c6b-a011-6f7ace06b725')
     def test_get_availability_zone_list(self):
         (super(AvailabilityZoneV1TestJSON, self)
          .test_get_availability_zone_list())

Thoughts?

[Ghanshyam Mann (ghanshyammann)]

Yea so tests with subclass will be having the same UUID as they are same tests not different.

If any tests testing any feature with different version of APIs, it means this is same tests and can have same UUID.

Let me put more points on Idempotent IDs. Idempotent IDs were mainly introduced to identify the different tests by unique id instead of long test name which can be refactored any time.

Tests in inheritance, mainly are different by either API version or any configuration options etc, which means they are same tests and should have same Idempotent IDs.

If there is any tests having same Idempotent IDs or kept in inheritance which supposed to or are testing different feature than we should have different id and in separate class so.

So here Idempotent IDs is useful when refactoring the tests or name etc.

[Ghanshyam Mann (ghanshyammann)]

I am marking this invalid as this is as per design and same tests(even 2 in subclass) has single Idempotent IDs.