test_list_servers_by_limits_greater_than_actual_count fails if test account has pre-existing servers

[Daryl Walleck]

When running Tempest with tenant isolation disabled, test_list_servers_filter_by_exceed_limit of tempest.api.compute.servers.test_list_servers_negative will fail if there are any pre-existing servers on the test account. This test also seems very similar to test_list_servers_filter_by_exceed_limit from test_list_server_filters, which does not have assumptions about the number of servers on the account. If nothing else, test_list_servers_filter_by_exceed_limit should probably be skipped when run without tenant isolation enabled.

[Ghanshyam Mann (ghanshyammann)]

I did not find test_list_servers_filter_by_exceed_limit in tempest.api.compute.servers.test_list_servers_negative.

Do you means test_list_servers_by_limits_greater_than_actual_count test?

But in case of tenant isolation disabled, tests will run 1. serially - which does not have any issue as all tests will run serially and all of those should delete their servers during exit. 2. Predefined accounts- Here also tests case runs with predefined account and cleanup all created resources.

[Daryl Walleck (dwalleck)]

Sorry, I had a typo. You were correct that the test I was referring to was test_list_servers_by_limits_greater_than_actual_count.

This test assumes that only the two servers created in this fixture exist at the time this test runs. If for any reason (some other teardown fixture was unable to delete a server, manually created data, etc) the account has any other servers when this test runs, it will fail. Outside of tenant isolation, at best you can assume that these two servers should exist.

[Ghanshyam Mann (ghanshyammann)]

Yea, There should not be any situation where any server should be left on cleanup. If so then there is some issue. But I really do not find any much meaning of those tests as negative tests. Those are positive tests and same are present in test_list_server_filter.

[Ghanshyam Mann (ghanshyammann)]

Marking this incomplete till we have actual failure on gate for server leak by any tests. If so then we need to fix the tests which leaking the server.

Please feel free to restore this if you find any such failure or you have logs of failure on any jobs.

[Daryl Walleck (dwalleck)]

You would likely never see this test fail for one of the gate jobs. To re-state my original concern, this test creates a single resource in it's fixture, and then assumes that only that single server exists for that tenant. There is no checking that occurs to verify that the created server is the only one that exists for the tenant. For the gate jobs where tenants are being generated on the fly, you would likely never see this error. I'm running Tempest against a public cloud where we use a static list of accounts. For any number of reasons, it is possible that other servers exist for a given tenant.

My point is that a test should only rely on assumptions that are set in the fixture. If the assumption we want to make is that this created server is the only one that exists, then we should verify that assumption in the fixture and skip/fail the test otherwise. Otherwise an assumption is being made with no guarantee of its accuracy.

[Ghanshyam Mann (ghanshyammann)]

Yes, assumption is as per behavior.

gate has jobs for all three type of configuration-
1. tenant isolation - all current jobs
2. pre-existing account - experimental - "gate-tempest-dsvm-full-test-accounts"
3. serial - experimental - "gate-tempest-dsvm-full-non-isolated"

In case of pre-existing account,
 1. tests for single account runs in serial but there might be other tests running in parallel but with "another" account.

 2. List servers of this tests should not return servers created by another tenants. As this is not admin.

 3. Servers created by any tests ran on same tenant should not be left without cleanup.

So here only failure can happen on 3rd point and if so then, we should fix those particular cleanup instead of adjusting list tests which can hide cleanup issues.

[Ghanshyam Mann (ghanshyammann)]

So please provide if you have any failure logs which can help here to find such tests which does not cleaning up the servers. Otherwise I would like to see gate failure.

[Launchpad Janitor (janitor)]

[Expired for tempest because there has been no activity for 60 days.]