race condition in test_list_servers_negative.py

[Ivan-Zhu]

when run test_list_servers_negative.py in parallel mode, the following tests will be failed but running it serially will be ok.
 the config: allow_tenant_isolation = False

./run_tests.sh tempest.api.compute.servers.test_list_servers_negative
tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON
test_list_servers_by_changes_since[gate] FAIL
test_list_servers_by_limits_greater_than_actual_count[gate,negative]FAIL
tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestXML
test_list_servers_by_changes_since[gate] FAIL
test_list_servers_by_limits_greater_than_actual_count[gate,negative]FAIL

./run_tests.sh -t tempest.api.compute.servers.test_list_servers_negative
all tests will pass.

FAIL: tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_limits_greater_than_actual_count[gate,negative]
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/api/compute/servers/test_list_servers_negative.py", line 168, in test_list_servers_by_limits_greater_than_actual_count
    self.assertEqual(len(self.existing_fixtures), len(body['servers']))
  File "/opt/stack/tempest/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 322, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/tempest/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertThat
    raise MismatchError(matchee, matcher, mismatch, verbose)
MismatchError: 2 != 4

FAIL: tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestXML.test_list_servers_by_changes_since[gate]
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/api/compute/servers/test_list_servers_negative.py", line 193, in test_list_servers_by_changes_since
    (num_expected, body['servers']))
  File "/opt/stack/tempest/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 322, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/tempest/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertThat
    raise MismatchError(matchee, matcher, mismatch, verbose)
MismatchError: 3 != 6: Number of servers 3 is wrong in [{'link': {'href': 'http://9.115.127.125:8774/673fe79528b8463ba53bbe56d4a2400b/servers/2b42d0e3-ab62-4d81-8605-959696a0dfe7', 'rel': 'bookmark'}, 'name': 'ListServersNegativeTestXML-instance-tempest-1676945972', 'id': '2b42d0e3-ab62-4d81-8605-959696a0dfe7'}, {'link': {'href': 'http://9.115.127.125:8774/673fe79528b8463ba53bbe56d4a2400b/servers/f845ff0b-050e-4993-a5a6-9700d86b0adc', 'rel': 'bookmark'}, 'name': 'ListServersNegativeTestJSON-instance-tempest-1898524703', 'id': 'f845ff0b-050e-4993-a5a6-9700d86b0adc'}, {'link': {'href': 'http://9.115.127.125:8774/673fe79528b8463ba53bbe56d4a2400b/servers/63d84439-89aa-4a1c-86dc-73a13bd3fb9c', 'rel': 'bookmark'}, 'name': 'ListServersNegativeTestXML-instance-tempest-1143246935', 'id': '63d84439-89aa-4a1c-86dc-73a13bd3fb9c'}, {'link': {'href': 'http://9.115.127.125:8774/673fe79528b8463ba53bbe56d4a2400b/servers/78a2b19a-36ad-4644-afaa-5db69e56daae', 'rel': 'bookmark'}, 'name': 'ListServersNegativeTestJSON-instance-tempest-111422242', 'id': '78a2b19a-36ad-4644-afaa-5db69e56daae'}, {'link': {'href': 'http://9.115.127.125:8774/673fe79528b8463ba53bbe56d4a2400b/servers/92b50ccd-c55f-4ac6-8141-ef4a3f2dee5b', 'rel': 'bookmark'}, 'name': 'ListServersNegativeTestJSON-instance-tempest-1255923648', 'id': '92b50ccd-c55f-4ac6-8141-ef4a3f2dee5b'}, {'link': {'href': 'http://9.115.127.125:8774/673fe79528b8463ba53bbe56d4a2400b/servers/c229b92d-a1af-4401-9950-d95825595216', 'rel': 'bookmark'}, 'name': 'ListServersNegativeTestXML-instance-tempest-43758300', 'id': 'c229b92d-a1af-4401-9950-d95825595216'}]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/57384

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/57384
Committed: http://github.com/openstack/tempest/commit/892d43567bebc31283b56f9e22d554ee98b60e06
Submitter: Jenkins
Branch: master

commit 892d43567bebc31283b56f9e22d554ee98b60e06
Author: ivan-zhu <email address hidden>
Date: Wed Nov 20 15:13:08 2013 +0800

    Forced isolation for the tests in test_list_servers_negative.py

    In bug 1252971, race condition will arise if the allow_tenant_isolation
    is false. ListServersNegativeTestJSON and ListServersNegativeTestXML
    will share the tenant, if both of them check if there are existing servers
    or not before both of them create new servers, the actual servers will
    be double of the expected.
    this also removes the redundant code in this files.

    Closes-bug: 1252971

    Change-Id: I2f859057a71ccada86b5733c1a27386fcd6bc4b5

[David Kranz (david-kranz)]

It seems odd to me to force tenant isolation when the config says it is not allowed. Why don't we skip this test, and others like it, if isolation is not being used?

[Ivan-Zhu (ivan-zhu)]

hi david-kranz
Skip is ok for me. The approach: "force tenant isolation" is more like what we do in the quota tests.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/179401
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=a907622fc93fcfcd77e2767b90391f67b7d4b4c6
Submitter: Jenkins
Branch: master

commit a907622fc93fcfcd77e2767b90391f67b7d4b4c6
Author: David Kranz <email address hidden>
Date: Fri May 1 10:56:00 2015 -0400

    Remove unnecessary force_tenant_isolation

    The existing use of force_tenant_isolation seems to have been based on this
    bug 1252971. We no longer have that
    issue since there is no XML version, but I'm not sure why it was needed even
    then if running tempest in a valid way. If tenant isolation is on then there
    is no need. If not, then you are running serially, and there is no issue, or
    you are using test accounts in which case there is also no issue.

    Related-Bug: #1252971

    Change-Id: I8bce43c73538128d530688f1e3c8ac9a0c119e68