Tests fail if preprovisioned accounts picked from the same project.

[Mitya Eremeev]

env:
- use preprovisioned accounts for tests.
- static accounts file contains accounts. Some accounts of static accounts file are from the same project and some accounts from different projects.

Many tests fail:

Traceback (most recent call last):

  File "/var/lib/openstack/lib/python3.6/site-packages/tempest/api/compute/servers/test_servers_negative.py", line 615, in test_update_server_of_another_tenant

    name=new_name)

  File "/var/lib/openstack/lib/python3.6/site-packages/testtools/testcase.py", line 489, in assertRaises

    self.assertThat(our_callable, matcher)

  File "/var/lib/openstack/lib/python3.6/site-packages/testtools/testcase.py", line 502, in assertThat

    raise mismatch_error

testtools.matchers._impl.MismatchError: <bound method ServersClient.update_server of <tempest.lib.services.compute.servers_client.ServersClient object at 0x7fe5e9692898>> returned {'server': {'id': '9fafdc92-7782-466f-948b-94dddbed12bb', 'name': '9fafdc92-7782-466f-948b-94dddbed12bb_new', 'status': 'ACTIVE', 'tenant_id': 'a363b884e59e453d93b34dd70888e9c1', 'user_id': 'c614029d381049dd80f9e4fb2df3f3a2', 'metadata': {}, 'hostId': '939e120cb312fde986e5ce8c29e4e68799344615f563f347079ddc83', 'image': {'id': '49361c2b-ac92-4597-ad93-fc77fe9e61df', 'links': [{'rel': 'bookmark', 'href': 'https://nova.it.just.works/a363b884e59e453d93b34dd70888e9c1/images/49361c2b-ac92-4597-ad93-fc77fe9e61df'}]}, 'flavor': {'id': 'c46c6773-8559-4cfe-a7aa-48003f5ff2af', 'links': [{'rel': 'bookmark', 'href': 'https://nova.it.just.works/a363b884e59e453d93b34dd70888e9c1/flavors/c46c6773-8559-4cfe-a7aa-48003f5ff2af'}]}, 'created': '2021-10-19T06:01:01Z', 'updated': '2021-10-19T06:01:10Z', 'addresses': {'tempest-manual-0': [{'version': 4, 'addr': '192.168.0.185'}]}, 'accessIPv4': '', 'accessIPv6': '', 'links': [{'rel': 'self', 'href': 'https://nova.it.just.works/v2.1/a363b884e59e453d93b34dd70888e9c1/servers/9fafdc92-7782-466f-948b-94dddbed12bb'}, {'rel': 'bookmark', 'href': 'https://nova.it.just.works/a363b884e59e453d93b34dd70888e9c1/servers/9fafdc92-7782-466f-948b-94dddbed12bb'}], 'OS-DCF:diskConfig': 'MANUAL', 'progress': 0}}

Trace:

testtools.testresult.real._StringException: pythonlogging:'': {{{

2021-10-19 06:00:51,132 24 DEBUG [urllib3.connectionpool] Resetting dropped connection: nova.it.just.works

2021-10-19 06:00:51,637 24 DEBUG [urllib3.connectionpool] https://nova.it.just.works:443 "GET /v2.1/a363b884e59e453d93b34dd70888e9c1/servers/d464ca40-cc8e-4be7-9909-899c88e41899 HTTP/1.1" 200 1680

2021-10-19 06:00:51,648 24 DEBUG [urllib3.connectionpool] Starting new HTTPS connection (1): nova.it.just.works:443

2021-10-19 06:00:52,058 24 DEBUG [urllib3.connectionpool] https://nova.it.just.works:443 "DELETE /v2.1/a363b884e59e453d93b34dd70888e9c1/servers/d464ca40-cc8e-4be7-9909-899c88e41899 HTTP/1.1" 204 0

}}}

Traceback (most recent call last):

  File "/var/lib/openstack/lib/python3.6/site-packages/tempest/api/compute/servers/test_servers_negative.py", line 623, in test_delete_a_server_of_another_tenant

    self.server_id)

  File "/var/lib/openstack/lib/python3.6/site-packages/testtools/testcase.py", line 489, in assertRaises

    self.assertThat(our_callable, matcher)

  File "/var/lib/openstack/lib/python3.6/site-packages/testtools/testcase.py", line 502, in assertThat

    raise mismatch_error

testtools.matchers._impl.MismatchError: <bound method ServersClient.delete_server of <tempest.lib.services.compute.servers_client.ServersClient object at 0x7fe5e9692898>> returned {}

Tests fail because accounts from the same project were picked from static accounts file.
Accounts are picked randomly and there is not validation which forbid to pick accounts from the same project.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/817156

[OpenStack Infra (hudson-openstack)]

Change abandoned by "mitya-eremeev-2 <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/817156
Reason: Another patch will be uploaded soon.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/819284

[Martin Kopec (mkopec)]

I can't reproduce the issue, I created accounts.yaml for concurrency 2, I pasted the accounts and tempest.conf files here as an example:
https://paste.openstack.org/show/bXfEd8U4KlnQtnpqVRJM/

I ran the following command with concurrency 2 and 4 and none of the tests failed:
$ tempest run --regex tempest.api.compute.servers

Any recommendation? Maybe I forgot something. I'd like to reproduce the issue so that I can verify and more closely examine your proposed solution.

Thanks

[Mitya Eremeev (mitos)]

Hi,

you prepared the best config do not reproduce the issue.
The title contains "preprovisioned accounts picked from the same project".
Every account is from different projects in your config file.
Please do all accounts from the same project.

[Martin Kopec (mkopec)]

Hi, thanks for the reply. I see, I generated the accounts.yaml using 'tempest account-generator' tool and it created the accounts in different projects by default.
Let me try again.

[Martin Kopec (mkopec)]

Hi Mitya,

what is the use case of having multiple accounts from the same project? Based on the tempest's documentation [1], tempest expects that all accounts with preprovisioned credentials belong to different projects:

"""
It is worth pointing out that each set of credentials in the accounts.yaml should have a unique project. This is required to provide proper isolation to the tests using the credentials, and failure to do this will likely cause unexpected failures in some tests.
""" [1]

[1] https://docs.openstack.org/tempest/latest/configuration.html#pre-provisioned-credentials

[Mitya Eremeev (mitos)]

Hi Martin,

documentation says "set of credentials", not one account.

IMHO, one project should have at least one admin and one non-admin accounts and probably some other roles.
Test classes need accounts of different roles.

[Lukas Piwowarski (lukas-piwowarski)]

I'm putting it here just for reference. This [1] bug is also caused by the fact that we select accounts randomly from the pre-provisioned credentials.

[1] https://bugs.launchpad.net/tempest/+bug/1963119

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/852748

[Martin Kopec (mkopec)]

There are unanswered comments in the related patches, i'm gonna mark this as Incomplete to reflect that.