tempest.api.object_storage.test_account_services mismatch failure

[Martin Kopec]

tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_end_marker[id-5ca164e4-7bde-43fa-bafb-913b53b9e786]
tempest.api.object_storage.test_account_services.AccountTest.test_list_no_containers[id-884ec421-fbad-4fcc-916b-0580f2699565]

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/api/object_storage/test_account_services.py", line 188, in test_list_containers_with_end_marker
    self.assertEmpty(container_list)
  File "/opt/stack/tempest/tempest/test.py", line 912, in assertEmpty
    self.assertFalse(items, msg)
  File "/usr/lib/python3.8/unittest/case.py", line 759, in assertFalse
    raise self.failureException(msg)
AssertionError: [b'tempest-TestContainer-1002387681', b'tempest-TestContainer-1808953131'] is not false : sequence or collection is not empty: [b'tempest-TestContainer-1002387681', b'tempest-TestContainer-1808953131']

The above tests are failing in tempest-full-test-account-no-admin-py3 job:
https://zuul.opendev.org/t/openstack/builds?job_name=tempest-full-test-account-no-admin-py3&project=openstack/tempest
See e.g.:
https://3f1311e6ef6e59fbb50d-639cee2bb13a4a48aa71c9492956441b.ssl.cf2.rackcdn.com/857756/8/check/tempest-full-test-account-no-admin-py3/3db407a/testr_results.html

tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_limit_and_end_marker[id-888a3f0e-7214-4806-8e50-5e0c9a69bb5e]

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/api/object_storage/test_account_services.py", line 241, in test_list_containers_with_limit_and_end_marker
    self.assertEqual(len(container_list),
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/testtools/testcase.py", line 393, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/testtools/testcase.py", line 480, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 5 != 3

https://bd90009aa1732b7b8d4a-e998c5625939f617052baaae6f827bb8.ssl.cf5.rackcdn.com/857756/8/check/tempest-full-test-account-py3/2a71be6/testr_results.html

Also the following test fails from time to time in the same tempest-full-test-account-py3 job:

tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_list_application_credentials[id-ff0cd457-6224-46e7-b79e-0ada4964a8a6]

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/api/identity/v3/test_application_credentials.py", line 96, in test_list_application_credentials
    self.assertEqual(2, len(app_creds))
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/testtools/testcase.py", line 393, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/testtools/testcase.py", line 480, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 2 != 4

[Lukas Piwowarski (lukas-piwowarski)]

I'm moving this to confirmed as there are other instances of unexpected failure of tests in tempest.api.object_storage.test_account_services in the tempest-full-test-account-py3 job [1] and tempest-full-test-account-no-admin-py3 job [1].

(It is possible to check status of the test in "Subunit - Test results" dashboard here [3] -- username: openstack, password: openstack).

[1] https://zuul.opendev.org/t/openstack/build/c09478b0615445309be93e07ef00fbde
[2] https://zuul.opendev.org/t/openstack/build/90c0789c6bb54c14aa43366013c4e797/logs
[3] https://opensearch.logs.openstack.org

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/880630

[Martin Kopec (mkopec)]

Update:
Most of the tests mentioned in this LP pass now.

2 fail occasionally though:
tempest-full-test-account-no-admin-py3 job

tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_list_application_credentials[id-ff0cd457-6224-46e7-b79e-0ada4964a8a6]
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/api/identity/v3/test_application_credentials.py", line 96, in test_list_application_credentials
    self.assertEqual(2, len(app_creds))
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/testtools/testcase.py", line 394, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/testtools/testcase.py", line 481, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 2 != 4

tempest.api.object_storage.test_account_services.AccountTest.test_list_no_containers[id-884ec421-fbad-4fcc-916b-0580f2699565]
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/api/object_storage/test_account_services.py", line 100, in test_list_no_containers
    self.assertEmpty(container_list)
  File "/opt/stack/tempest/tempest/test.py", line 959, in assertEmpty
    self.assertFalse(items, msg)
  File "/usr/lib/python3.10/unittest/case.py", line 681, in assertFalse
    raise self.failureException(msg)
AssertionError: [b'volumebackups'] is not false : sequence or collection is not empty: [b'volumebackups']

[Lukas Piwowarski (lukas-piwowarski)]

The second failure mentioned in Martin's comment is about to be fixed by this patch => https://review.opendev.org/c/openstack/tempest/+/890798

[Martin Kopec (mkopec)]

speaking about tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_list_application_credentials

there are 2 test classes that create application_credentials.

tempest/api/identity/v3/test_application_credentials.py
tempest/api/identity/v3/test_access_rules.py

As the failure is that there are more application credentials than expected in the assert, the logical assumption would be that there is a race condition between those 2 classes - the race condition is amplified by the fact that we run with accounts.yaml = there is a possibility both tests use the same user and thus the resource conflict

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/892663

[Martin Kopec (mkopec)]

tempest.api.object_storage.test_account_services.AccountTest.test_list_no_containers test might get fixed by https://bugs.launchpad.net/tempest/+bug/2028671 -> https://review.opendev.org/c/openstack/tempest/+/890798

[Lukas Piwowarski (lukas-piwowarski)]

Here are steps to reproduce the failure of tempest.api.object_storage.test_account_services.AccountTest.test_list_no_containers:

tempest account-generator -r 1 accounts.yaml
# Modify tempest.conf (use_dynamic_credentials, test_accounts_file)
tempest run --concurrency 1 --regex "tempest.api.identity.v3."

The issue is probably in a clean-up in one of the tests above the failing test (see list_credentials_failure.txt).

[Martin Kopec (mkopec)]

tempest.api.object_storage.test_account_services.AccountTest.test_list_no_containers[id-884ec421-fbad-4fcc-916b-0580f2699565] got fixed by https://review.opendev.org/c/openstack/tempest/+/890798

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/892663
Committed: https://opendev.org/openstack/tempest/commit/347ef034590449426f0e9fd5fcc96a008e2cf95b
Submitter: "Zuul (22348)"
Branch: master

commit 347ef034590449426f0e9fd5fcc96a008e2cf95b
Author: Martin Kopec <email address hidden>
Date: Thu Aug 24 12:36:43 2023 +0200

    Fix clean up for test_access_rules tests

    In the test_access_rules tests [1], the app credentials were
    created within setUp which means they were created per each test
    while they weren't properly cleaned up after the tests. That led
    to having those creds left behind which caused the assert issue
    in the LP 1996624

    This patch moves the app creds creation to the resource_setup so
    that it's a class level resource and thus not created per each
    test case separately. The test which tests deleting access rule
    will create its own app cred together with a new access rule and
    intead of asserting on number of found access rules it will check
    existence of the access rule which was created in the test. That
    will help us avoid any race conditions.

    [1] https://opendev.org/openstack/tempest/src/commit/b2fefaf0b56ce96f6116f89faf79458ef4f9c7f3/tempest/api/identity/v3/test_access_rules.py#L51

    Related-Bug: #1996624
    Change-Id: Ibdb40910e53b3b1d5f481d101dfcf1b8e04dc63f

[Martin Kopec (mkopec)]

All issues mentioned in this LP have been fixed.