tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_{end_marker,limit_and_end_marker,marker} tests failure in periodic jobs

Bug #1775553 reported by Masayuki Igawa on 2018-06-07
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tempest
Undecided
Unassigned

Bug Description

In periodic jobs, sometimes object_storage.test_account_services tests are failing.

----------
tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_end_marker
tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_limit_and_end_marker
tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_marker
tempest.api.object_storage.test_account_services.AccountTest.test_list_no_containers
----------

Here's kibana query:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20%5B'tempest-TestContainer-%5C%22%20AND%20message%3A%5C%22is%20not%20false%20%3A%20sequence%20or%20collection%20is%20not%20empty%3A%5C%22

14 times happened in the last week.

It looks there are some race conditions in these tests.

Masayuki Igawa (igawa) wrote :

Here's one example of the errors

http://paste.openstack.org/show/722861/

Masayuki Igawa (igawa) wrote :

I think the root of cause is like this.

In the failed tests, it is expected that there's no containers before the test. However, some other tests create containers at the same time.

Changed in tempest:
assignee: nobody → Manik Bindlish (manikbindlish)
Dr. Jens Harbott (j-harbott) wrote :

Looking at logstash, all of the failures are happening when using static accounts, not sure why that case is special.

Dr. Jens Harbott (j-harbott) wrote :
Download full text (4.0 KiB)

One possibly related observation:

tempest.api.object_storage.test_account_services.AccountTest uses two accounts, but with the default settings "tempest account-generator" only generates one account, causing the tempest run to fail:

setUpClass (tempest.api.object_storage.test_account_services.AccountTest)
-------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/test.py", line 172, in setUpClass
        six.reraise(etype, value, trace)
      File "tempest/test.py", line 157, in setUpClass
        cls.setup_credentials()
      File "tempest/api/object_storage/test_account_services.py", line 38, in setup_credentials
        super(AccountTest, cls).setup_credentials()
      File "tempest/api/object_storage/base.py", line 77, in setup_credentials
        super(BaseObjectTest, cls).setup_credentials()
      File "tempest/test.py", line 401, in setup_credentials
        force_new=True)
      File "tempest/test.py", line 670, in get_client_manager
        creds = cred_provider.get_creds_by_roles(**params)
      File "tempest/lib/common/preprov_creds.py", line 315, in get_creds_by_roles
        net_creds = self._get_creds(roles=roles)
      File "tempest/lib/common/preprov_creds.py", line 248, in _get_creds
        free_hash = self._get_free_hash(useable_hashes)
      File "/opt...

Read more...

Changed in tempest:
assignee: Manik Bindlish (manikbindlish) → nobody
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers