tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_{end_marker,limit_and_end_marker,marker} tests failure in periodic jobs

[Masayuki Igawa]

In periodic jobs, sometimes object_storage.test_account_services tests are failing.

----------
tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_end_marker
tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_limit_and_end_marker
tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_marker
tempest.api.object_storage.test_account_services.AccountTest.test_list_no_containers
----------

Here's kibana query:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20%5B'tempest-TestContainer-%5C%22%20AND%20message%3A%5C%22is%20not%20false%20%3A%20sequence%20or%20collection%20is%20not%20empty%3A%5C%22

14 times happened in the last week.

It looks there are some race conditions in these tests.

[Masayuki Igawa (igawa)]

Here's one example of the errors

http://paste.openstack.org/show/722861/

[Masayuki Igawa (igawa)]

I think the root of cause is like this.

In the failed tests, it is expected that there's no containers before the test. However, some other tests create containers at the same time.

[Dr. Jens Harbott (j-harbott)]

Looking at logstash, all of the failures are happening when using static accounts, not sure why that case is special.

[Dr. Jens Harbott (j-harbott)]

One possibly related observation:

tempest.api.object_storage.test_account_services.AccountTest uses two accounts, but with the default settings "tempest account-generator" only generates one account, causing the tempest run to fail:

setUpClass (tempest.api.object_storage.test_account_services.AccountTest)
-------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/test.py", line 172, in setUpClass
        six.reraise(etype, value, trace)
      File "tempest/test.py", line 157, in setUpClass
        cls.setup_credentials()
      File "tempest/api/object_storage/test_account_services.py", line 38, in setup_credentials
        super(AccountTest, cls).setup_credentials()
      File "tempest/api/object_storage/base.py", line 77, in setup_credentials
        super(BaseObjectTest, cls).setup_credentials()
      File "tempest/test.py", line 401, in setup_credentials
        force_new=True)
      File "tempest/test.py", line 670, in get_client_manager
        creds = cred_provider.get_creds_by_roles(**params)
      File "tempest/lib/common/preprov_creds.py", line 315, in get_creds_by_roles
        net_creds = self._get_creds(roles=roles)
      File "tempest/lib/common/preprov_creds.py", line 248, in _get_creds
        free_hash = self._get_free_hash(useable_hashes)
      File "/opt...

[Martin Kopec (mkopec)]

Is the issue still happening?

[Masayuki Igawa (igawa)]

It looks no error has happened against AccountTest.test_list_containers_with_end_marker test for these 6 months at least.

http://status.openstack.org/openstack-health/#/test/tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_end_marker?resolutionKey=day&duration=P6M

[Martin Kopec (mkopec)]

Ok then, it seems it got fixed somehow. I'm gonna mark this as Invalid now, but if the issue happens again, feel free to reopen it.