test_container_synchronization does not work

Bug #1317133 reported by Matthew Treinish on 2014-05-07
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Undecided
Unassigned
tempest
Medium
Unassigned

Bug Description

The slow tagged test test_container_synchronization always fails. We only run the test in the periodic job periodic-tempest-dsvm-all-master which runs all tests cases including those tagged as slow.

The traceback from the failure is:

Traceback (most recent call last):
      File "tempest/api/object_storage/test_container_sync.py", line 120, in test_container_synchronization
        'Different object lists in containers.')
       File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 321, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 406, in assertThat
         raise mismatch_error
     MismatchError: !=:
    reference = {u'TestSyncObject-1761008986': {u'bytes': 25,
                                     u'content_type': u'application/json',
                                    u'hash': u'329ffccc7bb9bd0de1a190ebf0401d0c',
                                    u'last_modified': u'2014-05-07T06:52:01.624750',
                                    u'name': u'TestSyncObject-1761008986'}}
    actual = {u'TestSyncObject-1274184654': {u'bytes': 25,
                                     u'content_type': u'application/json',
                                     u'hash': u'5e9bd6e512bbf796a7f41cd81d0e3bba',
                                    u'last_modified': u'2014-05-07T06:52:01.691010',
                                     u'name': u'TestSyncObject-1274184654'}}
  : Different object lists in containers.

The logs from a recent failure are here:

http://logs.openstack.org/periodic-qa/periodic-tempest-dsvm-all-master/520a2c8/

Matthew Treinish (treinish) wrote :

I added swift to the bug, because I'm not sure whether it's just an issue with the test case, or it's exposing an underlying swift issue.

Reviewed: https://review.openstack.org/92606
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=56c1a55d269bc48d16da1d7350f616b479c715ff
Submitter: Jenkins
Branch: master

commit 56c1a55d269bc48d16da1d7350f616b479c715ff
Author: Matthew Treinish <email address hidden>
Date: Wed May 7 10:59:15 2014 -0400

    Add bug skip for test_container_synchronization()

    This commit adds a skip to test_container_synchronization(). This test
    currently only gets run during the periodic -all job because of it's
    slow tag. It is currently failing 100% of the time and has been in this
    state for quite some time. So let's skip it until the underlying bug is
    fixed.

    Related-Bug: #1317133

    Change-Id: Iff71b74a377cf80be21516c546031e784a61f836

Changed in tempest:
status: Triaged → Fix Released
Matt Riedemann (mriedem) wrote :

Looks like this can still fail with a 401 from keystone:

http://logs.openstack.org/79/247579/1/check/gate-tempest-dsvm-full-ceph-src-glance_store/3033331/console.html#_2015-11-19_20_01_06_679

2015-11-19 20:01:06.412 | tempest.api.object_storage.test_container_sync_middleware.ContainerSyncMiddlewareTest.test_container_synchronization[id-ea4645a1-d147-4976-82f7-e5a7a3065f80,slow]
2015-11-19 20:01:06.412 | ------------------------------------------------------------------------------------------------------------------------------------------------------------------
2015-11-19 20:01:06.412 |
2015-11-19 20:01:06.412 | Captured traceback:
2015-11-19 20:01:06.412 | ~~~~~~~~~~~~~~~~~~~
2015-11-19 20:01:06.412 | Traceback (most recent call last):
2015-11-19 20:01:06.412 | File "tempest/test.py", line 172, in wrapper
2015-11-19 20:01:06.412 | return func(*func_args, **func_kwargs)
2015-11-19 20:01:06.412 | File "tempest/api/object_storage/test_container_sync_middleware.py", line 52, in test_container_synchronization
2015-11-19 20:01:06.412 | self._test_container_synchronization(make_headers)
2015-11-19 20:01:06.412 | File "tempest/api/object_storage/test_container_sync.py", line 110, in _test_container_synchronization
2015-11-19 20:01:06.413 | cont, params=params)
2015-11-19 20:01:06.413 | File "tempest/services/object_storage/container_client.py", line 160, in list_container_contents
2015-11-19 20:01:06.413 | resp, body = self.get(url, headers={})
2015-11-19 20:01:06.413 | File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 274, in get
2015-11-19 20:01:06.413 | return self.request('GET', url, extra_headers, headers)
2015-11-19 20:01:06.413 | File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 639, in request
2015-11-19 20:01:06.413 | resp, resp_body)
2015-11-19 20:01:06.413 | File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 682, in _error_checker
2015-11-19 20:01:06.413 | raise exceptions.Unauthorized(resp_body, resp=resp)
2015-11-19 20:01:06.413 | tempest_lib.exceptions.Unauthorized: Unauthorized
2015-11-19 20:01:06.413 | Details: <html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22Response%20-%20Headers:%20%7B'status':%20'401'%5C%22%20AND%20message:%5C%22'Swift%20realm%5C%22%20AND%20tags:%5C%22console%5C%22

Only 3 hits in 7 days though, so not a high percentage failure.

Matthew Treinish (treinish) wrote :

Matt:

The test from this bug is not the test from that traceback. If you look at the test name it's the test_container_sync_middleware test, which the test from this bug is test_container_sync. The middleware test should be skipped because it's tagged slow, the job definition was broken, it's fixed here: https://review.openstack.org/247832

Changed in swift:
status: New → Incomplete
Alistair Coles (alistair-coles) wrote :

The test log link has expired so there's not sufficient information to know how to proceed with this bug. The way the test is written it is possible that container sync daemon had not completed sync updates before the test loop ran out of attempts and gave up checking.

Swift does have probe tests for container sync that are passing on a third party CI at the time of writing, for example here:

https://review.openstack.org/#/c/499634/

Reviewed: https://review.opendev.org/737452
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=173c5c0009b1afdd8f59d55eed5ccf0b5b844f5a
Submitter: Zuul
Branch: master

commit 173c5c0009b1afdd8f59d55eed5ccf0b5b844f5a
Author: Martin Kopec <email address hidden>
Date: Tue Jun 23 07:46:00 2020 +0000

    Unstable test_container_synchronization

    The bug which was the reason for skipping the test is considered
    resolved. To make sure the test works, let's mark it unstable
    for a while.

    Change-Id: I7f02d82f5758bc2e32b06234652f755512a74787
    Related-Bug: #1317133

Walt Boring (walter-boring) wrote :

Added this review as well, which is related to the same issue with another test that uses the same underlying code.

https://review.opendev.org/#/c/741961/

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers