test_container_synchronization does not work

[Matthew Treinish]

The slow tagged test test_container_synchronization always fails. We only run the test in the periodic job periodic-tempest-dsvm-all-master which runs all tests cases including those tagged as slow.

The traceback from the failure is:

Traceback (most recent call last):
      File "tempest/api/object_storage/test_container_sync.py", line 120, in test_container_synchronization
        'Different object lists in containers.')
       File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 321, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 406, in assertThat
         raise mismatch_error
     MismatchError: !=:
    reference = {u'TestSyncObject-1761008986': {u'bytes': 25,
                                     u'content_type': u'application/json',
                                    u'hash': u'329ffccc7bb9bd0de1a190ebf0401d0c',
                                    u'last_modified': u'2014-05-07T06:52:01.624750',
                                    u'name': u'TestSyncObject-1761008986'}}
    actual = {u'TestSyncObject-1274184654': {u'bytes': 25,
                                     u'content_type': u'application/json',
                                     u'hash': u'5e9bd6e512bbf796a7f41cd81d0e3bba',
                                    u'last_modified': u'2014-05-07T06:52:01.691010',
                                     u'name': u'TestSyncObject-1274184654'}}
  : Different object lists in containers.

The logs from a recent failure are here:

http://logs.openstack.org/periodic-qa/periodic-tempest-dsvm-all-master/520a2c8/

[Matthew Treinish (treinish)]

I added swift to the bug, because I'm not sure whether it's just an issue with the test case, or it's exposing an underlying swift issue.

[Openstack Gerrit (openstack-gerrit)]

Related fix proposed to branch: master
Review: https://review.openstack.org/92606

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92606
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=56c1a55d269bc48d16da1d7350f616b479c715ff
Submitter: Jenkins
Branch: master

commit 56c1a55d269bc48d16da1d7350f616b479c715ff
Author: Matthew Treinish <email address hidden>
Date: Wed May 7 10:59:15 2014 -0400

    Add bug skip for test_container_synchronization()

    This commit adds a skip to test_container_synchronization(). This test
    currently only gets run during the periodic -all job because of it's
    slow tag. It is currently failing 100% of the time and has been in this
    state for quite some time. So let's skip it until the underlying bug is
    fixed.

    Related-Bug: #1317133

    Change-Id: Iff71b74a377cf80be21516c546031e784a61f836

[Matt Riedemann (mriedem)]

Looks like this can still fail with a 401 from keystone:

http://logs.openstack.org/79/247579/1/check/gate-tempest-dsvm-full-ceph-src-glance_store/3033331/console.html#_2015-11-19_20_01_06_679

2015-11-19 20:01:06.412 | tempest.api.object_storage.test_container_sync_middleware.ContainerSyncMiddlewareTest.test_container_synchronization[id-ea4645a1-d147-4976-82f7-e5a7a3065f80,slow]
2015-11-19 20:01:06.412 | ------------------------------------------------------------------------------------------------------------------------------------------------------------------
2015-11-19 20:01:06.412 |
2015-11-19 20:01:06.412 | Captured traceback:
2015-11-19 20:01:06.412 | ~~~~~~~~~~~~~~~~~~~
2015-11-19 20:01:06.412 | Traceback (most recent call last):
2015-11-19 20:01:06.412 | File "tempest/test.py", line 172, in wrapper
2015-11-19 20:01:06.412 | return func(*func_args, **func_kwargs)
2015-11-19 20:01:06.412 | File "tempest/api/object_storage/test_container_sync_middleware.py", line 52, in test_container_synchronization
2015-11-19 20:01:06.412 | self._test_container_synchronization(make_headers)
2015-11-19 20:01:06.412 | File "tempest/api/object_storage/test_container_sync.py", line 110, in _test_container_synchronization
2015-11-19 20:01:06.413 | cont, params=params)
2015-11-19 20:01:06.413 | File "tempest/services/object_storage/container_client.py", line 160, in list_container_contents
2015-11-19 20:01:06.413 | resp, body = self.get(url, headers={})
2015-11-19 20:01:06.413 | File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 274, in get
2015-11-19 20:01:06.413 | return self.request('GET', url, extra_headers, headers)
2015-11-19 20:01:06.413 | File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 639, in request
2015-11-19 20:01:06.413 | resp, resp_body)
2015-11-19 20:01:06.413 | File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 682, in _error_checker
2015-11-19 20:01:06.413 | raise exceptions.Unauthorized(resp_body, resp=resp)
2015-11-19 20:01:06.413 | tempest_lib.exceptions.Unauthorized: Unauthorized
2015-11-19 20:01:06.413 | Details: <html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22Response%20-%20Headers:%20%7B'status':%20'401'%5C%22%20AND%20message:%5C%22'Swift%20realm%5C%22%20AND%20tags:%5C%22console%5C%22

Only 3 hits in 7 days though, so not a high percentage failure.

[Matthew Treinish (treinish)]

Matt:

The test from this bug is not the test from that traceback. If you look at the test name it's the test_container_sync_middleware test, which the test from this bug is test_container_sync. The middleware test should be skipped because it's tagged slow, the job definition was broken, it's fixed here: https://review.openstack.org/247832

[Alistair Coles (alistair-coles)]

The test log link has expired so there's not sufficient information to know how to proceed with this bug. The way the test is written it is possible that container sync daemon had not completed sync updates before the test loop ran out of attempts and gave up checking.

Swift does have probe tests for container sync that are passing on a third party CI at the time of writing, for example here:

https://review.openstack.org/#/c/499634/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/737452

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737452
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=173c5c0009b1afdd8f59d55eed5ccf0b5b844f5a
Submitter: Zuul
Branch: master

commit 173c5c0009b1afdd8f59d55eed5ccf0b5b844f5a
Author: Martin Kopec <email address hidden>
Date: Tue Jun 23 07:46:00 2020 +0000

    Unstable test_container_synchronization

    The bug which was the reason for skipping the test is considered
    resolved. To make sure the test works, let's mark it unstable
    for a while.

    Change-Id: I7f02d82f5758bc2e32b06234652f755512a74787
    Related-Bug: #1317133

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/741961

[Walt Boring (walter-boring)]

Added this review as well, which is related to the same issue with another test that uses the same underlying code.

https://review.opendev.org/#/c/741961/

[Martin Kopec (mkopec)]

Hi Walt,

what's the reason behind 741961? Are there any issues with the test? Looking at the commit message of https://review.opendev.org/c/openstack/tempest/+/737452 marking the test unstable is just temporary which reminds me, we need to check if it's the time to remove the unstable decorator now.

[Ghanshyam Mann (ghanshyammann)]

I am enabling the swift in tempest-slow-py3 jobs also so that we can run test test_container_synchronization
 in gate. And you can see the result there

- test_container_sync.ContainerSyncTest.test_container_synchronization fail with same bug

- test_container_sync_middleware.ContainerSyncMiddlewareTest.test_container_synchronization [75.923537s] ... ok

Logs: https://zuul.opendev.org/t/openstack/build/d95585963a79431b966d9d09749f7b19/log/job-output.txt#64123

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/849801

[Martin Kopec (mkopec)]

Seems this is still an issue, the test fails consistently.

[Roman Popelka (rpopelka1)]

This bug still isn't fixed and fails with the same issue as before. We've scanned the jobs in which the test appears and it's pass rate is 0%.

The bug will be reopened and the test will be skipped until the bug is fixed.

Here are some jobs (and logs) in which the featured test appears:
- tempest-slow-py3[1]
- cinder-tempest-plugin-lvm-tgt-barbican[2]
- devstack-plugin-ceph-tempest-py3[3]
- glance-ceph-thin-provisioning[4]
- tempest-slow-yoga[5]
- tempest-all[6]

[1] https://c0d0d8569e9d2100fdac-f465d94e970ed4b88220a4cc8527bc2b.ssl.cf1.rackcdn.com/848418/2/check/tempest-slow-py3/222f0c5/job-output.txt
[2] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_68a/850526/1/check/cinder-tempest-plugin-lvm-tgt-barbican/68a216e/job-output.txt
[3] https://ad6d2f4d612b4ce1fc0e-32acf2150fe2d2cb4d1cafb2a4fa127d.ssl.cf5.rackcdn.com/850311/7/check/devstack-plugin-ceph-tempest-py3/10627a7/job-output.txt
[4] https://9252355b19e6b9b2ecd6-c7c5829e17049c4cf47426e53829e931.ssl.cf2.rackcdn.com/828232/4/check/glance-ceph-thin-provisioning/5a2b969/job-output.txt
[5] https://26b69907c0680b5b898b-a7f43026294a53c04676a5990bff48a5.ssl.cf1.rackcdn.com/periodic-stable/opendev.org/openstack/tempest/master/tempest-slow-yoga/68df124/job-output.txt
[6] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_59c/periodic/opendev.org/openstack/tempest/master/tempest-all/59c268e/job-output.txt

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/849801
Committed: https://opendev.org/openstack/tempest/commit/48367f066965bcb7b866ddc2581fc6fbd5ed26ab
Submitter: "Zuul (22348)"
Branch: master

commit 48367f066965bcb7b866ddc2581fc6fbd5ed26ab
Author: Roman Popelka <email address hidden>
Date: Thu Jul 14 12:09:38 2022 +0200

    Skip test_container_synchronization

    Replace unstable_test decorator with skip due to
    low pass rate of this test. The test will be unskipped
    when the related bug is fixed.

    Related-Bug: #1317133
    Change-Id: I93613f4d89629c6dcfe72c510bced042e257695d

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/741961
Reason: no activities for a long time, feel free to restore it if needed.

[Martin Kopec (mkopec)]

I'll mark this as Won't Fix, at least for now. This is an issue in swift - we're also under the impression that users are discouraged from using the synchronization in most of the cases. Right now this bug is blocked by this anyway - https://bugs.launchpad.net/swift/+bug/2028952

If you feel otherwise, feel free to reopen it.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Object Storage (swift) because there has been no activity for 60 days.]