test_object_delete_at_aysnc_update occasionally fails

Bug #1514111 reported by John Dickinson
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Low
Unassigned

Bug Description

object server unit tests occasionally fail with the following error

======================================================================
FAIL: test_object_delete_at_aysnc_update (test.unit.obj.test_server.TestObjectController)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/swift/swift/test/unit/obj/test_server.py", line 3430, in test_object_delete_at_aysnc_update
    self.assertRaises(StopIteration, fake_conn.code_iter.next)
AssertionError: StopIteration not raised
-------------------- >> begin captured stdout << ---------------------
test ERROR: ERROR Container update failed (saving for async update later): 500 response from 10.0.0.2:6002/sda1
test DEBUG: Container update timeout (0.0000s) waiting for [('10.0.0.1:6001', 'sda1')]
test INFO: None - - [06/Nov/2015:21:57:35 +0000] "PUT /sda1/p/a/c/o" 201 - "-" "-" "-" 0.0167 "-" 12123 1
test ERROR: ERROR container update failed with 10.0.0.1:6001/sda1 (saving for async update later): ConnectionTimeout (0.5s)

--------------------- >> end captured stdout << ----------------------

I've been able to duplicate this by running the tests directly within a loop:

while [[ $? = 0 ]]; do nosetests test_server.py:TestObjectController.test_object_delete_at_aysnc_update; done

Revision history for this message
Sivasathurappan Radhakrishnan (siva-radhakrishnan) wrote :

Was able to replicate the issue.

Changed in swift:
status: New → Confirmed
Revision history for this message
Yatin Kumbhare (yatinkumbhare-c) wrote :

Out of ten runs of nearly 500 iterations, I was able to check this only once. Hard to get this on every run.

i=0; while [[ $? = 0 ]]; do echo $i; i=$((i+1)); nosetests test_server.py:TestObjectController.test_object_delete_at_aysnc_update; done

Ben Keller (bjkeller)
Changed in swift:
assignee: nobody → Ben Keller (bjkeller)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/291368

Ben Keller (bjkeller)
Changed in swift:
assignee: Ben Keller (bjkeller) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (master)

Reviewed: https://review.openstack.org/291368
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0bc5a69d416004924bd17c14d8455f65433751bd
Submitter: Jenkins
Branch: master

commit 0bc5a69d416004924bd17c14d8455f65433751bd
Author: Alistair Coles <email address hidden>
Date: Thu Mar 10 18:02:44 2016 +0000

    Wait on greenthreads before unmocking http_connect

    The fake_spawn() context manager wraps spawn and waits
    for greenthreads to complete (such as the async_update threads).
    The wait needs to be done before http_connect is un-mocked, so
    the fake_spawn context manager should exit *before* any context
    manager that mocks the http_connect method.

    Also add fake_spawn to _test_PUT_then_POST_async_pendings()

    Related-Bug: #1536376
    Related-Bug: #1514111
    Closes-Bug: #1555739

    Change-Id: I15f36e191cfe3ee6c82b4be56e8618ec0230e328

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (feature/crypto)

Related fix proposed to branch: feature/crypto
Review: https://review.openstack.org/299944

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (feature/crypto)
Download full text (37.8 KiB)

Reviewed: https://review.openstack.org/299944
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=77c181161a029ba8cea5d1ce65f979fe4b23ff37
Submitter: Jenkins
Branch: feature/crypto

commit 59bbe27fb0a40236108f09c9b3349e8faef0a95c
Author: Nguyen Hung Phuong <email address hidden>
Date: Wed Mar 30 11:07:46 2016 +0700

    Fix typos in Swift files

    Change-Id: I34e0c9a888127704ac1910e73ddd14e27ebade13

commit 7be55acf1bc4aa07d81b30fd93e144700889898d
Author: Tim Burke <email address hidden>
Date: Thu Feb 11 16:00:38 2016 -0800

    Simplify policy-name validation slightly

    _validate_policy_name always either returns True or raises an exception.
    Simplify it to just being a callable that may raise an exception.

    Also, move the check for blank/None names into _validate_policy_name, so
    it will be applied in more cases.

    Change-Id: I7832a0c9c895cd75ba4c6d0e8b5568a3c8a0ea25

commit 5902015fa8495ec0ef3c1ab92ae9a34c5bda4334
Author: OpenStack Proposal Bot <email address hidden>
Date: Sat Mar 26 06:35:18 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I3b5d401649fa3dea6dc43654516f7075bb06ee0d

commit 2f7d0f4a2ad2da7e6a35e5b054a47a2fafe5ed01
Author: Anh Tran <email address hidden>
Date: Fri Mar 25 11:44:26 2016 +0700

    Removing some redundant words

    This patch removes some redundant words.

    Change-Id: Ia79717664b06ed9a41c3c5dcf1a25e9e49e21cf2

commit 925546ae8a211b50cf7fad6634d47fd1dbfeb58e
Author: OpenStack Proposal Bot <email address hidden>
Date: Fri Mar 25 06:36:40 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I6ba2f35913e6ae83607b5e268645432d455d587c

commit 3407d737c705a7afedeed0159588ab4433a601f3
Author: David Liu <email address hidden>
Date: Thu Mar 24 16:08:19 2016 +0800

    Handle tempurl Content-Disposition header missing from HEAD

    Content-Disposition headers should make no difference between
    GET and HEAD according to HTTP rfc.

    Closes-Bug: #1539805

    Change-Id: Ifa41a7cda2f321eb8e36420ede7912ed0a549712

commit 2f24fb9683a57b67348d65864d5af8c3a03dee67
Author: Alistair Coles <email address hidden>
Date: Wed Mar 23 20:49:50 2016 +0000

    Check marker params in SimpleClient full listing requests

    Follow up for change [1] to add some assertions to check that
    marker param is included in sequential GET requests sent during
    a full listing.

    Extract multiple FakeConn class definitions to single class at
    module level and share between all classes.

    Also, explicitly unpack the return values from base request calls
    made in the full listing section of base_request, and explicitly
    return a list to make more consistent with rest of the method.

    [1] Change-Id: I6892390d72f70f1bc519b482d4f72603e1570163

    Change-Id: Iad038709f46364b8324d25ac79be4317add79df...

tags: added: in-feature-crypto
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (feature/hummingbird)

Related fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/323599

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (feature/hummingbird)
Download full text (84.7 KiB)

Reviewed: https://review.openstack.org/323599
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0330478b70d0a699a0f9c21ef87c7e639d92564b
Submitter: Jenkins
Branch: feature/hummingbird

commit 5fe392b562de3baed080704df433fb392cb4fb31
Author: Ondřej Nový <email address hidden>
Date: Tue May 31 16:25:50 2016 +0200

    Fixed typo

    Change-Id: I7a35c0076360c7a23cf405189828d3c252ec6708

commit b52eccb3b1ea0591f0040587228d3705b5d3f68d
Author: Clay Gerrard <email address hidden>
Date: Wed May 25 11:21:25 2016 -0700

    Clarify overload best practices in admin guide

    Change-Id: Ib7c08bdeab6374771bb8e2b05053e7e16973524d

commit f1fd50723bb84c4941e949895576733f6eb67793
Author: Christian Schwede <email address hidden>
Date: Wed May 25 09:53:31 2016 +0200

    Add dispersion --verbose example to admin guide

    Change-Id: I5f9cacedde2a329332ccf744800b6f2453e8b28e

commit b3ab715c055283ccfea9a504d6da20741d82e7ad
Author: Matthew Oliver <email address hidden>
Date: Wed May 25 14:35:54 2016 +1000

    Add ring-builder dispersion command to admin guide

    This change updates the admin guide to point out the dispersion command
    in swift-ring-builder and mentions the dispersion verbose table to make
    it more obvious to operators.

    Change-Id: I72b4c8b2d718e6063de0fdabbaf4f2b73694e0a4

commit fb7a8e9ab7596a36a6992a3a8f8c6d005a2c2829
Author: Tim Burke <email address hidden>
Date: Tue May 24 13:37:58 2016 -0700

    Add links to mitaka install guides

    Change-Id: I62331923751c521daded4468b5cc5f03655226bc

commit e09c4ee7800e82aa09ca2f6ae375420b766182a4
Author: Tim Burke <email address hidden>
Date: Fri Apr 29 12:12:00 2016 -0500

    Allow concurrent bulk deletes

    Before, server-side deletes of static large objects could take a long
    time to complete since the proxy would wait for a response to each
    segment DELETE before starting the next DELETE request.

    Now, operators can configure a concurrency factor for the slo and bulk
    middlewares to allow up to N concurrent DELETE requests. By default, two
    DELETE requests will be allowed at a time.

    Note that objects and containers are now deleted in separate passes, to
    reduce the likelihood of 409 Conflict responses when deleting
    containers.

    Upgrade Consideration
    =====================
    If operators have enabled the bulk or slo middlewares and would like to
    preserve the prior (single-threaded) DELETE behavior, they must add the
    following line to their [filter:slo] and [filter:bulk] proxy config
    sections:

       delete_concurrency = 1

    This may be done prior to upgrading Swift.

    UpgradeImpact
    Closes-Bug: 1524454
    Change-Id: I128374d74a4cef7a479b221fd15eec785cc4694a

commit 226557afc42c245e050d84162497f46341407ef7
Author: Tim Burke <email address hidden>
Date: Thu May 19 18:55:40 2016 -0700

    Turn on H703, so our translators don't punch us

    Change-Id: I4ce3068f79563e4d4296c6e1078bc12f0cf84c96
    Related-Bug: 1559431

commit 7b706926a8ed5bbcec3a678e868e301c9a6ed8f1
Author: Alistair Coles <email address hidden>
Date: Mon May ...

tags: added: in-feature-hummingbird
Revision history for this message
Tim Burke (1-tim-z) wrote :

I'm fairly certain Alistair's change fixed this particular issue, so I'm going to mark it as fixed.

However, I still see this test sporadically failing:

======================================================================
FAIL: test_object_delete_at_async_update (test.unit.obj.test_server.TestObjectController)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/vagrant/swift/test/unit/obj/test_server.py", line 4826, in test_object_delete_at_async_update
    resp = req.get_response(self.object_controller)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/vagrant/swift/test/unit/__init__.py", line 1075, in mocked_http_conn
    raise AssertionError('left over status %r' % left_over_status)
AssertionError: left over status [500, 500]
-------------------- >> begin captured stdout << ---------------------
test INFO: None - - [26/Apr/2017:22:32:13 +0000] "PUT /sda1/p/a/c/o" 400 19 "-" "-" "-" 0.0003 "-" 23801 0

--------------------- >> end captured stdout << ----------------------
>> raise AssertionError('left over status %r' % [500, 500])

----------------------------------------------------------------------

...apparently because we return a 400 with a body of "X-Delete-At in past". About to post a fix for that.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/460341

Tim Burke (1-tim-z)
Changed in swift:
status: Confirmed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (master)

Reviewed: https://review.openstack.org/460341
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=20072570d9a58416fbd30a71cf0719c1db3bb686
Submitter: Jenkins
Branch: master

commit 20072570d9a58416fbd30a71cf0719c1db3bb686
Author: Tim Burke <email address hidden>
Date: Wed Apr 26 15:50:59 2017 -0700

    Fix sporadic failure in test/unit/obj/test_server.py

    In particular, in TestObjectController.test_object_delete_at_async_update

    Rarely (<0.1% of the time?), it would fail with:

    ======================================================================
    FAIL: test_object_delete_at_async_update
    (test.unit.obj.test_server.TestObjectController)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/vagrant/swift/test/unit/obj/test_server.py", line 4826, in
    test_object_delete_at_async_update
        resp = req.get_response(self.object_controller)
      File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
        self.gen.next()
      File "/vagrant/swift/test/unit/__init__.py", line 1075, in
    mocked_http_conn
        raise AssertionError('left over status %r' % left_over_status)
    AssertionError: left over status [500, 500]
    -------------------- >> begin captured stdout << ---------------------
    test INFO: None - - [26/Apr/2017:22:32:13 +0000] "PUT /sda1/p/a/c/o" 400
    19 "-" "-" "-" 0.0003 "-" 23801 0

    --------------------- >> end captured stdout << ----------------------
    >> raise AssertionError('left over status %r' % [500, 500])

    ----------------------------------------------------------------------

    Related-Bug: 1514111
    Change-Id: I1af4a291fb67cf4b1829f167998a08644117a800

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.