SSYNC subrequests does not log the body of the error

Bug #1821204 reported by Romain LE DISEZ
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
New
Undecided
Unassigned

Bug Description

In case of SSYNC replication error, the body of the error is not logged, which does not help to diagnose the issue. Example:

object-server: ssync subrequest failed with 500: POST /disk-00-001/34159/AUTH_***/***/***

It should log body to provid emore informations. In this case:

object-server: ssync subrequest failed with 500: POST /disk-00-001/34159/AUTH_***/***/*** (Traceback (most recent call last):#012 File "/opt/swift-2.18.0-ovh398/local/lib/python2.7/site-packages/swift/obj/server.py", line 1242, in __call__#012 res = getattr(self, req.method)(req)#012 File "/opt/swift-2.18.0-ovh398/local/lib/python2.7/site-packages/swift/common/utils.py", line 1796, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/opt/swift-2.18.0-ovh398/local/lib/python2.7/site-packages/swift/obj/server.py", line 602, in POST#012 if new_delete_at and new_delete_at < req_timestamp:#012 File "/usr/lib/python2.7/functools.py", line 56, in <lambda>#012 '__lt__': [('__gt__', lambda self, other: not (self < other or self == other)),#012 File "/opt/swift-2.18.0-ovh398/local/lib/python2.7/site-packages/swift/common/utils.py", line 1180, in __lt__#012 other = Timestamp(other)#012 File "/opt/swift-2.18.0-ovh398/local/lib/python2.7/site-packages/swift/common/utils.py", line 1092, in __init__#012 raise ValueError('timestamp too large')#012ValueError: timestamp too large#012)

Revision history for this message
clayg (clay-gerrard) wrote :

Just to be clear you're thinking the ssync_reciever might be able to grab the subrequest response error and log it in the source replicator/reconstructor logs?

Revision history for this message
Romain LE DISEZ (rledisez) wrote :

I was thinking that the ssync_receiver should grab the error and log it in its own log file. It's not perfect but easy to do. It already logs the HTTP code and method, so it's just about adding resp.body [1]

Having the error on the sender side would be nice, but it's more complex and require some refactoring (I've been working on something related recently to fix [2])

[1] https://github.com/openstack/swift/blob/fb0e7837af119dfb6bd30627ac8f9f4eb5a3e890/swift/obj/ssync_receiver.py#L478-L480

[2] https://bugs.launchpad.net/swift/+bug/1821346

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (master)

Reviewed: https://review.opendev.org/702950
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=57ca3570e911bd290ce0e44bdebaa4b5b4ceca13
Submitter: Zuul
Branch: master

commit 57ca3570e911bd290ce0e44bdebaa4b5b4ceca13
Author: Tim Burke <email address hidden>
Date: Thu Jan 16 10:07:33 2020 -0800

    Allow Timestamp comparisons against out-of-range values

    Prior to the related change, clients may have written down X-Delete-At headers
    that are outside of the Timestamp range, for example.

    Change-Id: Ib8ae7ebcbdb32e0aa58446bd1ef949e5e2f63e74
    Related-Change: I23666ec8a067d829eaf9bfe54bd086c320b3429e
    Related-Bug: 1821204
    Partial-Bug: 1860149

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (feature/losf)

Related fix proposed to branch: feature/losf
Review: https://review.opendev.org/713632

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (feature/losf)
Download full text (40.5 KiB)

Reviewed: https://review.opendev.org/713632
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=79bd2e59e5c15ee84814ec1c4f0893176ba79412
Submitter: Zuul
Branch: feature/losf

commit f2ffd900593db2829a39a073f0c033d82985c40f
Author: Clay Gerrard <email address hidden>
Date: Fri Feb 28 11:09:51 2020 -0600

    Apply limit to list versioned containers

    Change-Id: I28e062273d673c4f07cd3c5da088aa790b77a599
    Closes-Bug: #1863841

commit dc40779307095b8d0b2761b77b9cb2904ec721ae
Author: Clay Gerrard <email address hidden>
Date: Fri Feb 28 10:00:25 2020 -0600

    Use float consistently for proxy timeout settings

    Change-Id: I433c97df99193ec31c863038b9b6fd20bb3705b8

commit 55049beda5b9d7038a3604a87f28312d7702ccb2
Author: Tim Burke <email address hidden>
Date: Fri Feb 28 18:59:32 2020 -0800

    tests: Use timedelta to adjust dates, not string manipulations

    Change-Id: I8f65ccd7f2a79d5b877bfbef0274fb7857e21391

commit 3b65a5998cc921d2763cf1a9ec1e40b88491262d
Author: Tim Burke <email address hidden>
Date: Wed Jan 10 06:16:41 2018 +0000

    Fix up some Content-Type handling in account/container listings

    Update content type on 204 (not just 200) to properly handle HEAD
    requests from xml/txt listings.

    Add "Vary: Accept" header to listings, since otherwise, browsers may
    serve the wrong content type from cache (even though we *would have*
    sent the *right* type if it actually sent the request).

    Change-Id: Iaa333aaca36a8dc2df65d38ef2173e3a6e2000ee

commit ecca23eb806e11cf6517f0456483da7a065350a8
Author: Clay Gerrard <email address hidden>
Date: Fri Feb 21 15:33:21 2020 -0600

    Extend eventlet_debug logging to GreenAsyncPile

    Change-Id: Ibd9fe5c9a1e75b86eb7d540594d5cf516758e17e

commit 0fb3371484f1d0f629d0b0e33f6aafbff0e43ee9
Author: Sam Morrison <email address hidden>
Date: Tue Feb 18 10:17:50 2020 +1100

    Delay importing swiftclient until after monkey-patching

    Commit message below partly copied from nova:

    Eventlet monkey patching should be as early as possible

    We were seeing infinite recursion opening an ssl socket when running
    various combinations of python3, eventlet, and urllib3. It is not
    clear exactly what combination of versions are affected, but for
    background there is an example of this issue documented here:

    https://github.com/eventlet/eventlet/issues/371

    The immediate cause in swift's case was that we were calling
    eventlet.monkey_patch() after importing swiftclient (which imported
    requests, which finally imported urllib3).

    We only use the imported function in one place, however; hold off on
    importing until we actually need it to ensure that monkey patching
    happens first. Note that we *don't* want to monkey-patch at import time,
    as we've previously had bugs related to import-time side-effects.

    Change-Id: I24f4bcc3d62dc37fd9559032bfd25f5b15f98745
    Closes-bug: #1863680
    Related-bug: #1380815

commit a5afe767581d2cb97cf3690067e6d626c7682c2c
Author: Tim Burke <email address hidden>
Date: Wed Feb 19 10:09:49 2020 -0800

    Revert "Make roll...

tags: added: in-feature-losf
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.