object-replicator rsync error syslog line too long

Bug #903406 reported by Florian Hines
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Undecided
Kun Huang

Bug Description

If a bad rsync return code is encountered we dump the command run into the log, which in some cases can causes the line to be truncated (depending on max message length set or whether udp is used ?):

Dec 12 21:05:27 172.24.9.208 object-replicator Bad rsync return code: ['rsync', '--recursive', '--whole-file', '--human-readable', '--xattrs', '--itemize-changes', '--ignore-existing', '--timeout=30', '--contimeout=30', '/srv/node/sdv/objects/928710/9fe', '/srv/node/sdv/objects/928710/9ff', '/srv/node/sdv/objects/928710/9fb', '/srv/node/sdv/objects/928710/1e9', '/srv/node/sdv/objects/928710/798', '/srv/node/sdv/objects/928710/29f', '/srv/node/sdv/objects/928710/29d', '/srv/node/sdv/objects/928710/29b', '/srv/node/sdv/objects/928710/c0e', '/srv/node/sdv/objects/928710/344', '/srv/node/sdv/objects/928710/346', '/srv/node/sdv/objects/928710/342', '/srv/node/sdv/objects/928710/348', '/srv/node/sdv/objects/928710/349', '/srv/node/sdv/objects/928710/9f6', '/srv/node/sdv/objects/928710/9f7', '/srv/node/sdv/objects/928710/9f0', '/srv/node/sdv/objects/928710/9f2', '/srv/node/sdv/objects/928710/9f3', '/srv/node/sdv/objects/928710/9f9', '/srv/node/sdv/objects/928710/34d', '/srv/node/sdv/objects/928710/34a', '/srv/node/sdv/objects/928710/34b', '/srv/node/sdv/objects/928710/c08', '/srv/node/sdv/objects/928710/c09', '/srv/node/sdv/objects/928710/298', '/srv/node/sdv/objects/928710/c00', '/srv/node/sdv/objects/928710/297', '/srv/node/sdv/objects/928710/c02', '/srv/node/sdv/objects/928710/295', '/srv/node/sdv/objects/928710/c04', '/srv/node/sdv/objects/928710/293', '/srv/node/sdv/objects/928710/291', '/srv/node/sdv/objects/928710/10d', '/srv/node/sdv/objects/928710/10e', '/srv/node/sdv/objects/928710/d80', '/srv/node/sdv/objects/928710/272', '/srv/node/sdv/objects/928710/ecd', '/srv/node/sdv/objects/928710/274', '/srv/node/sdv/objects/928710/275', '/srv/node/sdv/objects/928710/c15', '/srv/node/sdv/objects/928710/eca', '/srv/node/sdv/objects/928710/6ff', '/srv/node/sdv/objects/928710/6fe', '/srv/node/sdv/objects/928710/6fc', '/srv/node/sdv/objects/928710/6fa', '/srv/node/sdv/objects/928710/27a', '/srv/node/sdv/objects/928710/27c', '/srv/node/sdv/objects/928710/27d', '/srv/node/sdv/objects/928710/27f', '/srv/node/sdv/objects/928710/108', '/srv/node/sdv/objects/928710/9cb', '/srv/node/sdv/objects/928710/102', '/srv/node/sdv/objects/928710/100', '/srv/node/sdv/objects/928710/101', '/srv/node/sdv/objects/928710/106', '/srv/node/sdv/objects/928710/107', '/srv/node/sdv/objects/928710/da6', '/srv/node/sdv/objects/928710/6f8', '/srv/node/sdv/objects/928710/6f7', '/srv/node/sdv/objects/928710/6f6', '/srv/node/sdv/objects/928710/6f5', '/srv/node/sdv/objects/928710/6f2', '/srv/node/sdv/objects/928710/6f1', '/srv/node/sdv/objects/928710/a21', '/srv/node/sdv/objects/928710/074', '/srv/node/sdv/objects/928710/059', '/srv/node/sdv/objects/928710/054', '/srv/node/sdv/objects/928710/055', '/srv/node/sdv/objects/928710/057', '/srv/node/sdv/objects/928710/541', '/srv/node/sdv/objects/928710/7fb', '/srv/node/sdv/objects/928710/7fa', '/srv/node/sdv/objects/928710/7fd', '/srv/node/sdv/objects/928710/7fe', '/srv/node/sdv/objects/928710/d54', '/srv/node/sdv/objects/928710/05d', '/srv/node/sdv/objects/928710/05e', '/srv/node/sdv/objects/928710/05b', '/srv/node/sdv/objects/928710/7f9', '/srv/node/sdv/objects/928710/7f2', '/srv/node/sdv/objects/928710/7f3', '/srv/node/sdv/objects/928710/7f1', '/srv/node/sdv/objects/928710/7f6', '/srv/node/sdv/objects/928710/7f7', '/srv/node/sdv/objects/928710/7f4', '/srv/node/sdv/objects/928710/7f5', '/srv/node/sdv/objects/928710/559', '/srv/node/sdv/objects/928710/555', '/srv/node/sdv/objects/928710/554', '/srv/node/sdv/objects/928710/557', '/srv/node/sdv/objects/928710/551', '/srv/node/sdv/objects/928710/550', '/srv/node/sdv/objects/928710/553', '/srv/node/sdv/objects/928710/552', '/srv/node/sdv/objects/928710/68e', '/srv/node/sdv/objects/928710/68a', '/srv/node/sdv/objects/928710/68c', '/srv/node/sdv/objects/928710/68b', '/srv/node/sdv/objects/928710/55a', '/srv/node/sdv/objects/928710/55c', '/srv/node/sdv/objects/928710/689', '/srv/node/sdv/objects/928710/688', '/srv/node/sdv/objects/928710/685', '/srv/node/sdv/objects/928710/684', '/srv/node/sdv/objects/928710/687', '/srv/node/sdv/objects/928710/686', '/srv/node/sdv/objects/928710/aa5', '/srv/node/sdv/objects/928710/f41', '/srv/node/sdv/objects/928710/f43', '/srv/node/sdv/objects/928710/f42', '/srv/node/sdv/objects/928710/f45', '/srv/node/sdv/objects/928710/f47', '/srv/node/sdv/objects/928710/f46', '/srv/node/sdv/objects/928710/d4c', '/srv/node/sdv/objects/928710/d4a', '/srv/node/sdv/objects/928710/37b', '/srv/node/sdv/objects/928710/37e', '/srv/node/sdv/objects/928710/37d', '/srv/node/sdv/objects/928710/d4e', '/srv/node/sdv/objects/928710/e91', '/srv/node/sdv/objects/928710/406', '/srv/node/sdv/objects/928710/405', '/srv/node/sdv/objects/928710/e94', '/srv/node/sdv/objects/928710/e97', '/srv/node/sdv/objects/928710/400', '/srv/node/sdv/objects/928710/e99', '/srv/node/sdv/objects/928710/e98', '/srv/node/sdv/objects/928710/4a9', '/srv/node/sdv/objects/928710/e9a', '/srv/node/sdv/objects/928710/40e', '/srv/node/sdv/objects/928710/40d', '/srv/node/sdv/objects/928710/40c', '/srv/node/sdv/objects/928710/40a', '/srv/node/sdv/objects/928710/f4c', '/srv/node/sdv/objects/928710/f4b', '/srv/node/sdv/objects/928710/f4e', '/srv/node/sdv/objects/928710/f4f', '/srv/node/sdv/objects/928710/d43', '/srv/node/sdv/objects/928710/370', '/srv/node/sdv/objects/928710/d41', '/srv/node/sdv/objects/928710/372', '/srv/node/sdv/objects/928710/375', '/srv/node/sdv/objects/928710/d46', '/srv/node/sdv/objects/928710/d45', '/srv/node/sdv/objects/928710/24a', '/srv/node/sdv/objects/928710/24c', '/srv/node/sdv/objects/928710/393', '/srv/node/sdv/objects/928710/9e8', '/srv/node/sdv/objects/928710/391', '/srv/node/sdv/objects/928710/390', '/srv/node/sdv/objects/928710/397', '/srv/node/sdv/objects/928710/d9e', '/srv/node/sdv/objects/928710/399', '/srv/node/sdv/objects/928710/398', '/srv/node/sdv/objects/928710/9e5', '/srv/node/sdv/objects/928710/9e4', '/srv/node/sdv/objects/928710/9e7', '/srv/node/sdv/objects/928710/9e6', '/srv/node/sdv/objects/928710/c39', '/srv/node/sdv/objects/928710/c37', '/srv/node/sdv/objects/928710/c31', '/srv/node/sdv/objects/928710/c32', '/srv/node/sdv/objects/928710/39c', '/srv/node/sdv/objects/928710/39b', '/srv/node/sdv/objects/928710/39a', '/srv/node/sdv/objects/928710/39d', '/srv/node/sdv/objects/928710/9eb', '/srv/node/sdv/objects/928710/9ee', '/srv/node/sdv/objects/928710/9ef', '/srv/node/sdv/objects/928710/245', '/srv/node/sdv/objects/928710/244', '/srv/node/sdv/objects/928710/247', '/srv/node/sdv/objects/928710/241', '/srv/node/sdv/objects/928710/243', '/srv/node/sdv/objects/928710/242', '/srv/node/sdv/objects/928710/248', '/srv/node/sdv/objects/928710/c03', '/srv/node/sdv/objects/928710/c3e', '/srv/node/sdv/objects/928710/a2b', '/srv/node/sdv/objects/928710/2a6', '/srv/node/sdv/objects/928710/2a7', '/srv/node/sdv/objects/928710/2a4', '/srv/node/sdv/objects/928710/2a5', '/srv/node/sdv/objects/928710/06e', '/srv/node/sdv/objects/928710/06d', '/srv/node/sdv/objects/928710/176', '/srv/node/sdv/objects/928710/174', '/srv/node/sdv/objects/928710/173', '/srv/node/sdv/objects/928710/2a8', '/srv/node/sdv/objects/928710/e59', '/srv/node/sdv/objects/928710/e58', '/srv/node/sdv/objects/928710/2af', '/srv/node/sdv/objects/928710/060', '/srv/node/sdv/objects/928710/063', '/srv/node/sdv/objects/928710/062', '/srv/node/sdv/objects/928710/065', '/srv/node/sdv/objects/928710/067', '/srv/node/sdv/objects/928710/2aa', '/srv/node/sdv/objects/928710/069', '/srv/node/sdv/objects/928710/17f', '/srv/node/sdv/objects/928710/a39', '/srv/node/sdv/objects/928710/a38', '/srv/node/sdv/objects/928710/17c', '/srv/node/sdv/objects/928710/17b', '/srv/node/sdv/objects/928710/17a', '/srv/node/sdv/objects/928710/ec2', '/srv/node/sdv/objects/928710/b20', '/srv/node/sdv/objects/928710/1ee', '/srv/node/sdv/objects/928710/4b0', '/srv/node/sdv/objects/928710/7a6', '/srv/node/sdv/objects/928710/7a5', '/srv/node/sdv/objects/928710/7a4', '/srv/node/sdv/objects/928710/7a2', '/srv/node/sdv/objects/928710/7a1', '/srv/node/sdv/objects/928710/7a0', '/srv/node/sdv/objects/928710/4b2', '/srv/node/sdv/objects/928710/d90', '/srv/node/sdv/objects/928710/7af', '/srv/node/sdv/objects/928710/7ad', '/srv/node/sdv/objects/928710/7ac', '/srv/node/

Changed in swift:
status: New → Triaged
Revision history for this message
Chmouel Boudjnah (chmouel) wrote :

This is still hapening right ? I know there was some changes lately w.r.t the rsync code and wondering if that was fixed along the way

Changed in swift:
status: Triaged → Confirmed
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack Object Storage (swift) because there has been no activity for 60 days.]

Changed in swift:
status: Incomplete → Expired
Revision history for this message
Sahdev Zala (spzala) wrote : AUTO: Sahdev P Zala is out of the office (returning 12/03/2012)

I am out of the office until 12/03/2012.

For any emergency, please contact my manager Cynthia L Waldhoff.

Note: This is an automated response to your message "[Bug 903406] Re:
object-replicator rsync error syslog line too long" sent on 11/30/2012
21:17:32.

This is the only notification you will receive while this person is away.

Mike Barton (redbo)
Changed in swift:
status: Expired → Confirmed
Revision history for this message
Kun Huang (academicgareth) wrote :

Is there any idea to make this more readable?
The purpose of this output is simply which command is failed.

Revision history for this message
Samuel Merritt (torgomatic) wrote :

About the only thing I can think of is to move the return code earlier in the message so that the really long command doesn't cause that to be lost.

Revision history for this message
Kun Huang (academicgareth) wrote :

But only return value is greater than 0, this long message will be logged. I mean we don't need watch this return value, because it mush none zero.

Revision history for this message
Samuel Merritt (torgomatic) wrote :

On my machine, the rsync man page lists 18 distinct nonzero exit codes, each with a particular cause. The presence of the message in the logs may indicate a nonzero exit, but the exact exit code is helpful in diagnosing the underlying problem.

Revision history for this message
Kun Huang (academicgareth) wrote :

You're right.
I should think one step further

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/37029

Changed in swift:
assignee: nobody → Kun Huang (academicgareth)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/37029
Committed: http://github.com/openstack/swift/commit/c7a79991f82a7e472c1767e5e56a96fde485d9ae
Submitter: Jenkins
Branch: master

commit c7a79991f82a7e472c1767e5e56a96fde485d9ae
Author: Kun Huang <email address hidden>
Date: Mon Jul 15 15:43:56 2013 +0800

    log return value before arguments when rsync error

    The arguments of rsync command seems too long. If some errors occur, it
    is hard to debug. Moving return value of rsync command first could help
    people at least know reason of rsync error by manuals.

    fixes bug #903406

    Change-Id: I1cac7bbca74824a6d47e6ceb9f654d4046fcbb9e

Changed in swift:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/ec)

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/39740

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (feature/ec)
Download full text (12.7 KiB)

Reviewed: https://review.openstack.org/39740
Committed: http://github.com/openstack/swift/commit/7a404cdea400e12d1ee8660c9f9477a87e419602
Submitter: Jenkins
Branch: feature/ec

commit 3035a93ed2c467fb248a77389274033ff540590c
Author: David Goetz <email address hidden>
Date: Tue Jul 30 11:52:23 2013 -0700

    Tell swift to figure out content type

    Be able to tell swift to figure out the content-type even if it is
    sent because old client code / curl has trouble sending blank
    content-type headers.

    Change-Id: Ie65ddf8993a19ea74e0b85a2ae56da84a617c19d

commit 47e34cf15eb652bd54d53b0cfb5238d04e4416d1
Author: TheSriram <email address hidden>
Date: Tue Jul 30 14:57:48 2013 -0400

    fix(gitignore) : ignore *.egg and *.egg-info

    Change-Id: I9496dab9e8f779a7ae045bb73b2cf8e77bda0d30

commit 22e7cbceed4730f93ee601a9955215e208e64295
Author: Alex Gaynor <email address hidden>
Date: Mon Jul 29 22:41:29 2013 -0700

    When iterating over a range of a file, always close it

    This is needed on Pythons without reference counting garbage collectors (e.g.
    PyPy).

    Change-Id: I1d06eb8fe08ee6eeb45caa47b653d6af0bb18267

commit 1e3ad44784bed7cbd8a8f367d633cc1d7c1d4172
Author: Peter Portante <email address hidden>
Date: Mon Jul 29 15:49:37 2013 -0400

    Merge object base module into diskfile.

    All of the module methods of the (now defunct) base module we really
    concerned with the on-disk layout which is what the DiskFile module is
    really about.

    Change-Id: I96e022c5f96e31537ced74139185851a2751701c
    Signed-off-by: Peter Portante <email address hidden>

commit eb99e8f84cf78c20e03f4a4fd92e243a22171b23
Author: Donagh McCabe <email address hidden>
Date: Tue Jul 23 15:10:09 2013 +0100

    Obscure the X-Auth-Token in proxy log

    The X-Auth-Token is sensitive data. If revealed to an unauthozied person,
    they can now make requests against an account until the token expires.

    This implementation maintains current behavior (i.e, the token
    is logged). Implementers can choose to set reveal_sensitive_prefix
    to (e.g.) 12 so only first 12 characters of the token are logged.
    Or, set to 0 to replace the token with "...".

    DocImpact

    Part of bug #1004114

    Change-Id: Iecefa843d8f9ef59b9dcf0860e7a4d0e186a6cb5

commit f9e73a0fe6da7da94955c40ffb50ddc6526c1466
Author: Pete Zaitcev <email address hidden>
Date: Mon Jul 29 19:12:10 2013 -0600

    Eliminate can_delete_db

    The method can_delete_db() appears to be not only unused, but has
    always been so. I verified this going back to Austin release. It is
    very strange that we never noticed it until now, but here it is.

    Change-Id: I4445c4b2c4721f880c9dbb1eac055c0601ae6372

commit be688c31562a5789fba678a0675eff1040308202
Author: Alex Gaynor <email address hidden>
Date: Fri Jul 19 20:07:27 2013 -0700

    Encode unicode from JSON before using it as a string.

    Right now this code fails when used with a JSON
    decoder that always produces unicode. This isn't
    usually the case with CPython, where simplejson
    is used most of the time, however w...

Thierry Carrez (ttx)
Changed in swift:
milestone: none → 1.9.1
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.