python-swiftclient

Segmented object upload hangs w/client 2.0

Bug #1284360 reported by Samuel Merritt on 2014-02-24

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	python-swiftclient	Fix Released	Undecided	Luis de Bethencourt

Bug Description

Try this with swiftclient 2.0.2, and it'll hang:

$ swift upload con ChangeLog -s 1024

I used ChangeLog from python-swiftclient's source tree, but any file that's at least 1025 bytes will do.

This doesn't occur with 1.9.

Revision history for this message

Samuel Merritt (torgomatic) wrote on 2014-02-24:

Well, it doesn't hang forever, but it sure doesn't actually work:

swift@saio:~/python-swiftclient$ time swift upload swiftclient ChangeLog -S 2000
ChangeLog segment 0
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000010 408 Request Timeout [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000001 408 Request Timeout [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000006 408 Request Timeout [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000005 408 Request Timeout [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000003 408 Request Timeout [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000002 408 Request Timeout [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000004 408 Request Timeout [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000007 408 Request Timeout [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000008 408 Request Timeout [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000009 408 Request Timeout [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Aborting manifest creation because not all segments could be uploaded. swiftclient/ChangeLog

real 6m32.499s
user 0m1.572s
sys 0m0.884s
swift@saio:~/python-swiftclient$

Looks like swiftclient is failing to send data to the server, and eventually the server gets tired of waiting and shuts down the connections.

Well, it doesn't hang forever, but it sure doesn't actually work:

swift@saio:~/python-swiftclient$ time swift  upload swiftclient ChangeLog -S 2000
ChangeLog segment 0
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000010 408 Request Timeout  [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000001 408 Request Timeout  [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000006 408 Request Timeout  [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000005 408 Request Timeout  [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000003 408 Request Timeout  [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000002 408 Request Timeout  [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000004 408 Request Timeout  [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000007 408 Request Timeout  [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000008 408 Request Timeout  [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Object PUT failed: http://localhost:8080:8080/v1/AUTH_test/swiftclient_segments/ChangeLog/1345150987.000000/20448/2000/00000009 408 Request Timeout  [first 60 chars of response] <html><h1>Request Timeout</h1><p>The server has waited too l
Aborting manifest creation because not all segments could be uploaded. swiftclient/ChangeLog

real	6m32.499s
user	0m1.572s
sys	0m0.884s
swift@saio:~/python-swiftclient$

Looks like swiftclient is failing to send data to the server, and eventually the server gets tired of waiting and shuts down the connections.

Revision history for this message

Luis de Bethencourt (l2is) wrote on 2014-02-24:

I will look at this :)

Thanks

Changed in python-swiftclient:
assignee:	nobody → Luis de Bethencourt (l2is)

Revision history for this message

clayg (clay-gerrard) wrote on 2014-02-25:

wrap contents in a LengthWrapper Edit (1.8 KiB, text/plain)

ok, so we broke the contact for swiftclient.client.put_object

    :param content_length: value to send as content-length header; also limits
                           the amount read from contents; if None, it will be
                           computed via the contents or chunked transfer
                           encoding will be used

^ there's nothing enforcing the "also limits the amount read from contents" bit...

there's a mention of "Fixes https://github.com/kennethreitz/requests/issues/1648" - but the "solution" doesn't really work for files:

            try:
                contents.len = content_length
            except AttributeError:
                pass

You can set an arbitrary attribute on a file object in python 2, so the AttributeErrors passes it through directly and requests will stat the file to set the "Content-Length" on the request.

But since the _segment_job in bin/swift is seeking out into the middle of the file requests is unable to send all of the bytes it promises and the proxy server throws a HTTPRequestTimeout.

Solution is attached...

Revision history for this message

clayg (clay-gerrard) wrote on 2014-02-25:

*can't* set an arbitrary attribute

anyway:

https://review.openstack.org/#/c/76102/