s3cmd put commands fail with files larger than 5GB

[Adam Horwich]

Affects: Description: Ubuntu 12.04.1 LTS Release: 12.04
s3cmd:
  Installed: 1.0.0-1

In the current 12.04 LTS release of s3cmd 1.0.0-1 mutlipart uploads are broken. When the file size threshold is met to trigger a multipart upload, the command complains of a 'connection reset by peer' type error and attempts to throttle the upload but ultimately fails. This bug was fixed in the later 1.1.0beta2 release of s3cmd (however this version has a bug preventing it from working on Ubuntu 12.10) and the current stable release for 12.10 1.1.0beta3 works as expected.

dd if=/dev/zero of=s3file bs=1M count=6000
s3cmd --progress put s3file s3://mybucket/s3file
s3file -> s3://mybucket/s3file [1 of 1]
    5881856 of 6291456000 0% in 1s 5.05 MB/s failed
WARNING: Upload failed: /s3file ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
s3file -> s3://mybucket/s3file [1 of 1]
    6070272 of 6291456000 0% in 1s 5.65 MB/s failed
WARNING: Upload failed: /s3file ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
s3file -> s3://mybucket/s3file [1 of 1]
     401408 of 6291456000 0% in 1s 375.99 kB/s failed
WARNING: Upload failed: /s3file ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
s3file -> s3://mybucket/s3file [1 of 1]
      90112 of 6291456000 0% in 1s 81.64 kB/s failed
WARNING: Upload failed: /s3file ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
s3file -> s3://s3file [1 of 1]
      28672 of 6291456000 0% in 1s 18.48 kB/s failed
WARNING: Upload failed: /s3file ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
s3file -> s3://mybucket/s3file [1 of 1]
      12288 of 6291456000 0% in 2s 4.77 kB/s failed
ERROR: Upload of 's3file' failed too many times. Skipping that file.

A put on a smaller file succeeds without issue.

[Scott Moser (smoser)]

Adam,
  Thanks for taking the time to file a bug to make Ubuntu better.
  I've marked this as "fix released", based on your statement:
     "and the current stable release for 12.10 1.1.0beta3 works as expected."

  Since 12.10 is fixed the "ubuntu" task is officially "fix released". If you would like this to come back to 12.04 in a StableReleaseUpdate [1], and I'm guessing you would, then you can "Nominate" this bug for 12.04.

  If I've incorrectly read your statements, and this is still broken in 12.10 and 13.03, please mark it as such.

--
[1] https://wiki.ubuntu.com/StableReleaseUpdates

[Adam Horwich (adam-i95o51082ne8r)]

Hi Scott,

Your summation is accurate. I shall update the bug description accordingly and go through the process to nominate it for 12.04.

Thanks,

Adam.