Large File upload fails when uploading to S3

[someone1]

I started to use the s3-use-multiprocessing flag when uploading to S3 to maximize my upload throughput. However, on full backups I would notice half-way through the "Full signatures" file (mine comes out to be about 3GB), I'd repeatedly get the following errors even after retries:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/duplicity/backends/_boto_multi.py", line 398, in _upload
    mp.upload_part_from_file(fd, offset + 1, cb=_upload_callback)
  File "/usr/lib/python2.7/site-packages/boto/s3/multipart.py", line 246, in upload_part_from_file
    query_args=query_args, size=size)
  File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1121, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 827, in send_file
    query_args=query_args)
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 490, in make_request
    override_num_retries=override_num_retries)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 932, in make_request
    return self._mexe(http_request, sender, override_num_retries)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 894, in _mexe
    raise e
error: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/duplicity/backends/_boto_multi.py", line 398, in _upload
    mp.upload_part_from_file(fd, offset + 1, cb=_upload_callback)
  File "/usr/lib/python2.7/site-packages/boto/s3/multipart.py", line 246, in upload_part_from_file
    query_args=query_args, size=size)
  File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1121, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 827, in send_file
    query_args=query_args)
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 490, in make_request
    override_num_retries=override_num_retries)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 932, in make_request
    return self._mexe(http_request, sender, override_num_retries)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 894, in _mexe
    raise e
timeout: timed out

So I dug into the code and found out that the processor pool being used by the multipart upload was setting the number of processes to the number of chunks, which is fine for the 30MB volume size archives I upload, but the signature file isn't split according to this size (I believe there is another bug for this already). Having over 500 processes kick off was resulting in some bug with my network driver or making Amazon rate-limit my attempts to upload. Retries and such were failing so I needed to find a better option.

I did some research (http://improve.dk/pushing-the-limits-of-amazon-s3-upload-performance/) and found my bandwidth would be more than saturated at 8 processes (they use threads here, but the concept is the same since we are I/O dependent). I set this limit in my code (changed http://bazaar.launchpad.net/~duplicity-team/duplicity/0.6-series/view/head:/duplicity/backends/_boto_multi.py#L393 to: pool = multiprocessing.Pool(processes=min(8,chunks))

This seems to have fixed my issue. So I'm attaching a patch that tries to make this a little more dynamic in that the maximum number of processes can be controlled via command-line arguments.

I have yet to test this patch but I did test and verify that limiting the number of processes fixes the issues I was experiencing.

[Kenneth Loafman (kenneth-loafman)]

Thank you for the fix. I will try to get it into the next release.

...Ken

On Mon, May 20, 2013 at 9:24 AM, someone1 <email address hidden>wrote:

> Public bug reported:
>
> I started to use the s3-use-multiprocessing flag when uploading to S3 to
> maximize my upload throughput. However, on full backups I would notice
> half-way through the "Full signatures" file (mine comes out to be about
> 3GB), I'd repeatedly get the following errors even after retries:
>
> Traceback (most recent call last):
> File
> "/usr/lib/python2.7/site-packages/duplicity/backends/_boto_multi.py", line
> 398, in _upload
> mp.upload_part_from_file(fd, offset + 1, cb=_upload_callback)
> File "/usr/lib/python2.7/site-packages/boto/s3/multipart.py", line 246,
> in upload_part_from_file
> query_args=query_args, size=size)
> File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1121, in
> set_contents_from_file
> chunked_transfer=chunked_transfer, size=size)
> File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 827, in
> send_file
> query_args=query_args)
> File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 490,
> in make_request
> override_num_retries=override_num_retries)
> File "/usr/lib/python2.7/site-packages/boto/connection.py", line 932, in
> make_request
> return self._mexe(http_request, sender, override_num_retries)
> File "/usr/lib/python2.7/site-packages/boto/connection.py", line 894, in
> _mexe
> raise e
> error: [Errno 32] Broken pipe
> Traceback (most recent call last):
> File
> "/usr/lib/python2.7/site-packages/duplicity/backends/_boto_multi.py", line
> 398, in _upload
> mp.upload_part_from_file(fd, offset + 1, cb=_upload_callback)
> File "/usr/lib/python2.7/site-packages/boto/s3/multipart.py", line 246,
> in upload_part_from_file
> query_args=query_args, size=size)
> File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1121, in
> set_contents_from_file
> chunked_transfer=chunked_transfer, size=size)
> File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 827, in
> send_file
> query_args=query_args)
> File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 490,
> in make_request
> override_num_retries=override_num_retries)
> File "/usr/lib/python2.7/site-packages/boto/connection.py", line 932, in
> make_request
> return self._mexe(http_request, sender, override_num_retries)
> File "/usr/lib/python2.7/site-packages/boto/connection.py", line 894, in
> _mexe
> raise e
> timeout: timed out
>
> So I dug into the code and found out that the processor pool being used
> by the multipart upload was setting the number of processes to the
> number of chunks, which is fine for the 30MB volume size archives I
> upload, but the signature file isn't split according to this size (I
> believe there is another bug for this already). Having over 500
> processes kick off was resulting in some bug with my network driver or
> making Amazon rate-limit my attempts to upload. Retries and such were
> failing so I needed to find a better option.
>
> I did some research (http://improve.dk/pushing-the-limits-of-amazon-s3
> -upload-perfor...

[Kenneth Loafman (kenneth-loafman)]

The patch will not apply to the current source. Would you mind recreating the patch to match?

[someone1 (prateek)]

This was included in the s3-glacier branch. The fix was already committed.