Comment 32 for bug 385495

Hello,

I am also having the same problem with backup of 900GB and large signature files (>10 GB).

If the full fix for "Large backup signature and manifest files should be split with --volsize too", is too complex to get into the next release, at least for Amazon S3, there is a hack around the issue:

Amazon S3 announced an increase in the maximum size of an object from 5 gigabytes to 5 terabytes (http://aws.amazon.com/about-aws/whats-new/2010/12/09/Announcing-Amazon-S3-Large-Object-Support/) in december 2010.
What is needed to use this feature is to upload S3 objects in parts. Parts can be uploaded independently, in any order, and in parallel. (http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html).

boto supports multipart uploads, but they are not transparent (ie calling key.set_contents_from_filename(...) will not use boto's mutlipart upload capability for large files).

Best,
Antoine