"Multipart upload failed" when running backup on s3

[Michael Apozyan]

python 2.7
duplicity 0.8.05
Linux version 4.15.0-1045-aws (buildd@lcy01-amd64-026) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #47-Ubuntu SMP Fri Aug 2 13:50:30 UTC 2019

Running backup with following arguments:

> /usr/local/bin/duplicity --no-compression --no-encryption --s3-use-multiprocessing --s3-unencrypted-connection --s3-use-new-style --s3-multipart-max-procs 32 --s3-multipart-chunk-size 5 --volsize 500 --asynchronous-upload --verbosity 9 --progress <local path> s3://s3-eu-central-1.amazonaws.com/<s3 path>

Repeatedly getting following exception:

Backtrace of previous error: Traceback (innermost last):
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 371, in inner_retry
    return fn(self, *args)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 531, in put
    self.__do_put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 517, in __do_put
    self.backend._put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 264, in _put
    self.upload(source_path.name, key, headers)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backends/_boto_multi.py", line 188, in upload
    raise BackendException(u"Multipart upload failed. Aborted.")
 BackendException: Multipart upload failed. Aborted.

Attempt 2 failed. BackendException: Multipart upload failed. Aborted.

[Michael Apozyan (mapozyan)]

The bug was introduced during 2to3 transition, after which some integer division was converted to float division thus breaking logic of multipart upload.

Proposed patch solves the issue.

[Jussi Pollari (jussipol)]

Hi,

I am getting this same error with multipart upload on duplicity 0.8.13 (and Python3):
. Backtrace of previous error: Traceback (innermost last):
. File "/usr/local/lib/python3.6/dist-packages/duplicity/backend.py", line 374, in inner_retry
. return fn(self, *args)
. File "/usr/local/lib/python3.6/dist-packages/duplicity/backend.py", line 545, in put
. self.__do_put(source_path, remote_filename)
. File "/usr/local/lib/python3.6/dist-packages/duplicity/backend.py", line 531, in __do_put
. self.backend._put(source_path, remote_filename)
. File "/usr/local/lib/python3.6/dist-packages/duplicity/backends/_boto_single.py", line 264, in _put
. self.upload(source_path.name, key, headers)
. File "/usr/local/lib/python3.6/dist-packages/duplicity/backends/_boto_multi.py", line 192, in upload
. raise BackendException(u"Multipart upload failed. Aborted.")
. duplicity.errors.BackendException: Multipart upload failed. Aborted.

duplicity \
--name backup \
--full-if-older-than=1W \
--s3-use-new-style \
--num-retries=5 \
--volsize=2048 \
--s3-use-multiprocessing \
--s3-multipart-chunk-size=256 \
--log-file=/backup-logdir/duplicity_backup.log \
--archive-dir=/duplicity_archive \
-v9 \
/back-me-up s3+http://my-bucket/my_app

Package Version
--------------- ---------------
awscli 1.18.58
boto 2.49.0
botocore 1.16.8
certifi 2018.1.18
cffi 1.14.0
chardet 3.0.4
colorama 0.4.3
cryptography 2.9.2
docutils 0.15.2
duplicity 0.8.13
fasteners 0.15
future 0.18.2
idna 2.6
jmespath 0.10.0
lockfile 0.12.2
monotonic 1.5
ndg-httpsclient 0.5.1
oauthlib 3.1.0
pip 20.1
psutil 5.7.0
pyasn1 0.4.8
pycparser 2.20
pycrypto 2.6.1
pygobject 3.26.1
pyOpenSSL 19.1.0
python-apt 1.6.5+ubuntu0.2
python-dateutil 2.8.1
PyYAML 5.3.1
requests 2.23.0
rsa 3.4.2
s3transfer 0.3.3
setuptools 39.0.1
setuptools-scm 3.5.0
six 1.11.0
urllib3 1.25.9
wheel 0.34.2

[Kenneth Loafman (kenneth-loafman)]

Please try boto3+s3:// instead of just s3://

[Jussi Pollari (jussipol)]

Hi Kenneth,

Thanks for the quick response and confirmed that it works now when changing to boto3+s3://