S3: Unable to access european buckets, new authorization mechanism not supported AWS4-HMAC-SHA256

[dfoerster]

Accessing a bucket that was created in the AWS console in the German location Frankfurt is not working, due to Amazon requiring some new authentication method.

I made a minor modification to the code as outlined in bug #1278529 to have the exception raised instead of displaying "No connection to backend".

To reproduce create a bucket in the Frankfurt location and try to access it using duplicity.

duplicity full -v9 --s3-european-buckets --s3-use-new-style /path/to/backup s3+http://my-european-bucket/my-backup-path
Exit code: 30
Using archive dir: /home/dave/.cache/duplicity/251e8f41ea8795b16a9fc8f0be2d4fa2
Using backup name: 251e8f41ea8795b16a9fc8f0be2d4fa2
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.~par2wrapperbackend Succeeded
Using temporary directory /tmp/duplicity-uPZnQI-tempdir
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1509, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1503, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1336, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1059, in ProcessCommandLine
    backup, local_pathname = set_backend(args[0], args[1])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 952, in set_backend
    globals.backend = backend.get_backend(bend)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 163, in get_backend
    return _backends[pu.scheme](pu)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 163, in __init__
    self.resetConnection()
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 186, in resetConnection
    self.bucket = self.conn.lookup(self.bucket_name)
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 376, in lookup
    bucket = self.get_bucket(bucket_name, validate, headers=headers)
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 371, in get_bucket
    bucket.get_all_keys(headers, maxkeys=0)
  File "/usr/lib/python2.7/dist-packages/boto/s3/bucket.py", line 347, in get_all_keys
    '', headers, **params)
  File "/usr/lib/python2.7/dist-packages/boto/s3/bucket.py", line 314, in _get_all
    response.status, response.reason, body)
S3ResponseError: S3ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>E3E4915B08D90EA6</RequestId><HostId>tkxe/V3CcQ6fnyLsvXDp3uABC1ppfBpvmZkA17wyGyIOjhhjyylTNhxm+5jGNRDY4SWLCQH+brg=</HostId></Error>

[Kenneth Loafman (kenneth-loafman)]

This is a boto issue, not a duplicity issue. You will need to upgrade to at least boto 2.36.0. To do so:

$ sudo pip install --upgrade boto

[Albin (albgus)]

I am experiencing this exact bug with the duplicity package installed from the main Ubuntu repo in 18.04. So the only workaround right now is to update the boto package from outside of the package manager?

I don't think that this is an acceptable solution as this defeats the point of having a package manager in the first place.

[Kenneth Loafman (kenneth-loafman)]

Then get the repo managers to change their policy. duplicity is up to 0.8.12 now, but it's not even in the repo to go out with Ubuntu 20.04. Other packages like boto are in the same shape.

[Sebastien Bacher (seb128)]

Comment #1 stated the issue here was fixed in boto 2.36, that version has been in Ubuntu since at least Xenial, best to open a new report if you still have issue in newer versions