boto3+s3 bucket fails

[Reed Dier]

I have a duplicity job that runs on multiple hosts, and am now having failures on my 20.04 hosts.
The syntax for the job (as I am currently testing to debug) is:
duplicity --dry-run list-current-files --name $name --s3-use-ia --archive-dir $archiveDir --tempdir $tmpDir --s3-multipart-chunk-size 256 --s3-use-multiprocessing --volsize 1024 --verbosity 8 boto3+s3://s3.amazonaws.com/bucket.name

Using the duplicity packages from focal/main
duplicity=0.8.11.1612-1
And I installed the boto packages from focal/universe
python3-boto3=1.9.253-1
python3-botocore=1.14.14+repack-1

This runs into the error:
Using temporary directory $tmpDir/duplicity-iyhq3se4-tempdir
Traceback (innermost last):
  File "/usr/bin/duplicity", line 106, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 92, in with_tempdir
    fn()
  File "/usr/lib/python3/dist-packages/duplicity/dup_main.py", line 1525, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/duplicity/commandline.py", line 1175, in ProcessCommandLine
    globals.backend = backend.get_backend(args[0])
  File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 225, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 211, in get_backend_object
    return factory(pu)
  File "/usr/lib/python3/dist-packages/duplicity/backends/s3_boto3_backend.py", line 85, in __init__
    self.reset_connection()
  File "/usr/lib/python3/dist-packages/duplicity/backends/s3_boto3_backend.py", line 96, in reset_connection
    self.s3.meta.client.head_bucket(Bucket=self.bucket_name)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
 botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden

I then tried the duplicity snap from the snap store, both the stable and edge builds, with the same issue:
Main action: list-current
================================================================================
duplicity 0.8.17
Args: /snap/duplicity/138/bin/duplicity --dry-run --ignore-errors list-current-files --name $name --s3-use-ia --archive-dir $archiveDir --tempdir $tmpDir --s3-multipart-chunk-size 256 --s3-use-multiprocessing --volsize 1024 --verbosity 8 boto3+s3://s3.amazonaws.com/bucket.name
Linux duplicity 5.4.0-53-generic #59-Ubuntu SMP Wed Oct 21 09:38:44 UTC 2020 x86_64 x86_64
/snap/duplicity/141/usr/bin/python3 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0]
================================================================================
Attempt 1 failed. ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden

Main action: list-current
================================================================================
duplicity 0.8.18.dev11
Args: /snap/duplicity/141/bin/duplicity --dry-run --ignore-errors list-current-files --name $name --s3-use-ia --archive-dir $archiveDir --tempdir $tmpDir --s3-multipart-chunk-size 256 --s3-use-multiprocessing --volsize 1024 --verbosity 8 boto3+s3://s3.amazonaws.com/bucket.name
Linux duplicity 5.4.0-53-generic #59-Ubuntu SMP Wed Oct 21 09:38:44 UTC 2020 x86_64 x86_64
/snap/duplicity/141/usr/bin/python3 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0]
================================================================================
Attempt 1 failed. ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden

I also tried to update the boto3/botocore packages by way of pip3:
pip3 freeze | grep boto
boto3==1.16.19
botocore==1.19.19

And lastly as a last ditch, I tried to pull the packages from the groovy/universe repo, bringing them to:
python3-boto3 1.13.14-1
python3-botocore 1.17.22+repack-1_all

I have also tried in a 20.10 system as well to see if any changes there, but same results.

Hoping for something I can look at to get this back working as it was in 18.04 since 14.04.

For good measure, I verified in a bionic build that using the same arguments as above with duplicity=0.7.17
python-boto=2.44.0-1ubuntu2.18.04.1

I then verified by uninstalling the deb and installing the snap (0.8.17) on bionic that the behavior stays the same.

And just to satisfy the 1-4 guidelines:
1. The main system I am having issues with:
$ lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04

Also tested against
$ lsb_release -rd
Description: Ubuntu 18.04.5 LTS
Release: 18.04
$ lsb_release -rd
Description: Ubuntu 20.10
Release: 20.10

2. $ apt-cache policy duplicity
duplicity:
  Installed: 0.8.11.1612-1
  Candidate: 0.8.11.1612-1
  Version table:
 *** 0.8.11.1612-1 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

3. Expect duplicity to function against my s3 bucket.
4. botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden

[Kenneth Loafman (kenneth-loafman)]

Have you tried reading your files with s3cmd or another tool?

If that works, check permission keys etc..

If that does not work, I'd like a log file, please. This is going big and I only need a portion. Please run with -v9, redirect to a log file, then run:

$ head -n 200 < log > log.head
$ tail -n 200 < log > log.tail

Attach, do NOT copy/paste, log.head and log.tail to this bug report. See green button below.

[Reed Dier (reedacus39)]

For the sake of pointing out the AWS creds which I think is what you are hinting at with "check permission keys".
As I was in my original bash script, and for testing I am exporting in my shell:
AWS_ACCESS_KEY_ID=$akey
AWS_SECRET_ACCESS_KEY=$sakey
PASSPHRASE=$passphrase

s3cmd seemed to like this without intervention, but I manually made ~/.s3cfg and verified that working as well.

I also created an .aws/credentials file with
[default]
aws_access_key_id = $akey
aws_secret_access_key = $sakey

I then verified that boto3 was seeing this with
ubuntu@duplicity-f:~$ python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> client = boto3.client('s3')
>>> client.list_buckets()
{'ResponseMetadata': {'RequestId': '$responseid', 'HostId': '$hostid', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': '$amz-id-2', 'x-amz-request-id': 'amz-request-id',[snip],}}

And this properly returns the buckets in this IAM account, so I assume boto3 is properly able to hit the s3 credentials file. I sanitized some data, hopefully which doesn't nullify the utility.
Attaching the full sanitized response as boto3_demo_working.log

My python is very rusty, I'm more or less bashing my way through verifying all of this so if something would be more effective please let me know.

That said, I am still unable to get past the 403 in the duplicity run.
Attaching full 2>&1 output as duplicity_demo_nonworking.log.
Again, sanitized as much as possible, while still not obfuscating the important parts (hopefully).

And just to provide some clean and clear repro steps:
I am using a fresh focal image in multipass for consistent clean testing.
# apt update; apt dist-upgrade; apt install duplicity python3-boto3

ubuntu@duplicity-f:~$ lsb_release -a ; uname -a ; dpkg -l | grep duplicity ; dpkg -l | grep -i boto
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
Linux duplicity-f 5.4.0-54-generic #60-Ubuntu SMP Fri Nov 6 10:37:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
ii duplicity 0.8.11.1612-1 amd64 encrypted bandwidth-efficient backup
ii python3-boto3 1.9.253-1 all Python interface to Amazon's Web Services - Python 3.x
ii python3-botocore 1.16.19+repack-1ubuntu0.20.04.1 all Low-level, data-driven core of boto 3 (Python 3)

Please let me know what else I may be able to provide.

[Kenneth Loafman (kenneth-loafman)]

Please upgrade to the current version of duplicity.  This will assure that any bugs fixed since your release are available and may fix your issue. Current release is 0.8.17.

I would suggest using one of the snap builds if at all possible.

There are multiple options both stable and daily:
- Source - https://gitlab.com/duplicity/duplicity
- Stable tarball install - https://launchpad.net/duplicity/+download
- Stable duplicity PPA - https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-release-git
- Daily duplicity PPA - https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-develop-git
- Stable snap builds - “sudo snap install duplicity —classic"
- Latest snap builds - “sudo snap install duplicity —classic —edge"
- Stable pip3 builds - “sudo pip3 install duplicity"
- Latest pip3 builds - “sudo pip3 install --pre duplicity"

Note 1: UNINSTALL duplicity first if it was installed from a different source..  This is due to divergent install locations, especially between repository installs and the other forms.

Note 2: Launchpad PPAs contain builds for Bionic 18.04, Eoan 19.10, Focal 20.04, and Hirsute 20.10.  Xenial 16.04 works with Snap and Pip installs, but cannot be built under Launchpad PPAs at the moment.

[Reed Dier (reedacus39)]

Attached output is from a fresh multipass instance of focal, only repro steps after multipass launch were to create aws creds in ~/.aws/credentials and $ sudo snap install duplicity --classic.

[Kenneth Loafman (kenneth-loafman)]

Thanks for the thorough analysis.

[Kenneth Loafman (kenneth-loafman)]

Sorry, it took me a while to grok that you were using the boto3 backend. The format for a boto3 URL is boto3+s3://bucket_name[/prefix]. It's in the man page and usage screen.

Make that change and let me know how it went.

[Reed Dier (reedacus39)]

Apologies for the wild goose chase, you are, expectedly, correct.

I had seen the boto/S3 sections in the man page, however I never connected the dots that between the boto -> boto3 transition, the hostname of the s3 url was deprecated in favor of specifying only the bucket name.

I merely transitioned my bucket URL from s3://s3.amazonaws.com/$bucket to boto3+s3://s3.amazonaws.com/$bucket , but boto3+s3://$bucket is in fact the correct syntax.