duplicity fails to use existing S3 bucket in boto backend

Bug #1568677 reported by Florian Kruse on 2016-04-11
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Unassigned

Bug Description

I recently updated to duplicity 0.7.07 and suddenly my backup to S3 fails. I believe, I managed to trace the issue to duplicity/backends/_boto_single.py. I am running

/usr/bin/duplicity --name duply_system --verbosity 9 / s3://s3-eu-central-1.amazonaws.com/my_bucket/duplicity

with the environment variables

S3_USE_SIGV4=True
AWS_ACCESS_KEY_ID=MYACCESSKEY
AWS_SECRET_ACCESS_KEY=MYSECRET

This fails with the output given below (line number in _boto_single.py approximate as I added some debug output). I managed to reconstruct: In my case, BotoBackend._list fails as self.bucket is None. Before, BotoBackend.resetConnection is called, which sets self.bucket to None. Inside BotoBackend.resetConnection, self.bucket is only set to a valid bucket in case the bucket does not yet exist (in which case it is created via self.conn.create_bucket). In case the bucket already exists, self.bucket is left None. No other attempt inside BotoBackend is made to set self.bucket, eventually letting BotoBackend._list fail.

Output of call:

Using archive dir: /root/.cache/duplicity/duply_system
Using backup name: duply_system
Import of duplicity.backends.acdclibackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Main action: inc
================================================================================
duplicity 0.7.07 ($reldate)
Args: /usr/bin/duplicity --name duply_system --verbosity 9 / s3://s3-eu-central-1.amazonaws.com/backup-git.point-8.de-gitlab/duplicity
Linux git 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Fri Mar 18 16:37:35 UTC 2016 x86_64 x86_64
/usr/bin/python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2]
================================================================================
Using temporary directory /tmp/duplicity-OwOHZf-tempdir
Registering (mkstemp) temporary file /tmp/duplicity-OwOHZf-tempdir/mkstemp-gDTnhP-1
Temp has 35986862080 available, backup will use approx 34078720.
Backtrace of previous error: Traceback (innermost last):
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 367, in inner_retry
    return fn(self, *args)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 569, in list
    return [tobytes(x) for x in self.backend._list()]
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 253, in _list
    raise BackendException("No connection to backend")
 BackendException: No connection to backend

Attempt 1 failed. BackendException: No connection to backend

Florian Kruse (fkruse) wrote :

Modifying resetConnection to

    def resetConnection(self):
        import boto
        if getattr(self, 'conn', False):
            self.conn.close()
        self.bucket = None
        self.conn = None
        self.storage_uri = None
        del self.conn
        del self.storage_uri
        self.storage_uri = boto.storage_uri(self.boto_uri_str)
        self.conn = get_connection(self.scheme, self.parsed_url, self.storage_uri)
        if not self.conn.lookup(self.bucket_name):
            if globals.s3_european_buckets:
                self.bucket = self.conn.create_bucket(self.bucket_name,
                                                      location=Location.EU)
            else:
                self.bucket = self.conn.create_bucket(self.bucket_name)
        else:
            self.bucket = self.conn.get_bucket(self.bucket_name)

seems to resolve the issue. Although, I am not sure if this is wise (found this while trial-and-erroring through the file).

Florian Kruse (fkruse) wrote :
summary: - duplicity failing to use existing bucket in boto backend
+ duplicity fails to use existing S3 bucket in boto backend
l13t (liet-5) wrote :

Got the same issue. Solution suggested by @fkruse helped to resolve issue.

l13t (liet-5) wrote :

Additional info about my problem.

S3_USE_SIGV4=<default value> setting it to True didn't help.
I'm using eu-west-1 dc to store my backups.

Download full text (4.9 KiB)

As far as I can tell, you made no changes to the file that was released.
Could you provide a diff of the file?

On Sun, Apr 10, 2016 at 11:53 PM, Florian Kruse <email address hidden> wrote:

> Issue was introduced here, I believe:
>
> http://bazaar.launchpad.net/~duplicity-
> team/duplicity/0.7-series/revision/1175
>
>
> ** Summary changed:
>
> - duplicity failing to use existing bucket in boto backend
> + duplicity fails to use existing S3 bucket in boto backend
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1568677
>
> Title:
> duplicity fails to use existing S3 bucket in boto backend
>
> Status in Duplicity:
> New
>
> Bug description:
> I recently updated to duplicity 0.7.07 and suddenly my backup to S3
> fails. I believe, I managed to trace the issue to
> duplicity/backends/_boto_single.py. I am running
>
> /usr/bin/duplicity --name duply_system --verbosity 9 / s3://s3-eu-
> central-1.amazonaws.com/my_bucket/duplicity
>
> with the environment variables
>
> S3_USE_SIGV4=True
> AWS_ACCESS_KEY_ID=MYACCESSKEY
> AWS_SECRET_ACCESS_KEY=MYSECRET
>
> This fails with the output given below (line number in _boto_single.py
> approximate as I added some debug output). I managed to reconstruct:
> In my case, BotoBackend._list fails as self.bucket is None. Before,
> BotoBackend.resetConnection is called, which sets self.bucket to None.
> Inside BotoBackend.resetConnection, self.bucket is only set to a valid
> bucket in case the bucket does not yet exist (in which case it is
> created via self.conn.create_bucket). In case the bucket already
> exists, self.bucket is left None. No other attempt inside BotoBackend
> is made to set self.bucket, eventually letting BotoBackend._list fail.
>
> Output of call:
>
> Using archive dir: /root/.cache/duplicity/duply_system
> Using backup name: duply_system
> Import of duplicity.backends.acdclibackend Succeeded
> Import of duplicity.backends.azurebackend Succeeded
> Import of duplicity.backends.b2backend Succeeded
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.copycombackend Succeeded
> Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.giobackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.hubicbackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.lftpbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.mediafirebackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.multibackend Succeeded
> Import of duplicity.backends.ncftpbackend Succeeded
> Import of duplicity.backends.onedrivebackend Succeeded
> Import of duplicity.backends.par2backend Succeeded
> Import of duplicity.backends.pydrivebackend Succeeded
> Import of duplicity.backends.rsyncbackend Succeeded
> Import of dupli...

Read more...

Changed in duplicity:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Kenneth Loafman (kenneth-loafman)
milestone: none → 0.7.08
milestone: 0.7.08 → 0.8.00
milestone: 0.8.00 → 0.7.08
Florian Kruse (fkruse) wrote :
Florian Kruse (fkruse) wrote :

Sure, here you go.

OK, here's a patch that will work when the bucket does not exist.

Changed in duplicity:
status: In Progress → Fix Committed
assignee: Kenneth Loafman (kenneth-loafman) → nobody
Marcos BL (marcosbl) wrote :

Same problem here after 0.7.07 update. Tried the proposed patch but it doesn't seem to fix it when using EU buckets:

GPG_KEY='disabled'
TARGET='s3://s3-eu-west-1.amazonaws.com/my/bucket'
SOURCE='/'
MAX_AGE=1M
MAX_FULL_BACKUPS=2
MAX_FULLBKP_AGE=15D
VOLSIZE=250
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE --s3-use-multiprocessing --volsize $VOLSIZE --tempdir /home/cache"
VERBOSITY=5

---------------------------------------

Start duply v1.5.10, time is 2016-04-11 23:19:40.
Using profile '/root/.duply/cloud'.
Using installed duplicity version 0.7.07, python 2.7.6, gpg 1.4.16 (Home: ~/.gnupg), awk 'GNU Awk 4.0.1'.
Test - En/Decryption skipped. (GPG disabled)

--- Start running command STATUS at 23:19:41.142 ---
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1537, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1531, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1369, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1109, in ProcessCommandLine
    globals.backend = backend.get_backend(args[0])
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 223, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 209, in get_backend_object
    return factory(pu)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_multi.py", line 83, in __init__
    BotoSingleBackend.__init__(self, parsed_url)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 161, in __init__
    self.resetConnection()
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 191, in resetConnection
    self.bucket = self.conn.create_bucket(self.bucket_name)
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 504, in create_bucket
    response.status, response.reason, body)
S3ResponseError: S3ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>IllegalLocationConstraintException</Code><Message>The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.</Message><RequestId>XXXXXXXXX</RequestId><HostId>XXXXXXXXX</HostId></Error>

23:19:41.425 Task 'STATUS' failed with exit code '30'.
--- Finished state FAILED 'code 30' at 23:19:41.425 - Runtime 00:00:00.282 ---

---------------------------------------

This should simplify the code and fix the problem as well.

Richard (rimcaw) wrote :

Something is missing:

  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 162, in __init__
    self.my_location = Location.EU
NameError: global name 'Location' is not defined

Richard (rimcaw) wrote :

In resetConnection(), is my_location self.my_location ?
Also:
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 194, in resetConnection
    location=self.my_location)
TypeError: get_bucket() got an unexpected keyword argument 'location'

Blasted JIT imports got me again!

Marcos BL (marcosbl) wrote :

Could you please give some guidance/point to instructions on how to edit files/apply correct patch ?

We are right now unable to create nor download any backups, using duplicity-team ppa, at duplicity 0.7.07, but I'm more than happy to manually edit if that brings a working system until a fix is propagated

Thanks in advance

Download full text (5.0 KiB)

The ppa is up-to-date now. The latest version is in there.

https://code.launchpad.net/~duplicity-team/+recipe/duplicity-daily

On Tue, Apr 12, 2016 at 6:08 PM, Marcos BL <email address hidden> wrote:

> Could you please give some guidance/point to instructions on how to edit
> files/apply correct patch ?
>
> We are right now unable to create nor download any backups, using
> duplicity-team ppa, at duplicity 0.7.07, but I'm more than happy to
> manually edit if that brings a working system until a fix is propagated
>
> Thanks in advance
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1568677
>
> Title:
> duplicity fails to use existing S3 bucket in boto backend
>
> Status in Duplicity:
> Fix Committed
>
> Bug description:
> I recently updated to duplicity 0.7.07 and suddenly my backup to S3
> fails. I believe, I managed to trace the issue to
> duplicity/backends/_boto_single.py. I am running
>
> /usr/bin/duplicity --name duply_system --verbosity 9 / s3://s3-eu-
> central-1.amazonaws.com/my_bucket/duplicity
>
> with the environment variables
>
> S3_USE_SIGV4=True
> AWS_ACCESS_KEY_ID=MYACCESSKEY
> AWS_SECRET_ACCESS_KEY=MYSECRET
>
> This fails with the output given below (line number in _boto_single.py
> approximate as I added some debug output). I managed to reconstruct:
> In my case, BotoBackend._list fails as self.bucket is None. Before,
> BotoBackend.resetConnection is called, which sets self.bucket to None.
> Inside BotoBackend.resetConnection, self.bucket is only set to a valid
> bucket in case the bucket does not yet exist (in which case it is
> created via self.conn.create_bucket). In case the bucket already
> exists, self.bucket is left None. No other attempt inside BotoBackend
> is made to set self.bucket, eventually letting BotoBackend._list fail.
>
> Output of call:
>
> Using archive dir: /root/.cache/duplicity/duply_system
> Using backup name: duply_system
> Import of duplicity.backends.acdclibackend Succeeded
> Import of duplicity.backends.azurebackend Succeeded
> Import of duplicity.backends.b2backend Succeeded
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.copycombackend Succeeded
> Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.giobackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.hubicbackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.lftpbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.mediafirebackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.multibackend Succeeded
> Import of duplicity.backends.ncftpbackend Succeeded
> Import of duplicity.backends.onedrivebackend Succeeded
> Import of duplicity.backends.par2backend Succeeded
> Import of duplicity.backends.pydrivebacken...

Read more...

Marcos BL (marcosbl) wrote :

My fault, had to add ppa:duplicity-team/duplicity-daily

Backups seems to work fine again, thank you very much!

Any plans for a tarball release with just this fix (perhaps 0.7.07.1 or something like that)? Or is the fix going to have to wait for the full 0.7.08 release?

Download full text (4.8 KiB)

No plans for a point release. We have the trunk and 2 PPAs with the new
code.

On Sun, Apr 17, 2016 at 10:15 AM, Byron Clark <email address hidden>
wrote:

> Any plans for a tarball release with just this fix (perhaps 0.7.07.1 or
> something like that)? Or is the fix going to have to wait for the full
> 0.7.08 release?
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1568677
>
> Title:
> duplicity fails to use existing S3 bucket in boto backend
>
> Status in Duplicity:
> Fix Committed
>
> Bug description:
> I recently updated to duplicity 0.7.07 and suddenly my backup to S3
> fails. I believe, I managed to trace the issue to
> duplicity/backends/_boto_single.py. I am running
>
> /usr/bin/duplicity --name duply_system --verbosity 9 / s3://s3-eu-
> central-1.amazonaws.com/my_bucket/duplicity
>
> with the environment variables
>
> S3_USE_SIGV4=True
> AWS_ACCESS_KEY_ID=MYACCESSKEY
> AWS_SECRET_ACCESS_KEY=MYSECRET
>
> This fails with the output given below (line number in _boto_single.py
> approximate as I added some debug output). I managed to reconstruct:
> In my case, BotoBackend._list fails as self.bucket is None. Before,
> BotoBackend.resetConnection is called, which sets self.bucket to None.
> Inside BotoBackend.resetConnection, self.bucket is only set to a valid
> bucket in case the bucket does not yet exist (in which case it is
> created via self.conn.create_bucket). In case the bucket already
> exists, self.bucket is left None. No other attempt inside BotoBackend
> is made to set self.bucket, eventually letting BotoBackend._list fail.
>
> Output of call:
>
> Using archive dir: /root/.cache/duplicity/duply_system
> Using backup name: duply_system
> Import of duplicity.backends.acdclibackend Succeeded
> Import of duplicity.backends.azurebackend Succeeded
> Import of duplicity.backends.b2backend Succeeded
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.copycombackend Succeeded
> Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.giobackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.hubicbackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.lftpbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.mediafirebackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.multibackend Succeeded
> Import of duplicity.backends.ncftpbackend Succeeded
> Import of duplicity.backends.onedrivebackend Succeeded
> Import of duplicity.backends.par2backend Succeeded
> Import of duplicity.backends.pydrivebackend Succeeded
> Import of duplicity.backends.rsyncbackend Succeeded
> Import of duplicity.backends.ssh_paramiko_backend Succeeded
> Import of duplicity.backends.ssh_pexpect_backend Succeeded
> Import of duplic...

Read more...

Changed in duplicity:
milestone: 0.7.08 → 0.7.07.1
Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers