s3+http URL causes assertion failure

Bug #1411803 reported by Will Dietz on 2015-01-16
54
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Unassigned

Bug Description

== Basic Info ==

$ duplicity --version
duplicity 0.7.01

$ python2 --version
Python 2.7.9

OS: Arch Linux

Target Filesystem: S3

== Problem Description ==

When attempting to backup to S3, I've traditionally used the "s3+http" URL to avoid specifying the hostname.

The target URL is: "s3+http://dup-backup/thinkdtz-will"

While this used to work, in recent duplicity this no longer works and leads to an assertion in __boto_single.py: "assert scheme == 's3'".

== Outsider's Hopefully Help Analysis ==

A bit of digging shows the problem is that urlparse is parsing the first bit of the URL as a hostname instead of the bucket name.

I believe the problem was introduced here: http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/revision/1036

The commit seems like it's just a refactoring of how uses_netloc is populated, but it also adds in a few schemes that weren't
in the original definition. I'm not sure if that's intentional but it seems wrong and probably worth checking to see if other schemes besides "s3+http" were mistakenly added.

== Changes to use_netloc ==

uses_netloc before this change can be in the lines removed here: http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/revision/1036#duplicity/backend.py

Notice how only 's3' (not gs or s3+http) is included.

However, all three protocols are added to the new uses_netloc code as seen here:

http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/revision/1036#duplicity/backends/botobackend.py

There's also a comment in backends.py that reinforces the notion that "s3+http" shouldn't be in "uses_netloc": http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/view/head:/duplicity/backend.py#L66

== Proposed Resolution ==

Removing the line in botobackend that adds "s3+http" to uses_netloc resolves the issue for me.

Changed in duplicity:
importance: Undecided → Medium
milestone: none → 0.7.02
status: New → Fix Committed
Changed in duplicity:
status: Fix Committed → Fix Released

Is this fix supposed to be released in vivid? I had to manually remove the s3 from vivid to make it work.

That was an unfinished comment:

Is this fix supposed to be released in vivid? I had to do t to backend/botobackend.py in vivid to make the s3 backend work.

# diff -u botobackend.py.bak botobackend.py
--- botobackend.py.bak 2015-04-29 10:33:21.077085751 -0700
+++ botobackend.py 2015-04-29 10:27:54.387321687 -0700
@@ -31,4 +31,4 @@
 duplicity.backend.register_backend("gs", BotoBackend)
 duplicity.backend.register_backend("s3", BotoBackend)
 duplicity.backend.register_backend("s3+http", BotoBackend)
-duplicity.backend.uses_netloc.extend([ 'gs', 's3', 's3+http' ])
+duplicity.backend.uses_netloc.extend([ 'gs' ]) #, 's3', 's3+http' ])

Hi,

Just to confirm the issue originally reported and the observation from Warren in Vivid. The bug is already fixed in duplicity; it's just that vivid is behind (7.01 instead of 7.02).

Should someone wish to take another part than lauching their fav editor and update botobackend.py, an alternative is to add the duplicity ppa. Side note - *at this time* the duplicity ppa isn't vivid aware, so you have to configure the apt source to use the utopic update stream:

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu utopic main

sudo apt-get update
sudo apt-get upgrade

Apt should tell you duplicity can be updated.

Download full text (3.2 KiB)

Both daily and releases are vivid aware now. Daily has the latest trunk
now. Releases will get 0.7.03 when it comes out (soon).

On Thu, Apr 30, 2015 at 10:40 PM, François Jacques <
<email address hidden>> wrote:

> Hi,
>
> Just to confirm the issue originally reported and the observation from
> Warren in Vivid. The bug is already fixed in duplicity; it's just that
> vivid is behind (7.01 instead of 7.02).
>
> Should someone wish to take another part than lauching their fav editor
> and update botobackend.py, an alternative is to add the duplicity ppa.
> Side note - *at this time* the duplicity ppa isn't vivid aware, so you
> have to configure the apt source to use the utopic update stream:
>
> deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu utopic main
>
> sudo apt-get update
> sudo apt-get upgrade
>
> Apt should tell you duplicity can be updated.
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1411803
>
> Title:
> s3+http URL causes assertion failure
>
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> Fix Released
>
> Bug description:
> == Basic Info ==
>
> $ duplicity --version
> duplicity 0.7.01
>
> $ python2 --version
> Python 2.7.9
>
> OS: Arch Linux
>
> Target Filesystem: S3
>
>
> == Problem Description ==
>
> When attempting to backup to S3, I've traditionally used the "s3+http"
> URL to avoid specifying the hostname.
>
> The target URL is: "s3+http://dup-backup/thinkdtz-will"
>
> While this used to work, in recent duplicity this no longer works and
> leads to an assertion in __boto_single.py: "assert scheme == 's3'".
>
>
> == Outsider's Hopefully Help Analysis ==
>
> A bit of digging shows the problem is that urlparse is parsing the
> first bit of the URL as a hostname instead of the bucket name.
>
> I believe the problem was introduced here: http://bazaar.launchpad.net
> /~duplicity-team/duplicity/0.7-series/revision/1036
>
> The commit seems like it's just a refactoring of how uses_netloc is
> populated, but it also adds in a few schemes that weren't
> in the original definition. I'm not sure if that's intentional but it
> seems wrong and probably worth checking to see if other schemes besides
> "s3+http" were mistakenly added.
>
>
> == Changes to use_netloc ==
>
> uses_netloc before this change can be in the lines removed here:
> http://bazaar.launchpad.net/~duplicity-
> team/duplicity/0.7-series/revision/1036#duplicity/backend.py
>
> Notice how only 's3' (not gs or s3+http) is included.
>
> However, all three protocols are added to the new uses_netloc code as
> seen here:
>
> http://bazaar.launchpad.net/~duplicity-
> team/duplicity/0.7-series/revision/1036#duplicity/backends/botobackend.py
>
> There's also a comment in backends.py that reinforces the notion that
> "s3+http" shouldn't be in "uses_netloc": http://bazaar.launchpad.net
> /~duplicity-
> team/duplicity/0.7-series/view/head:/duplicity/backend.py#L66
>
>
> == Proposed Resolution ==
>
> Removing the line in botobackend that adds "s3+http" to uses_netloc
> resolves the issue for me.
>
> To manage n...

Read more...

Henry Gomersall (hgomersall) wrote :

@kenneth-loafman: Stable is not yet updated with Vivid, but Daily is.

Vej (vej) wrote :

Nominating for zesty because of high priority bug #1422289, which claims to be fixed by this.

Vej (vej) wrote :

Sorry, for my last comment, this should be already included downstream.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers