s3+http URL causes assertion failure

[Will Dietz]

== Basic Info ==

$ duplicity --version
duplicity 0.7.01

$ python2 --version
Python 2.7.9

OS: Arch Linux

Target Filesystem: S3

== Problem Description ==

When attempting to backup to S3, I've traditionally used the "s3+http" URL to avoid specifying the hostname.

The target URL is: "s3+http://dup-backup/thinkdtz-will"

While this used to work, in recent duplicity this no longer works and leads to an assertion in __boto_single.py: "assert scheme == 's3'".

== Outsider's Hopefully Help Analysis ==

A bit of digging shows the problem is that urlparse is parsing the first bit of the URL as a hostname instead of the bucket name.

I believe the problem was introduced here: http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/revision/1036

The commit seems like it's just a refactoring of how uses_netloc is populated, but it also adds in a few schemes that weren't
in the original definition. I'm not sure if that's intentional but it seems wrong and probably worth checking to see if other schemes besides "s3+http" were mistakenly added.

== Changes to use_netloc ==

uses_netloc before this change can be in the lines removed here: http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/revision/1036#duplicity/backend.py

Notice how only 's3' (not gs or s3+http) is included.

However, all three protocols are added to the new uses_netloc code as seen here:

http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/revision/1036#duplicity/backends/botobackend.py

There's also a comment in backends.py that reinforces the notion that "s3+http" shouldn't be in "uses_netloc": http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/view/head:/duplicity/backend.py#L66

== Proposed Resolution ==

Removing the line in botobackend that adds "s3+http" to uses_netloc resolves the issue for me.

[Wren Turkal (wt-penguintechs-org)]

Is this fix supposed to be released in vivid? I had to manually remove the s3 from vivid to make it work.

[Wren Turkal (wt-penguintechs-org)]

That was an unfinished comment:

Is this fix supposed to be released in vivid? I had to do t to backend/botobackend.py in vivid to make the s3 backend work.

# diff -u botobackend.py.bak botobackend.py
--- botobackend.py.bak 2015-04-29 10:33:21.077085751 -0700
+++ botobackend.py 2015-04-29 10:27:54.387321687 -0700
@@ -31,4 +31,4 @@
 duplicity.backend.register_backend("gs", BotoBackend)
 duplicity.backend.register_backend("s3", BotoBackend)
 duplicity.backend.register_backend("s3+http", BotoBackend)
-duplicity.backend.uses_netloc.extend([ 'gs', 's3', 's3+http' ])
+duplicity.backend.uses_netloc.extend([ 'gs' ]) #, 's3', 's3+http' ])

[François Jacques (francois-jacques)]

Hi,

Just to confirm the issue originally reported and the observation from Warren in Vivid. The bug is already fixed in duplicity; it's just that vivid is behind (7.01 instead of 7.02).

Should someone wish to take another part than lauching their fav editor and update botobackend.py, an alternative is to add the duplicity ppa. Side note - *at this time* the duplicity ppa isn't vivid aware, so you have to configure the apt source to use the utopic update stream:

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu utopic main

sudo apt-get update
sudo apt-get upgrade

Apt should tell you duplicity can be updated.

[Kenneth Loafman (kenneth-loafman)]

Both daily and releases are vivid aware now. Daily has the latest trunk
now. Releases will get 0.7.03 when it comes out (soon).

On Thu, Apr 30, 2015 at 10:40 PM, François Jacques <
<email address hidden>> wrote:

> Hi,
>
> Just to confirm the issue originally reported and the observation from
> Warren in Vivid. The bug is already fixed in duplicity; it's just that
> vivid is behind (7.01 instead of 7.02).
>
> Should someone wish to take another part than lauching their fav editor
> and update botobackend.py, an alternative is to add the duplicity ppa.
> Side note - *at this time* the duplicity ppa isn't vivid aware, so you
> have to configure the apt source to use the utopic update stream:
>
> deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu utopic main
>
> sudo apt-get update
> sudo apt-get upgrade
>
> Apt should tell you duplicity can be updated.
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1411803
>
> Title:
> s3+http URL causes assertion failure
>
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> Fix Released
>
> Bug description:
> == Basic Info ==
>
> $ duplicity --version
> duplicity 0.7.01
>
> $ python2 --version
> Python 2.7.9
>
> OS: Arch Linux
>
> Target Filesystem: S3
>
>
> == Problem Description ==
>
> When attempting to backup to S3, I've traditionally used the "s3+http"
> URL to avoid specifying the hostname.
>
> The target URL is: "s3+http://dup-backup/thinkdtz-will"
>
> While this used to work, in recent duplicity this no longer works and
> leads to an assertion in __boto_single.py: "assert scheme == 's3'".
>
>
> == Outsider's Hopefully Help Analysis ==
>
> A bit of digging shows the problem is that urlparse is parsing the
> first bit of the URL as a hostname instead of the bucket name.
>
> I believe the problem was introduced here: http://bazaar.launchpad.net
> /~duplicity-team/duplicity/0.7-series/revision/1036
>
> The commit seems like it's just a refactoring of how uses_netloc is
> populated, but it also adds in a few schemes that weren't
> in the original definition. I'm not sure if that's intentional but it
> seems wrong and probably worth checking to see if other schemes besides
> "s3+http" were mistakenly added.
>
>
> == Changes to use_netloc ==
>
> uses_netloc before this change can be in the lines removed here:
> http://bazaar.launchpad.net/~duplicity-
> team/duplicity/0.7-series/revision/1036#duplicity/backend.py
>
> Notice how only 's3' (not gs or s3+http) is included.
>
> However, all three protocols are added to the new uses_netloc code as
> seen here:
>
> http://bazaar.launchpad.net/~duplicity-
> team/duplicity/0.7-series/revision/1036#duplicity/backends/botobackend.py
>
> There's also a comment in backends.py that reinforces the notion that
> "s3+http" shouldn't be in "uses_netloc": http://bazaar.launchpad.net
> /~duplicity-
> team/duplicity/0.7-series/view/head:/duplicity/backend.py#L66
>
>
> == Proposed Resolution ==
>
> Removing the line in botobackend that adds "s3+http" to uses_netloc
> resolves the issue for me.
>
> To manage n...

[Henry Gomersall (hgomersall)]

@kenneth-loafman: Stable is not yet updated with Vivid, but Daily is.

[Vej (vej)]

Nominating for zesty because of high priority bug #1422289, which claims to be fixed by this.

[Vej (vej)]

Sorry, for my last comment, this should be already included downstream.