Duplicity fails to start

Bug #1855736 reported by Tim Passingham on 2019-12-09
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Duplicity
High
Unassigned
duplicity (Ubuntu)
High
Unassigned

Bug Description

I am on 19.10. Duplicity crashes as soon as I try to start a backup, as below. It was running fine on 19.04.

Traceback (innermost last):
  File "/usr/bin/duplicity", line 107, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 93, in with_tempdir
    fn()
  File "/usr/lib/python3/dist-packages/duplicity/dup_main.py", line 1522, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/duplicity/commandline.py", line 1200, in ProcessCommandLine
    globals.backend = backend.get_backend(args[0])
  File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 225, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 211, in get_backend_object
    return factory(pu)
  File "/usr/lib/python3/dist-packages/duplicity/backends/giobackend.py", line 81, in __init__
    ensure_dbus()
  File "/usr/lib/python3/dist-packages/duplicity/backends/giobackend.py", line 37, in ensure_dbus
    lines = output.split(u'\n')
 TypeError: a bytes-like object is required, not 'str'

Tim Passingham (tim-8aw3u04umo) wrote :

I notice that this problem (using deja-dup to run duplicity) only occurs when I run as sudo or pkexec (I need to do so to back up some kernel files).

You are on duplicity 0.8.04, the latest is 0.8.07, please upgrade.

There are three options:

* Release tarball Install - https://launchpad.net/duplicity/+download
* Daily duplicity builds - https://launchpad.net/~duplicity-team/+archive/ubuntu/daily-dev-trunk
* Stable snap builds - “sudo snap install duplicity —classic"

NOTE: UNinstall duplicity first if it was installed via the distribution repository. For Ubuntu, that would be "sudo apt-get purge duplicity".

Whoops, latest is 0.8.08. Please let me know if this fixes the error.

Changed in duplicity:
status: New → Incomplete
Tim Passingham (tim-8aw3u04umo) wrote :

I had looked for a ppa but failed to find yours.

Unfortunately deja-dup doesn't recognise this new version, saying "Could not understand duplicity version."

I tend to use GUIs for almost everything. I guess I will have to try to translate my deja-dup instructions into duplicity command line. This may take me quite a while.

If there is a simple fix to deja-dup (or duplicity) to get round this I'd be grateful.

Tim Passingham (tim-8aw3u04umo) wrote :

After a quick web search I tried running

duplicity --version

That told me I needed to install 'future' with pip. So I installed python-pip, future, and fasteners.

I now fail at:

duplicity --version
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 55, in <module>
    from duplicity.dup_main import main
ImportError: No module named duplicity.dup_main

And then:

pip install duplicity.dup_main
Collecting duplicity.dup_main
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/duplicity-dup-main/

Are there some comprehensive installation instructions somewhere?

How did you install it? The version should be OK for Deja Dup.

On Tue, Dec 10, 2019 at 11:45 AM Tim Passingham <email address hidden>
wrote:

> After a quick web search I tried running
>
> duplicity --version
>
> That told me I needed to install 'future' with pip. So I installed
> python-pip, future, and fasteners.
>
> I now fail at:
>
> duplicity --version
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 55, in <module>
> from duplicity.dup_main import main
> ImportError: No module named duplicity.dup_main
>
> And then:
>
> pip install duplicity.dup_main
> Collecting duplicity.dup_main
> Could not install packages due to an EnvironmentError: 404 Client Error:
> Not Found for url: https://pypi.org/simple/duplicity-dup-main/
>
> Are there some comprehensive installation instructions somewhere?
>
> --
> You received this bug notification because you are subscribed to
> duplicity in Ubuntu.
> https://bugs.launchpad.net/bugs/1855736
>
> Title:
> Duplicity fails to start
>
> Status in Duplicity:
> Incomplete
> Status in duplicity package in Ubuntu:
> New
>
> Bug description:
> I am on 19.10. Duplicity crashes as soon as I try to start a backup,
> as below. It was running fine on 19.04.
>
> Traceback (innermost last):
> File "/usr/bin/duplicity", line 107, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 93, in with_tempdir
> fn()
> File "/usr/lib/python3/dist-packages/duplicity/dup_main.py", line
> 1522, in main
> action = commandline.ProcessCommandLine(sys.argv[1:])
> File "/usr/lib/python3/dist-packages/duplicity/commandline.py", line
> 1200, in ProcessCommandLine
> globals.backend = backend.get_backend(args[0])
> File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 225,
> in get_backend
> obj = get_backend_object(url_string)
> File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 211,
> in get_backend_object
> return factory(pu)
> File
> "/usr/lib/python3/dist-packages/duplicity/backends/giobackend.py", line 81,
> in __init__
> ensure_dbus()
> File
> "/usr/lib/python3/dist-packages/duplicity/backends/giobackend.py", line 37,
> in ensure_dbus
> lines = output.split(u'\n')
> TypeError: a bytes-like object is required, not 'str'
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1855736/+subscriptions
>

Michael Terry (mterry) wrote :

I think this patch should fix the duplicity error you're seeing.

But deja-dup doesn't work well under root so much - it'll ask for root permissions as appropriate when restoring anyway?

tags: added: patch
Changed in duplicity:
status: Incomplete → In Progress
importance: Undecided → Medium
assignee: nobody → Kenneth Loafman (kenneth-loafman)
milestone: none → 0.8.09
Changed in duplicity:
status: In Progress → Won't Fix
status: Won't Fix → Fix Committed
assignee: Kenneth Loafman (kenneth-loafman) → nobody
Tim Passingham (tim-8aw3u04umo) wrote :

I'm not sure whether the fix mentioned is for the last problenm I encountered, but having updated duplicity at 20:20GMT (now version 0.8.09-0ubuntu0ppa1524~ubuntu19.10.1), when I type:

duplicity --version

I still get:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 55, in <module>
    from duplicity.dup_main import main
ImportError: No module named duplicity.dup_main

Did you remove the repo version of duplicity ($ sudo apt purge duplicity)?

If not, do the purge now, then do ($ sudo apt reinstall duplicity). It should install from the ppa.

Changed in duplicity:
status: Fix Committed → In Progress
Tim Passingham (tim-8aw3u04umo) wrote :

I had done, but just to be sure I purged it again, autoremove the un-needed items (python fasteners etc), reinstalled, rebooted, and get the same error on "duplicity --version"

Try this:

$ PYTHONPATH='.' duplicity ...

Looks like the current directory is not in the path. It should be.

Sorry to be difficult, but:

PYTHONPATH='.' duplicity --version
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 55, in <module>
    from duplicity.dup_main import main
ImportError: No module named duplicity.dup_main

You're not being difficult, the problem is vexing!

Try this:

    $ PYTHONPATH='.'duplicity/dup_main.py

It should work like this too:

    $ duplicity/dup_main.py

In either case there should be no output. Please log whatever happens. Perhaps there's a missing module or something. Python imports don't tell you a lot when they go wrong.

Sorry, that should be:

$ python3 duplicity/dup_main.py

in both cases.

I'm not quite sure what your last comment means, so here are 4 attempts.

PYTHONPATH='.' duplicity/dup_main.py --version
bash: duplicity/dup_main.py: No such file or directory

duplicity/dup_main.py --version
bash: duplicity/dup_main.py: No such file or directory

PYTHONPATH='.' python3 duplicity/dup_main.py --version
python3: can't open file 'duplicity/dup_main.py': [Errno 2] No such file or directory

python3 duplicity/dup_main.py --version
python3: can't open file 'duplicity/dup_main.py': [Errno 2] No such file or directory

The results are the same with or without --version

Sorry, my bad. You needed to be in the /usr/lib/python3.X/site-packages/ directory. This is not trying to execute duplicity, merely to import the installed duplicity/dup_main.py.

Well that may explain something. I have no python3.X directory. I have python3, and it contains dist-packages.

I did try:

/usr/lib/python3/dist-packages$ python3 duplicity/dup_main.py
Traceback (most recent call last):
  File "duplicity/dup_main.py", line 54, in <module>
    from . import asyncscheduler
ImportError: attempted relative import with no known parent package

Changed in duplicity:
status: In Progress → Won't Fix
status: Won't Fix → Fix Committed

I got that update, and things have changed a bit.

usr/lib/python3/dist-packages$ python3 duplicity/dup_main.py

returns nothing, which I think is what you wanted.

But:

/usr/lib/python3/dist-packages$ duplicity
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 55, in <module>
    from duplicity.dup_main import main
ImportError: No module named duplicity.dup_main

(the same if I repeat the command from my home directory)

Or:

/usr/lib/python3/dist-packages$ duplicity/dup_main.py
bash: duplicity/dup_main.py: Permission denied

Or:

sudo duplicity/dup_main.py
[sudo] password for ...:
sudo: duplicity/dup_main.py: command not found

As you can tell, I've no idea what to try, or what's really going on.

To be clear, I'm now on version 0.8.09-0ubuntu0ppa1526~ubuntu19.10.1 (eoan)

I rebooted after installing (just in case). I didn't purge beforehand, so I then did so, did an apt autoremove, and reinstalled it. No change.

The attachment "decode.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Do this and report the results:

$ python3 /usr/bin/duplicity --version

I think I know the problem, but want to validate.

If that fails for any reason, do these and report the results:

$ find /usr -type d -name duplicity -ls
$ find /usr -type f -name duplicity -ls

python3 /usr/bin/duplicity --version
duplicity 0.8.09

Success!

But as before:

duplicity --version
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 55, in <module>
    from duplicity.dup_main import main
ImportError: No module named duplicity.dup_main

Ditto for "/usr/bin/duplicity --version"

So something about not being explicity about python3 is now the issue?

OK, what's happening is two factor:

1) Something in setup is changing the shebang line to Python 2 and,
2) It's libraries are being installed on Python 3.

which means duplicity can't find its libraries.

Please run the following:

$ find /usr -type d -name duplicity -ls
$ find /usr -type f -name duplicity -ls

OK (I didn't do so before because the python3 command seemed to work).

find /usr -type d -name duplicity -ls
  1602725 4 drwxr-xr-x 2 root root 4096 Dec 15 20:10 /usr/share/doc/duplicity
  1602642 4 drwxr-xr-x 4 root root 4096 Dec 15 20:10 /usr/lib/python3/dist-packages/duplicity

find /usr -type f -name duplicity -ls
   658314 8 -rwxr-xr-x 1 root root 5265 Dec 15 17:37 /usr/bin/duplicity

I would have replied earlier but I don't seem to be getting emails regarding comments as quickly as I used to (in fact none about your last 2 posts).

OK, execute the following to fix the problem:

$ sudo sed -i s/python2/python3/g /usr/bin/duplicity

This will make duplicity run under Python3.

$ duplicity --version

should respond with: duplicity 0.8.09

Yes, that's fixed. Thanks very much.

Now all I have to do is get deja dup to work:

Traceback (innermost last):
  File "/usr/bin/duplicity", line 101, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 87, in with_tempdir
    fn()
  File "/usr/lib/python3/dist-packages/duplicity/dup_main.py", line 1526, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/duplicity/commandline.py", line 1172, in ProcessCommandLine
    globals.backend = backend.get_backend(args[0])
  File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 225, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 211, in get_backend_object
    return factory(pu)
  File "/usr/lib/python3/dist-packages/duplicity/backends/giobackend.py", line 82, in __init__
    ensure_dbus()
  File "/usr/lib/python3/dist-packages/duplicity/backends/giobackend.py", line 38, in ensure_dbus
    lines = output.decode(u'utf8').split(u'\n')
 AttributeError: 'str' object has no attribute 'decode'

Changed in duplicity (Ubuntu):
importance: Undecided → High
status: New → Fix Committed

Which particular fix has been applied?

The shebang points to python2 on all scripts. It was my fault. setup pointed to python3, duplicity pointed to python2. That caused all the problems.

Thanks very much. I'm glad I was able to help a little by testing things.

Have you any thoughts about why deja-dup fails as I reported in #26? Do I need to raise a separate report somewhere else?

Changed in duplicity:
importance: Medium → High

A few more deja-dup issues:

deja-dup (brings up GUI, I select backup...)
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 55, in <module>
    from duplicity.dup_main import main
ImportError: No module named duplicity.dup_main

sudo deja-dup (brings up GUI, I select backup...)
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 31, in <module>
    from future import standard_library
ImportError: No module named future

So:

pip install future
Requirement already satisfied: future in ./.local/lib/python2.7/site-packages (0.18.2)

But isn't in python3 now?

Michael Terry (mterry) wrote :

Tim, your issue in comment 26 is confusing to me. It looks like it's caused by my patch in comment 6?

Specifically, it looks like subprocess.Popen returns a unicode string as stdout for you. I can't reproduce locally, in either python2.7 or python3.7:

>>> import subprocess
>>> subprocess.Popen([u'ls'], stdout=subprocess.PIPE).communicate()[0].decode('utf8')

The documentation for Popen says:
"If encoding or errors are specified, or text is true, the file objects stdin, stdout and stderr are opened in text mode with the specified encoding and errors, as described above in Frequently Used Arguments. The universal_newlines argument is equivalent to text and is provided for backwards compatibility. By default, file objects are opened in binary mode."

None of the arguments that would cause Popen to return a unicode string seem to be passed to it by duplicity...?

It is indeed strange, because I can't reproduce it now either. I get the errors in #30 instead, with deja-dup saying it can't understand the duplicity version.

I think what may have happened is that because I couldn't get deja-dup to work I used the duplicity command line, and encrypted this particular backup. I don't normally encrypt this backup (which is mainly system files, config files etc). Is 'decode' something to do with encryption?

Once duplicity seemed to be sorted I redid the backup without encryption, and now I get the issues in #30.

With the latest update duplicity still fails. Version 0.8.09-0ubuntu0ppa1531-ubuntu19.10.1 (eoan).

I had to redo the sed from #25.

Which way does it fail? I thought we had that fixed for you.

It was fixed. My system later updated to version 1531 as above, and this then failed in the same way as before. I had to apply your sed patch to make it work.

If you redid the sed from #25, that caused the problem. You need to reinstall and let it run under Python 2.

Sadly not. I purged, installed, and it failed as before. I had to redo the sed.

1532 this morning fixed it. Thanks! :-)

With version 0.8.09-0ubuntu0ppa1535-ubuntu19.10.1 (eoan) deja-dup now seems to start to work as a normal user, but I need to backup some root stuff, and when run under sudo or pkexec I get the following immediately:

Traceback (innermost last):
  File "/usr/bin/duplicity", line 101, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 87, in with_tempdir
    fn()
  File "/usr/lib/python3/dist-packages/duplicity/dup_main.py", line 1525, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/duplicity/commandline.py", line 1172, in ProcessCommandLine
    globals.backend = backend.get_backend(args[0])
  File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 225, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 211, in get_backend_object
    return factory(pu)
  File "/usr/lib/python3/dist-packages/duplicity/backends/giobackend.py", line 82, in __init__
    ensure_dbus()
  File "/usr/lib/python3/dist-packages/duplicity/backends/giobackend.py", line 38, in ensure_dbus
    lines = output.decode(u'utf8').split(u'\n')
 AttributeError: 'str' object has no attribute 'decode'

Changed in duplicity:
status: Fix Committed → Fix Released

Thanks very much indeed. I really appreciate all the work you and others do in supporting us, the users.

It all works fine with deja-dup now. I'm now on duplicity version 0.8.09-0ubuntu0ppa1542-ubuntu19.10.1 (eoan), and deja-dup version 40.1-1ubuntu2

Should I close this report myself, or leave that you you?

You're welcome. The report is closed.

I had/have similar issue as "resolved" in comment #25:

Initially, after building and installing version 0.8.09, I would try
duplicity --version
and get
Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 55, in <module>
    from duplicity.dup_main import main
ImportError: No module named 'duplicity'

I came upon this bug, saw comment #25 where it was suggested to change
the shebang to 'python3'. I checked, and mine was ALREADY saying 'python3', but
I changed it to 'python2' and then
duplicity --version responds correctly with
duplicity 0.8.09

If it helps, one of the first messages from running
$ sudo python setup.py install --prefix=/usr/local
is
/usr/lib/python2.7/dist-packages/setuptools/dist.py:333: UserWarning: Normalizing '0.8.09' to '0.8.9'
  normalized_version,

So it appears to be using python2 (2.7, to be exact) "during the build"?

Let me know if you'd like another bug open, or if there already is one to improve build scripts?
OR, better yet, maybe your "fix" was a fix for 0.8.10 and it has already been taken care of?

Thanks,

If it helps, the output of the 'find' commands, mentioned prior to comment #25 are as you might guess:

$ sudo find /usr -type d -name duplicity -ls
  1471833 4 drwxr-xr-x 3 root root 4096 Jun 9 2017 /usr/lib/python2.7/dist-packages/duplicity
    54981 4 drwxr-xr-x 3 root root 4096 Jan 10 20:01 /usr/local/lib/python2.7/dist-packages/duplicity

$ sudo find /usr -type f -name duplicity -ls
    55524 8 -rwxr-xr-x 1 root root 5276 Jan 10 20:38 /usr/local/bin/duplicity
  1177994 64 -rwxr-xr-x 1 root root 62300 Jan 10 20:19 /usr/bin/duplicity

Grzegorz Jeszka (gjeszka) wrote :

Terrible app. You have backup complite, but when You really need use it:
deja-dup file "/usr/bin/duplicity", line 107, in <module>

Kubuntu 19.10

All of this Opensource must works like that? This is not first time when: python - Duplicity error while restoring - Ask Ubuntu

Grzegorz Jeszka (gjeszka) wrote :

Maybe this app need a button to test environment compatibility. Just for be sure.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers