Duplicity fails with UnicodeDecodeError in uexc function

Bug #1770929 reported by ettoretommasomarchetti on 2018-05-13
172
This bug affects 42 people
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Kenneth Loafman
duplicity (Ubuntu)
Undecided
Unassigned

Bug Description

backup worked under Ubuntu 17.10
It also worked on Ubuntu 18.4 before last week's update (run on Friday 11.5)
my drive is encrypted
this is the error message in the error frame
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1523, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 37: ordinal not in range(128)

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: duplicity 0.7.17-0ubuntu1
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Uname: Linux 4.15.0-20-generic x86_64
ApportVersion: 2.20.9-0ubuntu7
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Sun May 13 09:55:00 2018
InstallationDate: Installed on 2018-05-01 (11 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
SourcePackage: duplicity
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in duplicity (Ubuntu):
status: New → Confirmed
Ahmad Amr (aamr) wrote :

Failed with an unknown error.

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1523, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 37: ordinal not in range(128)

Changed in duplicity:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Kenneth Loafman (kenneth-loafman)
milestone: none → 0.8.00
K. Tibor (ha3flt) wrote :

I had an error with the same look in the last couple of days with the daily backup:

"Failed with an unknown error.
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)"
etc.

I found my external drives became read-only. I have a dual-boot Win10/Ubuntu system. Reboots/clean exit of Windows did not help, drives were still read only, and the backup process remained broken, but I realized that the Fast Boot option is reactivated. It should have been the April 2018 Windows update I just installed some days before... After I turned Fast Boot back off, backup has completed, seems to be perfect again.

Sergei (markovs-i-mail) wrote :

Affected by ascii error too.

Bud Roth (newazadude) wrote :

I am experiencing this problem too. I have an English language Ubuntu 18.04 install (fresh) with Japanese language added.

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 37: ordinal not in range(128)

Andrej Kapuš (andrej-kapus) wrote :

Hello all,

same issue here with Slovak environment and user with sudo rights.
I found workaround by changing storage location to folder, which is located in home directory and thus user has direct access to it.

It looks like access rights issues.

Andrej

Nick Meyne (nick-meyne) wrote :

Hello All - me too - identical error output to Bud Roth above. Not enough space available to try Andrej's workaround, unfortunately.
Nick

Juha Mäkeläinen (jpek) wrote :

May be connected to #1440372 – please port deja-dup to python3

Sebastien Bacher (seb128) wrote :

Similar to what is described also on https://bugzilla.redhat.com/show_bug.cgi?id=1470873

Robert R (rrstites) wrote :

This is the error message I receive when trying to use backup:

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 563, in full_backup
    bytes_written = dummy_backup(tarblock_iter)
  File "/usr/bin/duplicity", line 236, in dummy_backup
    while tarblock_iter.next():
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 523, in next
    result = self.process(self.input_iter.next())
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 195, in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 286, in collate2iters
    relem2 = riter2.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 237, in sigtar2path_iter
    tf = util.make_tarfile("r", sigtarobj)
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 123, in make_tarfile
    tf = tarfile.TarFile("arbitrary", mode, fp)
  File "/usr/lib/python2.7/tarfile.py", line 1543, in __init__
    self.name = os.path.abspath(name) if name else None
  File "/usr/lib/python2.7/posixpath.py", line 364, in abspath
    cwd = os.getcwd()
 OSError: [Errno 2] No such file or directory

summary: - backup fails on ubuntu 18.4
+ backup fails with UnicodeDecodeError

Same here unbuntu 18.04 lts french

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 392, in inner_retry
    util.uexc(e)), code=code, extra=extra)
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 36: ordinal not in range(128)

summary: - backup fails with UnicodeDecodeError
+ backup fails with UnicodeDecodeError in uexc function
summary: - backup fails with UnicodeDecodeError in uexc function
+ Duplicity fails with UnicodeDecodeError in uexc function

Thank you everybody for your bug reports and comments.

I believe that this bug is already fixed in the 0.8-series, but we have not yet released a package for this.

The code triggering this is used when there is an exception/error message containing a non-ASCII character (for example a filename), so this is sometimes 'fixed' by fixing the underlying problem. This is why some commenters say the problem went away when they resolved permissions issues. I do not know what has changed that is causing this to break for people it did not break for previously.

The reality is that version 0.7 and below were never actually written to accommodate non-ASCII characters and the fact it worked at all was largely luck. The 0.8 series does add proper support for non-ASCII, with the internals converted to Unicode and Python 3 support.

Many thanks to Pete Zaitcev (zaitcev) in Bug #1797928 for proposing a patch for this, which I had overlooked.

I have applied this in the branch here: lp:~aaron-whitehouse/duplicity/07-uexc-fix

It would be helpful if you can please test to see if this fixes your problem as follows:
1. Create a new destination directory to test with.
2. Test that duplicity fails as you have reported when using this destination.
3. Pull down a copy of the branch with the fix:
bzr branch lp:~aaron-whitehouse/duplicity/07-uexc-fix dup_patched
4. cd dup_patched/duplicity
5. run ./compilec.py
6. change directory back to wherever you were running duplicity from before
5. Test with the patched duplicity:
PYTHONPATH=[full path/]dup_patched [full path/]dup_patched/bin/duplicity [all of your options/arguments]

Please let me know how you get on.

== Simple illustration of failure ==
A simple example triggering this bug is to create a couple of empty directories, say "source1" and "dest".

Then try:
duplicity --exclude=bänana source1/ file://dest/

This should give a FilePrefixError (hideous in version 0.7 and much more informative in 0.8), but instead says:
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1611, in <module>
    if "Forced assertion for testing" in util.uexc(e):
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

The 0.8-series was affected by this (as I mentioned above, fixing the underlying problem avoids this error and a number of these have been fixed, so it is harder to trigger).

I have just proposed a fix for it be merged into the 0.8-series:
https://code.launchpad.net/~aaron-whitehouse/duplicity/08-uexc-fix

I would still appreciate any feedback on the 0.7-series branch above. I am more nervous about making changes in our stable branch without feedback.

Changed in duplicity:
status: In Progress → Fix Committed
Joachim R. (jro) wrote :

Hello,
what is needed for the fix to land in ubuntu's duplicity package ? This bug has bee fixed mainstream and the fix is available in other distro, so if any help needed, please tell me where to start.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.