Duplicity fails with UnicodeDecodeError in uexc function

Bug #1770929 reported by ettoretommasomarchetti on 2018-05-13
218
This bug affects 53 people
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Unassigned
duplicity (Ubuntu)
High
Unassigned

Bug Description

backup worked under Ubuntu 17.10
It also worked on Ubuntu 18.4 before last week's update (run on Friday 11.5)
my drive is encrypted
this is the error message in the error frame
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1523, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 37: ordinal not in range(128)

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: duplicity 0.7.17-0ubuntu1
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Uname: Linux 4.15.0-20-generic x86_64
ApportVersion: 2.20.9-0ubuntu7
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Sun May 13 09:55:00 2018
InstallationDate: Installed on 2018-05-01 (11 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
SourcePackage: duplicity
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in duplicity (Ubuntu):
status: New → Confirmed
Ahmad Amr (aamr) wrote :

Failed with an unknown error.

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1523, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 37: ordinal not in range(128)

Changed in duplicity:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Kenneth Loafman (kenneth-loafman)
milestone: none → 0.8.00
K. Tibor (ha3flt) wrote :

I had an error with the same look in the last couple of days with the daily backup:

"Failed with an unknown error.
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)"
etc.

I found my external drives became read-only. I have a dual-boot Win10/Ubuntu system. Reboots/clean exit of Windows did not help, drives were still read only, and the backup process remained broken, but I realized that the Fast Boot option is reactivated. It should have been the April 2018 Windows update I just installed some days before... After I turned Fast Boot back off, backup has completed, seems to be perfect again.

Sergei (markovs-i-mail) wrote :

Affected by ascii error too.

Bud Roth (newazadude) wrote :

I am experiencing this problem too. I have an English language Ubuntu 18.04 install (fresh) with Japanese language added.

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 37: ordinal not in range(128)

Andrej Kapuš (andrej-kapus) wrote :

Hello all,

same issue here with Slovak environment and user with sudo rights.
I found workaround by changing storage location to folder, which is located in home directory and thus user has direct access to it.

It looks like access rights issues.

Andrej

Nick Meyne (nick-meyne) wrote :

Hello All - me too - identical error output to Bud Roth above. Not enough space available to try Andrej's workaround, unfortunately.
Nick

Juha Mäkeläinen (jpek) wrote :

May be connected to #1440372 – please port deja-dup to python3

Sebastien Bacher (seb128) wrote :

Similar to what is described also on https://bugzilla.redhat.com/show_bug.cgi?id=1470873

Robert R (rrstites) wrote :

This is the error message I receive when trying to use backup:

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 563, in full_backup
    bytes_written = dummy_backup(tarblock_iter)
  File "/usr/bin/duplicity", line 236, in dummy_backup
    while tarblock_iter.next():
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 523, in next
    result = self.process(self.input_iter.next())
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 195, in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 286, in collate2iters
    relem2 = riter2.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 237, in sigtar2path_iter
    tf = util.make_tarfile("r", sigtarobj)
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 123, in make_tarfile
    tf = tarfile.TarFile("arbitrary", mode, fp)
  File "/usr/lib/python2.7/tarfile.py", line 1543, in __init__
    self.name = os.path.abspath(name) if name else None
  File "/usr/lib/python2.7/posixpath.py", line 364, in abspath
    cwd = os.getcwd()
 OSError: [Errno 2] No such file or directory

summary: - backup fails on ubuntu 18.4
+ backup fails with UnicodeDecodeError

Same here unbuntu 18.04 lts french

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 392, in inner_retry
    util.uexc(e)), code=code, extra=extra)
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 36: ordinal not in range(128)

summary: - backup fails with UnicodeDecodeError
+ backup fails with UnicodeDecodeError in uexc function
summary: - backup fails with UnicodeDecodeError in uexc function
+ Duplicity fails with UnicodeDecodeError in uexc function

Thank you everybody for your bug reports and comments.

I believe that this bug is already fixed in the 0.8-series, but we have not yet released a package for this.

The code triggering this is used when there is an exception/error message containing a non-ASCII character (for example a filename), so this is sometimes 'fixed' by fixing the underlying problem. This is why some commenters say the problem went away when they resolved permissions issues. I do not know what has changed that is causing this to break for people it did not break for previously.

The reality is that version 0.7 and below were never actually written to accommodate non-ASCII characters and the fact it worked at all was largely luck. The 0.8 series does add proper support for non-ASCII, with the internals converted to Unicode and Python 3 support.

Many thanks to Pete Zaitcev (zaitcev) in Bug #1797928 for proposing a patch for this, which I had overlooked.

I have applied this in the branch here: lp:~aaron-whitehouse/duplicity/07-uexc-fix

It would be helpful if you can please test to see if this fixes your problem as follows:
1. Create a new destination directory to test with.
2. Test that duplicity fails as you have reported when using this destination.
3. Pull down a copy of the branch with the fix:
bzr branch lp:~aaron-whitehouse/duplicity/07-uexc-fix dup_patched
4. cd dup_patched/duplicity
5. run ./compilec.py
6. change directory back to wherever you were running duplicity from before
5. Test with the patched duplicity:
PYTHONPATH=[full path/]dup_patched [full path/]dup_patched/bin/duplicity [all of your options/arguments]

Please let me know how you get on.

== Simple illustration of failure ==
A simple example triggering this bug is to create a couple of empty directories, say "source1" and "dest".

Then try:
duplicity --exclude=bänana source1/ file://dest/

This should give a FilePrefixError (hideous in version 0.7 and much more informative in 0.8), but instead says:
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1611, in <module>
    if "Forced assertion for testing" in util.uexc(e):
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

The 0.8-series was affected by this (as I mentioned above, fixing the underlying problem avoids this error and a number of these have been fixed, so it is harder to trigger).

I have just proposed a fix for it be merged into the 0.8-series:
https://code.launchpad.net/~aaron-whitehouse/duplicity/08-uexc-fix

I would still appreciate any feedback on the 0.7-series branch above. I am more nervous about making changes in our stable branch without feedback.

Changed in duplicity:
status: In Progress → Fix Committed
Joachim R. (jro) wrote :

Hello,
what is needed for the fix to land in ubuntu's duplicity package ? This bug has bee fixed mainstream and the fix is available in other distro, so if any help needed, please tell me where to start.

Sebastien Bacher (seb128) wrote :

Let's try to land the change in Ubuntu disco and see how it's going there first before condidering SRUing

Changed in duplicity (Ubuntu):
status: Confirmed → Fix Committed
importance: Undecided → High
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package duplicity - 0.7.18.2-1ubuntu3.1

---------------
duplicity (0.7.18.2-1ubuntu3.1) disco; urgency=medium

  * debian/patches/bzr_unicode_error.patch:
    - backport an upstream change to fix UnicodeDecodeError
      (lp: #1770929)

 -- Sebastien Bacher <email address hidden> Fri, 12 Apr 2019 16:37:20 +0200

Changed in duplicity (Ubuntu):
status: Fix Committed → Fix Released
Patrick Toschka (tishwings) wrote :

Still, or again, appearing in 0.7.19

duplicity --version
duplicity 0.7.19

deja-dup --version
deja-dup 37.0

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1581, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1567, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1419, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1537, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 577, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 459, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 458, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 347, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 37: ordinal not in range(128)

RA (ransoft) wrote :

I can also confirm that this is still happening with Ubuntu 16.04 LTS

duplicity-backup.sh v1.6.0
duplicity 0.7.19

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

-------[ Program versions ]-------
duplicity-backup.sh v1.6.0
duplicity 0.7.19
----------------------------------

LFTP version is 4.6.3a
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1581, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1567, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1419, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1441, in do_backup
    action).set_values()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 684, in set_values
    backend_filename_list = self.backend.list()
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 301: ordinal not in range(128)

----------------[ Duplicity Cleanup ]----------------
LFTP version is 4.6.3a
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1581, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1567, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1419, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1441, in do_backup
    action).set_values()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 684, in set_values
    backend_filename_list = self.backend.list()
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 301: ordinal not in range(128)

Kiarash Deyhimi (kiadey) on 2019-05-16
Changed in duplicity:
status: Fix Committed → Fix Released

All, to clarify the current position, this is fixed in the 0.8 series, but I set out the steps to test the fix in comment #15 and was waiting on testing feedback before merging the patch back into the 0.7 series. Nobody gave me any testing feedback. It is therefore not yet fixed in the 0.7 series.

It looks as though Ubuntu has separately patched this in Disco only (comment #19).

So it will not work in Bionic or Xenial.

We are now so close to a 0.8 release that I am inclined to wait for that and encourage affected users to move straight to that, rather than risk breaking people who are sticking to the 0.7 series.

Changed in duplicity:
assignee: Kenneth Loafman (kenneth-loafman) → nobody

All,

This should be fixed in 0.8.00, which has now been released. This is the first 0.8-series release, so expect issues and test it out on data you can afford to lose (if it comes to that).

I have now packaged up this version as a snap. You can install it by typing:
snap install duplicity --classic --beta
Please uninstall your current version of duplicity to ensure you are using the snap instead of one from apt etc.
(This will work out of the box on nearly all versions of Ubuntu. If you use a different distro and have never installed snaps before, visit this page:
https://docs.snapcraft.io/installing-snapd )

This includes the latest duplicity and all dependencies wrapped up in a self-contained bundle. You should not need to install anything else for this to work (including for all backends).

If you do test this, it would be very helpful if you could please comment here, even if the only feedback is that all worked as expected.

If you are giving any feedback, please also give the output of:
which duplicity
and
snap list | grep duplicity
and let me know which backends you used.

Michael Terry (mterry) wrote :

To add on to Aaron's comment, if you are coming here because you hit this bug while using deja-dup, you can install a version of deja-dup that bundles in duplicity 0.8.0, which has the fix:

snap install deja-dup --classic --candidate

Victor Douglas (jhngalt) wrote :

Hello,

I receive the same error on Kubuntu 18.04, so I purge both duplicity 0.7 and deja-dup and install the snap version of them, as instructed by Michael and Aaron. Nevertheless, I still get the same error when I run "snap run duplicity". Here's the output:

$ which duplicity
/snap/bin/duplicity

$ snap list | grep duplicity
duplicity 0.8.02 11 beta aaron-whitehouse classic

$ snap run duplicity
Traceback (innermost last):
  File "/snap/duplicity/11/bin/duplicity", line 1711, in <module>
    with_tempdir(main)
  File "/snap/duplicity/11/bin/duplicity", line 1699, in with_tempdir
    tempdir.default().cleanup()
  File "/snap/duplicity/11/lib/python2.7/site-packages/duplicity/tempdir.py", line 63, in default
    _defaultInstance = TemporaryDirectory(temproot=globals.temproot)
  File "/snap/duplicity/11/lib/python2.7/site-packages/duplicity/tempdir.py", line 135, in __init__
    log.Info(_(u"Using temporary directory %s") % self.__dir)
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 16: ordinal not in range(128)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.