deja-dup/duplicity backup fails with UnicodeDecodeError

Bug #1736164 reported by Gwyn Ciesla on 2017-12-04
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Duplicity
Undecided
Unassigned
Déjà Dup
High
Unassigned
deja-dup (Fedora)
In Progress
Medium

Bug Description

See: https://bugzilla.redhat.com/show_bug.cgi?id=1470873

Description of problem:
With the update from Fedora 25 to Fedora 26 Workstation, backups with deja-dup/duplicity are failing

Version-Release number of selected component (if applicable):
duplicity 0.7.13.1
deja-dup 34.3

How reproducible:
Always (scheduled and manually started backup)

Steps to Reproduce:
1. Wait for the scheduled backup to start or start it via the "Backup now" button
2. Some or a large number of files might get backed up, then the backup stops with "Failed with an unknown error"
3.

Actual results:
Backup fails

Expected results:
Backup completes as it always did

Additional info:
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1516, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 746: ordinal not in range(128)

Description of problem:
With the update from Fedora 25 to Fedora 26 Workstation, backups with deja-dup/duplicity are failing

Version-Release number of selected component (if applicable):
duplicity 0.7.13.1
deja-dup 34.3

How reproducible:
Always (scheduled and manually started backup)

Steps to Reproduce:
1. Wait for the scheduled backup to start or start it via the "Backup now" button
2. Some or a large number of files might get backed up, then the backup stops with "Failed with an unknown error"
3.

Actual results:
Backup fails

Expected results:
Backup completes as it always did

Additional info:
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1516, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 746: ordinal not in range(128)

Same problem here, also on Fedora 26, when trying to perform a backup:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1509, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 748: ordinal not in range(128)

I see this too:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1516, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 746: ordinal not in range(128)

I have same error with Fedora 26 with

duplicity 0.7.13.1
deja-dup 35.5

I've find a solution that work for me in this page

https://bugs.launchpad.net/duplicity/+bug/1386373

He say: "This error can be easily fixed by replacing body of the uexc function in the utils module with the following statement:
return ufn(unicode(e).encode('utf-8'))"

(In reply to <email address hidden> from comment #3)
> I have same error with Fedora 26 with
>
> duplicity 0.7.13.1
> deja-dup 35.5
>
> I've find a solution that work for me in this page
>
> https://bugs.launchpad.net/duplicity/+bug/1386373
>
> He say: "This error can be easily fixed by replacing body of the uexc
> function in the utils module with the following statement:
> return ufn(unicode(e).encode('utf-8'))"

I correct myself.

The solution doesn't work well.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1504, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 746: ordinal not in range(128)

Download full text (4.0 KiB)

I'm having the same bug/error:

[---@+++ ~]$ uname -a
Linux +++.*** 4.12.14-300.fc26.x86_64 #1 SMP Wed Sep 20 16:28:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[---@+++ ~]$ dnf info deja-dup
Última verificação de expiração de metadados: 10 days, 2:25:23 em seg 18 set 2017 17:26:58 WEST.
Pacotes Instalados
Nome : deja-dup
Versão : 36.1
Lançamento : 1.fc26
Arq. : x86_64
Tamanho : 4.0 M
Fonte : deja-dup-36.1-1.fc26.src.rpm
Repositório : @System
Do repositór : updates
Resumo : Simple backup tool and frontend for duplicity
URL : https://launchpad.net/deja-dup
Licença : GPLv3+
Descrição : Déjà Dup is a simple backup tool. It hides the complexity of
             : doing backups the 'right way' (encrypted, off-site, and regular)
             : and uses duplicity as the backend.
             :
             : Features:
             : • Support for local, remote, or cloud backup locations (Amazon
             : S3 or Rackspace) • Securely encrypts and compresses your data
             : • Incrementally backs up, letting you restore from any
             : particular backup • Schedules regular backups
             : • Integrates well into your GNOME desktop

[---@+++ ~]$ dnf info duplicity
Última verificação de expiração de metadados: 10 days, 2:26:58 em seg 18 set 2017 17:26:58 WEST.
Pacotes Instalados
Nome : duplicity
Versão : 0.7.13.1
Lançamento : 2.fc26
Arq. : x86_64
Tamanho : 2.4 M
Fonte : duplicity-0.7.13.1-2.fc26.src.rpm
Repositório : @System
Do repositór : updates
Resumo : Encrypted bandwidth-efficient backup using rsync algorithm
URL : http://www.nongnu.org/duplicity/
Licença : GPLv2+
Descrição : Duplicity incrementally backs up files and directory by encrypting
             : tar-format volumes with GnuPG and uploading them to a remote (or
             : local) file server. In theory many protocols for connecting to a
             : file server could be supported; so far ssh/scp, local file access,
             : rsync, ftp, HSI, WebDAV and Amazon S3 have been written.
             :
             : Because duplicity uses librsync, the incremental archives are space
             : efficient and only record the parts of files that have changed since
             : the last backup. Currently duplicity supports deleted files, full
             : unix permissions, directories, symbolic links, fifos, device files,
             : but not hard links.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1515, in do_backup
    check_last_manifest(col_stats) # not needed for full backup
  File "/usr/bin/duplicity", line 1219, in check_last_manifest
    last_backup_set.check_manifests()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 199, in check_manifests
    remote_manifest = self.get_remote_manifest()
  File "/usr/lib64/...

Read more...

I tried running
 DEJA_DUP_DEBUG=1 deja-dup --backup | tee ~/deja-dup-backup-debug.log
hoping the debug log would show which file would contain the 0xe2 byte in its filename so that I could delete or at least exclude it.
But although the debug.log is rather verbose it wouldn't tell which file made it fail.

Is there a way to make deja-dup reveal what file brings it down?

The same issue with duplicity 0.7.13.1 on CentOS7.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1406, in do_backup
    sync_archive(decrypt)
  File "/usr/bin/duplicity", line 1146, in sync_archive
    remlist = globals.backend.list()
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 836: ordinal not in range(128)

This is the function itself and a nice comment:
def uexc(e):
    # Exceptions in duplicity often have path names in them, which if they are
    # non-ascii will cause a UnicodeDecodeError when implicitly decoding to
    # unicode. So we decode manually, using the filesystem encoding.
    # 99.99% of the time, this will be a fine encoding to use.
    e = unicode(e).encode('utf-8')
    return ufn(str(e))

The problem persists in Fedora 27 Workstation:

$ uname -a
Linux hostname 4.13.16-300.fc27.x86_64 #1 SMP Mon Nov 27 18:19:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ rpm -qi deja-dup
Name : deja-dup
Version : 37.0
Release : 1.fc27
Architecture: x86_64
Install Date: Fri 01 Dec 2017 04:52:44 PM CET
Group : Applications/Archiving
Size : 4213682
License : GPLv3+
Signature : RSA/SHA256, Mon 20 Nov 2017 04:25:27 PM CET, Key ID f55e7430f5282ee4
Source RPM : deja-dup-37.0-1.fc27.src.rpm
Build Date : Mon 20 Nov 2017 04:03:58 PM CET
Build Host : buildvm-30.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager : Fedora Project
Vendor : Fedora Project
URL : https://launchpad.net/deja-dup

$ rpm -qi duplicity
Name : duplicity
Version : 0.7.15
Release : 1.fc27
Architecture: x86_64
Install Date: Tue 28 Nov 2017 09:11:38 AM CET
Group : Unspecified
Size : 2476520
License : GPLv2+
Signature : RSA/SHA256, Tue 14 Nov 2017 03:22:29 PM CET, Key ID f55e7430f5282ee4
Source RPM : duplicity-0.7.15-1.fc27.src.rpm
Build Date : Tue 14 Nov 2017 03:16:59 PM CET
Build Host : buildvm-30.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager : Fedora Project
Vendor : Fedora Project
URL : http://www.nongnu.org/duplicity/

Filed bug upstream.

Vej (vej) wrote :

Setting to confirmed because of duplicate.

Changed in deja-dup:
status: New → Confirmed
importance: Undecided → High
status: Confirmed → Triaged
Changed in deja-dup (Fedora):
importance: Unknown → Medium
status: Unknown → In Progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.