deja-dup fails to restore missing files, that contain "german umlaut" (ä,ö,ü,Ä,Ö,Ü).

Bug #1377873 reported by Vej on 2014-10-06
98
This bug affects 20 people
Affects Status Importance Assigned to Milestone
Déjà Dup
Undecided
Unassigned
deja-dup (Ubuntu)
High
Unassigned

Bug Description

I tried to restore some "missing files" (translated from german) deleted while deja-dup was creating a new backup.
This attempt failed with the following Stacktrace:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1494, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1488, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1337, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1422, in do_backup
    restore(col_stats)
  File "/usr/bin/duplicity", line 700, in restore
    % (globals.restore_dir,),
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 45: ordinal not in range(128)

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: deja-dup 30.0-0ubuntu4
ProcVersionSignature: Ubuntu 3.13.0-36.63-generic 3.13.11.6
Uname: Linux 3.13.0-36-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
CurrentDesktop: Unity
Date: Mon Oct 6 12:01:55 2014
EcryptfsInUse: Yes
InstallationDate: Installed on 2014-09-28 (7 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: deja-dup
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Vej (vej) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in deja-dup (Ubuntu):
status: New → Confirmed
bongo (bong0) wrote :

Nearly the same issue here, running on archlinux though with Version 32.0-1

Björn Engel (bengel-lued) wrote :

I have the same problem, but with the letters "Ö" and "ü".

Vej (vej) on 2015-02-10
summary: - deja-dup fails to restore missing files
+ deja-dup fails to restore missing files, that contain "german umlaut"
+ (ä,ö,ü,Ä,Ö,Ü).
Florian Stoll (flostoll) wrote :

My short-term workaround to recover old backup files with umlauts after UnicodeDecodeError:

export LANG=en_US.utf8
export LANGUAGE=
export LC_CTYPE="en_US.utf8"
export LC_NUMERIC="en_US.utf8"
export LC_TIME="en_US.utf8"
export LC_COLLATE="en_US.utf8"
export LC_MONETARY="en_US.utf8"
export LC_MESSAGES="en_US.utf8"
export LC_PAPER=en_US.UTF-8
export LC_NAME="en_US.utf8"
export LC_ADDRESS="en_US.utf8"
export LC_TELEPHONE="en_US.utf8"
export LC_MEASUREMENT="en_US.utf8"
export LC_IDENTIFICATION="en_US.utf8"
export LC_ALL=

deja-dup --restore folderWithUmlauts

maybe someone should set the Importance. It seems that this bug will affect all German, Danish, Norwegian and Swedish user and can result in Data corruption?

chris (chris-hn) wrote :

..the same here under Ubuntu 14.04 (64.-bit).
Trying to restore some files including the german "Umlaut" (that means the letters ä;ü;ö....) ends with the error message from the first thread. Maybe, it should be checked if it's the same with some french letters like ê,ë,é,è, ç.

Otherwise, with this bug it is impossible to use this good software for german, french and northern users.
It sould be fixed asap...

Vej (vej) wrote :

Hello.

I want to bring to your knowledge, that we (in the german-commmunity) still see a lot of people having this problem.

If you want some reference, you can find a new theme here: https://forum.ubuntuusers.de/topic/datensicherung-fehlgeschlagen-was-tun (in german).

With this bug, I can't recommend usage of Déjà-Dup for anyone speaking german. This is very bad, because this problem is the preinstalled software in Ubuntu and used to be the most user-friendly software in the past.

Please set the priority and fix it or tell me, what to do against it.

Best Regards

Vej

(Supporter of the (unofficial) german supportforum for backups: https://forum.ubuntuusers.de/forum/backups/)

PS.: If you want to have some discussion, feel free to contact me via my profile.

Damian Kunkis (damian-im-web) wrote :

Same with me as with Chris - so at the moment some of my backed-up files seem to be "lost".... :(

This Problem is since more as one year, incredible...

ITEAS (info-tux-pc) wrote :

Same problem! Unuseable for me!

Please fix it.

Thanks and best Regards

Martin Pitt (pitti) on 2016-01-07
Changed in deja-dup (Ubuntu):
importance: Undecided → High
Michael Terry (mterry) wrote :

This hasn't been on my radar, sorry. This looks more like a duplicity bug than a deja-dup one, will reassign. I might get some time to look at it later this week.

Michael Terry (mterry) wrote :

OK.... So this bug is a dup of 1356548. It's fixed upstream and in versions of Ubuntu past 14.04.

I'm going to mark this bug as a dup and consider backporting via SRU. Discussion can continue in that bug.

Felix Haller (felixhaller) wrote :

It seems this is not an exact duplicate of bug #1356548.

Even after that bug is fixed upstream it is not possible to/leads to errors when trying to do the following

1. go to the folder you backed up before (must contain a folder/file with umlauts e.g. "töstfolder")
2. launch: /usr/bin/deja-dup --restore-missing .
3. you will see malformed foldername/filename in the restore window which tells you that these entries are missing (which is wrong)
4. the restore of such a malformed file will also fail

Can anyone confirm this?

Greets

Vej (vej) wrote :

Hello Felix Haller,

did you used trusty-proposed for your test?

I tried your setup as well but was not able to see anything.

See bug #1356548 comment #16 for details.

Best Regards

Vej

Vej (vej) wrote :

Hello everyone!

I had a look at the diff from the fix of bug #1356548 and they only fixed the occurring of the Traceback in case the file was not found.

See this cite from comment #6 of this bug:

> You were experiencing the crash when duplicity was trying to say that the file you asked for didn't exist in the backup.
> When I tried restoring utf8 filenames that did exist in the backup, it worked fine.

So this is definitely not an duplicate of bug #1356548.

Best Regards

Vej

Karl Maier (w-wall2001) wrote :

Is bug #1406505 a duplicate of this bug?

Vej (vej) wrote :

@19 I agree, and marked bug #1406505 as a duplicate.

Thanks for pointing this out. This duplicate has some helpful additional informations.

Vej (vej) wrote :

I did some further tests with actual versions of deja-dup and found out, that the following restores does not work (I'm trying to restore a file named `dortö`, that is located in `Schreibtisch/tröstfolder` (''Schreibtisch'' means ''Desktop'', and ''tröstfolder'' is just some folder with an ö inside) :

Reproduction Steps:

0. Install your Ubuntu in german.
1. Create the folder Schreibtisch/tröstfolder and the file Schreibtisch/tröstfolder/dortö
2. Perform a fresh Backup to an empty folder (use encryption by setting a password).
3. Delete the file.
4. Try out the following commands:

{{{LC_ALL=C deja-dup --restore ~/Schreiptich/tröstfolder/dortö}} and {{{LC_ALL=C deja-dup --restore ~/Schreiptich/tröstfolder/dortö}}

work as expected.

{{{LC_ALL=C deja-dup --restore-missing tröstfolder/}} and
{{{deja-dup --restore-missing tröstfolder/da}}

result in empty lists of files.

I decrypted the backup using `gpg --multifile --decrypt duplicity-full.*.*.difftar.gpg` and unpacked the files manually using `for t in duplicity-full.20110127T131352Z.*.difftar; do tar xf $t; done` (see [[https://wiki.gnome.org/Apps/DejaDup/Help/Restore/WorstCase| "When Everything Goes Wrong" in Gnome Wiki ]]). `dortö` was inside of the folder `snapshot/home/test/Schreibtisch/tröstfolder`.

Versions: deja-dup `30.0-0ubuntu4` and duplicity `0.6.23-1ubuntu4.2`
OS.: Ubuntu Trusty Thar

Please Note: I installed the version `0.6.23-1ubuntu4.2` of duplicity from proposed, to avoid being hit by bug #1356548.

Michael Terry (mterry) wrote :

Vej, thanks for your clear instructions. And thanks to Martin Lex for poking me about this bug. I think I've fixed this in trunk. Hopefully will have a source release and an update to xenial shortly.

affects: duplicity → deja-dup
Changed in deja-dup:
status: New → Fix Committed
Michael Terry (mterry) on 2016-04-07
Changed in deja-dup:
status: Fix Committed → Fix Released
Michael Terry (mterry) wrote :

Once this hits xenial, I'd appreciate reports of whether that fixes it for people and they don't see any additional weirdness. Then I can prepare an SRU for trusty.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package deja-dup - 34.2-0ubuntu1

---------------
deja-dup (34.2-0ubuntu1) xenial; urgency=medium

  * New bug-fix upstream release:
    - Fixes restoring some files with unicode characters (LP: #1377873)
  * debian/patch/series:
    - Disable appstream patch, upstream merged it

 -- Michael Terry <email address hidden> Thu, 07 Apr 2016 12:11:31 -0400

Changed in deja-dup (Ubuntu):
status: Confirmed → Fix Released
Vej (vej) wrote :

Hello Michael,

thanks for your efforts in fixing this bug.

I will try tomorrow, with today's daily build of Xenial and let you know about the results.

Vej (vej) wrote :

Hello everyone,

the version 34.2-0ubuntu1 performs good if you follow my scenario given in comment #21. Thanks to Michael Terry for that.

But if you do the following, you may encounter problems:

0. Install the dailybuild from http://cdimage.ubuntu.com/daily-live/current/ (I choosed the one created at 08-Apr-2016 08:03), choose German ("Deutsch") as your language.
1. Perform a fresh backup to a local folder (set a password for encryption).
2. Create the folder Schreibtisch/tröstfolder and the file Schreibtisch/tröstfolder/dortö.
3. Restart the Backup (perform an incremental Backup)
4. Delete the file.
5. Restart the Backup (perform an incremental Backup)
6. Try to restore the file using LC_ALL=C deja-dup --restore-missing tröstfolder/

Result: The list is empty.

However: If you leave out step five and try to restore the file after it was deleted, it works very well :).

So I'm wondering now if the "restore-missing"-function is expected to look at the last backup only. Can anyone explain this to me?

Thanking you in anticipation

Vej

Michael Terry (mterry) wrote :

Vej, thanks for testing! I will try testing that scenario. I *thought* we looked further back than just the latest backup. I will confirm.

Maybe it make sense to open a new Bug for #26?

Laurent Bigonville (bigon) wrote :

I think this fix introduce a regression:

Running on my (debian) desktop the following command: deja-dup --restore-missing file:///home/bigon

With 34.1 I've more files/directories proposed than when using 34.2.

Most of the missing files are the one with accentuated character.

Locale: fr_BE.utf8

Vej (vej) wrote :

Hello Mr. Bigonville,

thanks for your contribution to this bug.

You wrote: "With 34.1 I've more files/directories proposed than when using 34.2."

This seems legit to me, if the accentuated files where proposed in 34.1 by mistake (because the comparison with the Unicode characters went wrong).

So could you please specify this? I would be interested in the following questions:
 - Are the files, that are not proposed in 34.2 actually missing in the folder/home/bigon ?
 - Were the missing files deleted before or after the last backup?
 - Was the last backup full or incremental?
 - Did you made a fresh clean backup with 34.2 or did you used an old one (initially created with 34.1 or older)?

Laurent Bigonville (bigon) wrote :

I think I spoke too fast. Sorry for the noise :(

Vej (vej) wrote :

Hello Michael Terry,

did you managed to test #26?

Are you able to confirm (as mentioned in #27)?

Best Regards

Vej

This but MUST be marked as critical...

Robert Oswald (robert-oswald) wrote :

The problem still exists on Ubuntu 16.04.

Robert Oswald (robert-oswald) wrote :

The problem stills exists on Ubuntu 16.04 when looking for --file-changed (version of files).

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers