Restore Failed with Unknown Error

[Thomas Hamilton Lipscomb]

The backup seems to work, but the Restore Failed with Unknown Error (see attached image)
$ deja_dup_DEBUG=1 deja-dup --backup
$ deja_dup_DEBUG=1 deja-dup --restore

Also I'm not sure where the debug information is when I do deja_dup_DEBUG=1 it seems to run the same as when I'm not in debug mode. Does debug mode create a log file and if so where is that file so I can upload it here to help you?

1. The distribution of Linux you're using:
Ubuntu 20.04.3 LTS

2. The version of deja-dup and duplicity:
deja-dup/focal-updates,now 40.7-0ubuntu1 amd64
duplicity/focal,now 0.8.11.1612-1 amd64

I deleted both the ~/.cache/deja-dup and the backup on my external SSD and that didn't fix it. Every time I tried one of the below I deleted those and made a fresh one. Attached is the screenshot showing the error.

I used the code below to trim all of my filenames in Desktop, Documents, Downloads, Music, Pictures, and Videos to be no more than 100 characters long, just in case filenames being too long was the issue. That didn't fix it.
export n=100
find . -type f \
     ! -name '.*' \
       -regextype egrep \
     ! -regex '.*\.[^/.]{'"$n"',}' \
       -regex '.*[^/]{'$((n+1))',}' \
       -execdir bash -c '
    for f in "${@#./}"; do
        ext=${f#"${f%.*}"}
        mv -- "$f" "${f:0:n-${#ext}}${ext}"
    done' bash {} +

I manually (not using Deja Dup) copied my files from my ext4 Linux filesystem to an NTFS external backup drive to check to make sure no filename has illegal characters like | because for example if the filename has | in it then a warning message pops up and says you can't copy that file to an NTFS file system do you want to skip it. I removed the illegal characters and now it copies to NTFS without any warnings but the backup still fails so that wasn't it.

I reformatted my Deja Dup backup drive Samsung_T5 from exFAT to ext4 just in case Deja Dup doesn’t like exFAT (cross-platform Solid State Drive filesystem) and likes ext4 (Linux filesystem) better. That didn't fix it.

I uninstalled Deja Dup using "sudo apt-get purge deja-dup" then reinstalled it from the Ubuntu Snap store. I also tried unencrypted backup using Deja Dup instead of encrypted. Doing both of those didn't fix it. Attached is the "Restore Failed: Failed with an unknown error." screenshot of my Ubuntu. I uninstalled and reinstalled Deja Dup because I was worried Deja Dup broke and needed to be reinstalled because my Ubuntu accidentally broke when I uninstalled Python3 not in my sandbox (conda create --name dnv) (conda activate dnv) but in base Ubuntu. I sent my laptop to the computer repair place and they didn't reinstall Ubuntu they rebuilt it reinstalling Python3. That rebuild didn't totally fix the laptop I had to reinstall my eBook reader to get it working again, so maybe Deja Dup broke and was not fixed by the Ubuntu rebuild but reinstalling Deja Dup might fix it but didn't. I checked and uninstalling Deja Dup also uninstalls Duplicity so reinstalling Duplicity isn't the solution.

I cannot find /tmp/deja-dup.gsettings anymore

At this point my computer repair guy recommended trying Bacula (https://ubuntu.com/server/docs/backups-bacula) and if that doesn't work reinstalling my Ubuntu from scratch. I will try Bacula and will post here again about whether that worked. If Bacula also doesn't work then we know something it's not the backup program itself not working it's the operating system or the files. If I then have to reinstall my Ubuntu from scratch then we can see if Deja Dup works on that then if copying my files back in breaks the Deja Dup that had just worked without my files, then we know it's not the operating system it's my files.

[Michael Terry (mterry)]

Hello! Thank you for your report. So this is reproducible and keeps happening? That's hopeful for being able to debug it.

You are setting deja_dup_DEBUG=1, but it's case-sensitive. You need to set DEJA_DUP_DEBUG=1 - try that and there may be more interesting logs.

[Thomas Hamilton Lipscomb (hari-seldon)]

The attached screenshot shows the backup still fails with an unknown error and the "$ tail -n 20 restore.txt" below shows that the last thing that happened was:
DUPLICITY: DEBUG 1
DUPLICITY: . Releasing lockfile b'/home/thomas/.cache/deja-dup/5e077b90f60a70f27231726b7a51958b/lockfile'

I looked at my lockfile after the restore attempt and it's empty.

$ tail -n 30 backup.txt
DUPLICITY: INFO 1
DUPLICITY: . Found 7221 volumes in manifest

DUPLICITY: DEBUG 1
DUPLICITY: . Registering (mktemp) temporary file /tmp/duplicity-ddavjj9y-tempdir/mktemp-x_9i29kl-2

DUPLICITY: INFO 1
DUPLICITY: . Making directory /home/thomas/.cache/deja-dup/metadata

DUPLICITY: INFO 7 'README' reg
DUPLICITY: . Writing README of type reg

DUPLICITY: INFO 1
DUPLICITY: . Deleting /tmp/duplicity-ddavjj9y-tempdir/mktemp-x_9i29kl-2

DUPLICITY: DEBUG 1
DUPLICITY: . Forgetting temporary file /tmp/duplicity-ddavjj9y-tempdir/mktemp-x_9i29kl-2

DUPLICITY: INFO 2 1 7221
DUPLICITY: . Processed volume 1 of 7221

DUPLICITY: DEBUG 1
DUPLICITY: . Releasing lockfile b'/home/thomas/.cache/deja-dup/5e077b90f60a70f27231726b7a51958b/lockfile'

DUPLICITY: DEBUG 1
DUPLICITY: . Removing still remembered temporary file /tmp/duplicity-ddavjj9y-tempdir/mkstemp-y1c91ihv-1

DUPLICITY: DEBUG 1
DUPLICITY: . Releasing lockfile b'/home/thomas/.cache/deja-dup/5e077b90f60a70f27231726b7a51958b/lockfile'

$ tail -n 20 restore.txt
DUPLICITY: . Mon Nov 16 12:00:41 2020 mnt/M2DATA/VirtualBoxVMs/Work/Logs/VBox.log.3

DUPLICITY: INFO 10 20201116T170936Z 'mnt/M2DATA/VirtualBoxVMs/Work/Test.vdi' reg
DUPLICITY: . Mon Nov 16 12:09:36 2020 mnt/M2DATA/VirtualBoxVMs/Work/Test.vdi

DUPLICITY: INFO 10 20201116T170936Z 'mnt/M2DATA/VirtualBoxVMs/Work/Work.vbox' reg
DUPLICITY: . Mon Nov 16 12:09:36 2020 mnt/M2DATA/VirtualBoxVMs/Work/Work.vbox

DUPLICITY: INFO 10 20201116T170936Z 'mnt/M2DATA/VirtualBoxVMs/Work/Work.vbox-prev' reg
DUPLICITY: . Mon Nov 16 12:09:36 2020 mnt/M2DATA/VirtualBoxVMs/Work/Work.vbox-prev

DUPLICITY: DEBUG 1
DUPLICITY: . Releasing lockfile b'/home/thomas/.cache/deja-dup/5e077b90f60a70f27231726b7a51958b/lockfile'

DUPLICITY: INFO 1
DUPLICITY: . Using temporary directory /tmp/duplicity-z78f60ev-tempdir

DUPLICITY: DEBUG 1
DUPLICITY: . Releasing lockfile b'/home/thomas/.cache/deja-dup/5e077b90f60a70f27231726b7a51958b/lockfile'

[Michael Terry (mterry)]

Hello! Thanks for the log fragments. They look... like a working backup and restore (at least what I can see - I'd expect some weird error at the end).

Do you notice the restored files actually being restored? It looks like it's restoring them at least. So this might be an issue of messaging, not of functioning...

The message you see (unknown error) is when we think an error happened, but never got an actual error message. So like, if duplicity returns with an error code, but never told us what was wrong or something like that. Usually this happens if duplicity crashes on us, but that's not what's going on here, based on your log files.

I've got one thought - try a newer Deja Dup & duplicity by trying the snap version? Maybe this is a known and fixed bug that I've already forgotten about? Or a newer duplicity will act better?

You can try it out by running:

snap install deja-dup --classic

And then you might also want to remove the system version to avoid any confusion about which deja-dup is active:

sudo apt remove deja-dup

If you do that, does it work any better?

[Thomas Hamilton Lipscomb (hari-seldon)]

I uninstalled Deja Dup from Ubuntu Software then did "sudo snap install deja-dup --classic" from these instructions:
https://snapcraft.io/install/deja-dup/ubuntu

But now I have new problems:

1. Command line "DEJA_DUP_DEBUG=1 deja-dup --restore > restore.txt" is not working

$ DEJA_DUP_DEBUG=1 deja-dup --backup > backup.txt
$ DEJA_DUP_DEBUG=1 deja-dup --restore > restore.txt
Please list files to restore

2. Deja Dup GUI not command line "Restore files to original locations" for the Home folder doesn't work "Restore" is greyed out because it doesn't have permission to restore some files (see attached image)

3. I am able to restore just Desktop and deleted files are in fact restored. So a workaround to Deja Dup's lack of permission for system files in the Home folder could be don't restore the entire Home folder just Desktop, Documents, Muisic, Pictures, and Videos.

Was it a mistake for me to do "sudo snap install deja-dup --classic" instead of installing it from the Ubuntu Software GUI? Is there a difference between installing using the command line and installing using Ubuntu Software?

[Michael Terry (mterry)]

> 1. Command line "DEJA_DUP_DEBUG=1 deja-dup --restore > restore.txt" is not working

Ah yes, "deja-dup --restore" no longer works, because I changed it to require a list of files to restore.

> Was it a mistake for me to do "sudo snap install deja-dup --classic" instead of installing it from the Ubuntu Software GUI? Is there a difference between installing using the command line and installing using Ubuntu Software?

No that's fine - should be the same as from the GUI.

> 3. I am able to restore just Desktop and deleted files are in fact restored

Yay! OK, good so your data is back. And now I think I understand - I remember now that that old version would show "unknown error" if you tried to restore over files you didn't have permission for.

Nowadays, we show that dialog listing the files that would present a problem before you attempt a backup, as a nicer experience than that awful error.

I'll close this then, since you have your data, and I believe I understand why the error occurred.