Allow people to exclude corrupted incremental backup from chain

[Eero]

The backup location is on a NAS mounted with NFS. I get this message every time I try to make a new backup:

Backup Failed

Invalid data - SHA1 hash mismatch for file:
 duplicity-inc.20161219T001458Z.to.20161220T001456Z.vol1.difftar.gpg
 Calculated hash: 3dde403f81e36268ce9b7f53478d9dd8301a439c
 Manifest hash: b46782fb799c7c5a543ac88d9bfb7cc1df25d38c

People have to use "a workaround" which isn't explained anywhere in the documentation if a backup gets corrupted. It isn't even obvious from the error message that an earlier backup got corrupted.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: deja-dup 34.2-0ubuntu1.1
ProcVersionSignature: Ubuntu 4.4.0-53.74-generic 4.4.30
Uname: Linux 4.4.0-53-generic x86_64
NonfreeKernelModules: nvidia_uvm nvidia_drm nvidia_modeset nvidia
ApportVersion: 2.20.1-0ubuntu2.4
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Dec 20 12:36:15 2016
InstallationDate: Installed on 2014-05-16 (949 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: deja-dup
UpgradeStatus: Upgraded to xenial on 2016-08-02 (139 days ago)

[Aaron Whitehouse (aaron-whitehouse)]

Have you tried to verify your backup or do a new full backup?

> Can someone recommend a better backup software for personal use?
http://lmgtfy.com/?q=personal+backup+software+linux

There is a reason, though, that Deja Dup (which you are already using) is the default in Ubuntu -- it is a pretty good balance of ease of use and reliability.

Kind regards,

Aaron

[Eero (eero+launchpad)]

> Have you tried to verify your backup or do a new full backup?

http://lmgtfy.com/?q=deja+dup+new+full+backup

Shows basically nothing... Where's the option for that? Maybe you should tell me something concrete instead of pasting smart ass links?

[Naël (nathanael-naeri)]

To create a new full backup, simply backup to a new location, or move the files in the current location to some other place. Déjà Dup will detect the absence of previous backups and start a new full backup.

The backup location is the directory that stores the duplicity files that Déjà Dup produces (compressed and possibly encrypted duplicity-<full|inc>.<timestamp>.<volumenumber>.<extensions> files). For instance ftp://user@server:port/path/to/backup/location.

You may also want to run the incremental backup that fails, from the command line, with verbose output, to try and understand what leads to that failure, and help triage this bug, because right now we unfortunately don't know enough to reproduce or fix it:

  $ DEJA_DUP_DEBUG=1 deja-dup --backup

As Aaron said Déjà Dup is a good balance between ease of use and reliability. Since it is a front-end to duplicity, I would recommend using duplicity directly for more reliability (see manpage for usage). You would then avoid all the potential front-end bugs.

[Eero (eero+launchpad)]

The only error message that shows up is the one I already posted here.

DUPLICITY: ERROR 21
DUPLICITY: . Invalid data - SHA1 hash mismatch for file:
DUPLICITY: . duplicity-inc.20161219T001458Z.to.20161220T001456Z.vol1.difftar.gpg
DUPLICITY: . Calculated hash: 3dde403f81e36268ce9b7f53478d9dd8301a439c
DUPLICITY: . Manifest hash: b46782fb799c7c5a543ac88d9bfb7cc1df25d38c
DUPLICITY: .

[Naël (nathanael-naeri)]

Thanks for this log. It shows that your current backup chain started on Oct 10, contains the offending incremental backup (Dec 19 to Dec 20), and has later incremental backups (three more on Dec 20, one on Dec 21, one on Dec 22, one on Dec 30). You can see for yourself under "primary backup chain" (you also have 9 older backup chains).

So your backups are apparently still performed, despite the error message (!?!).

As far as I understand, the SHA1 hash mismatch means that the indicated file (first volume of Dec 19 to Dec 20 incremental backup) is different from when it was created and its SHA1 hash was written in the accompanying .manifest file (a text file that stores the hashes of each volume at creation time). Is it conceivable that the transfer of this file to your backup location failed somehow? Or that it was modified afterwards?

I think the easiest way for you to get rid of this issue is to delete (or rather, move away) the tail of your current backup chain, starting at the offending file, and launch an incremental backup to cover the Dec 19 to today period.

That won't tell us how this file got corrupted however, but I for one am not able to help further.

You may also want to delete (or rather move away) Déjà-Dup's cache directory (~/.cache/deja-dup) in case something weird creeped into it and is causing a problem. It will be recreated with fresh information downloaded from the backup location (manifest and signature files).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alessander Botti Benevides (alessanderbotti)]

Thanks Naël, your suggestion on Comment #6 worked for me on Ubuntu 17.10.

[Vej (vej)]

Hello!

There is obviously nothing we can do about packages which got corrupted on the backup media or during transfers.

So what exactly is this bugreport asking for?

Best Regards

Vej

[Eero (eero+launchpad)]

Can't you read Vej?

People have to use "a workaround" which isn't explained anywhere in the documentation if a backup gets corrupted. It isn't even obvious from the error message that an earlier backup got corrupted.

[Vej (vej)]

Hello.

I included this in the bugs description to make it something specific to work on.

Please note that this is somehow similar to bug #826712, which asks for an automatic restore option for corrupted manifests.

Best Regards

Vej