Duplicity fails with "Invalid data - SHA1 hash mismatch for file", does not retry download.

[Eric O'Connor]

There appears to have been a corruption issue in my file download, about halfway through an 80GB restore.

Invalid data - SHA1 hash mismatch for file:
 duplicity-full.20140720T050902Z.vol43.difftar.gpg
 Calculated hash: ee843c75152a61fd20c59e8cd3345288c0c52203
 Manifest hash: 5971136422f4646cb06d722cf188bb04b7df810f

When I checked the file hash on Amazon, it was exactly 5971136422f4646cb06d722cf188bb04b7df810f.

I would expect duplicity to retry the download when the hash doesn't match.

Duplicity version is: duplicity 0.6.24
Python: 2.7.8
Debian jessie/sid

[Eric O'Connor (oconnore)]

Today it failed on vol442:

Invalid data - SHA1 hash mismatch for file:
 duplicity-full.20140720T050902Z.vol442.difftar.gpg
 Calculated hash: 9bbdf149dc6307788b3b10a528fa98e2abf15d09
 Manifest hash: 08c31918f9b113329f35e57aa9a9005755d16f39

ec:lair → s3cmd get s3://7e628b/backup/duplicity-full.20140720T050902Z.vol442.difftar.gpg ./
s3://7e628b/backup/duplicity-full.20140720T050902Z.vol442.difftar.gpg -> ./duplicity-full.20140720T050902Z.vol442.difftar.gpg [1 of 1]
    66172 of 67175826 0% in 51s 1290.94 B/s failed
WARNING: Retrying failed request: /backup/duplicity-full.20140720T050902Z.vol442.difftar.gpg (string indices must be integers, not str)
WARNING: Waiting 3 sec...
s3://7e628b/backup/duplicity-full.20140720T050902Z.vol442.difftar.gpg -> ./duplicity-full.20140720T050902Z.vol442.difftar.gpg [1 of 1]
 67175826 of 67175826 100% in 48s 1347.18 kB/s done
ec:lair → sha1sum duplicity-full.20140720T050902Z.vol442.difftar.gpg
08c31918f9b113329f35e57aa9a9005755d16f39 duplicity-full.20140720T050902Z.vol442.difftar.gpg

Is there a better way than deleting the restore folder and restarting?

[John (johnniedoe)]

I've run into kinda similar issue and I needed really really bad to restore those files.
So I downloaded all volumes to local directory and restored it locally. It went fine and without problems. For me it seems that it is connected with S3 download/upload speeds (which for me peaks between 2-2000kB/s).

[Tommy Winther (twinther)]

I experience this issue with amazon cloud drive backend.
Both verify and restore operations fail.

I'm running on debian:
$ apt-cache policy duplicity
duplicity:
  Installed: 0.7.17-1

It appears the file isn't retrieved fully, though that is not indicated by duplicity.
This is the output of three verify attempts. See that the calculated hash for duplicity-full.20180428T053553Z.vol1.difftar.gpg is different in the first and third attempt.
In the second attempt it's vol 5 that failed, so vol 1 must have passed the hash check.

Invalid data - SHA1 hash mismatch for file:
 duplicity-full.20180428T053553Z.vol1.difftar.gpg
 Calculated hash: ca4a4d473c864d59954f4b73dba872c496e1e3d7
 Manifest hash: f565798e09e6ef935ba6561f30858571fff54c9f

Invalid data - SHA1 hash mismatch for file:
 duplicity-full.20180428T053553Z.vol5.difftar.gpg
 Calculated hash: c2d9c32a488aa75520d1e83e94f91c6e41b9be23
 Manifest hash: acdae328f19801dacd52eb3074cd2d0b1ab0d759

Invalid data - SHA1 hash mismatch for file:
 duplicity-full.20180428T053553Z.vol1.difftar.gpg
 Calculated hash: 0c4d8261c7fd8be8f4c0c58bd65d083a2afb96de
 Manifest hash: f565798e09e6ef935ba6561f30858571fff54c9f

This is the output of a restore attempt with -vdebug:

Registering (mktemp) temporary file /tmp/duplicity-kfTP0K-tempdir/mktemp-A3QhA9-2
Invalid data - SHA1 hash mismatch for file:
 duplicity-full.20180428T053553Z.vol1.difftar.gpg
 Calculated hash: 301e42faddce0482202935dc8eb850c7a0213d87
 Manifest hash: f565798e09e6ef935ba6561f30858571fff54c9f

Releasing lockfile /root/.cache/duplicity/625d77069cec6b4d3eda0670aed5ced5/lockfile
Removing still remembered temporary file /tmp/duplicity-kfTP0K-tempdir/mkstemp-sYTPie-1
Removing still remembered temporary file /tmp/duplicity-kfTP0K-tempdir/mktemp-A3QhA9-2
Releasing lockfile /root/.cache/duplicity/625d77069cec6b4d3eda0670aed5ced5/lockfile
Releasing lockfile /root/.cache/duplicity/625d77069cec6b4d3eda0670aed5ced5/lockfile