corrupt manifest disallows file restore
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Duplicity |
Won't Fix
|
Undecided
|
Unassigned |
Bug Description
First, thanks for creating duplicity. I hope I can use it to replace my current commercial backup solution.
After 2 months of backing up, with a number of disconnects/
Manifest file '' is corrupt: File count says 261395, File list contains 547668
Besides this, I note there are several files in the include path that Duplicity thinks are backed up, i.e. that aren't scheduled for upload on the incremental. However, when I restore these files Duplicity says it cannot find them.
Duplicity version: 0.7.14-
Python version: 2.7.12
OS Distro / version: Ubuntu 16.04.4 LTS
Target: B2 backblaze
Log output (deleted some stuff for privacy):
Using archive dir: /tank/dec/
Using backup name: a7f18478fedc07b
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Main action: inc
=======
duplicity 0.7.14 ($reldate)
Args: /usr/bin/duplicity --num-retries 100 --archive-dir /tank/dec/duplicity -v8 --volsize 100 --encrypt-key FF05B6E2 --exclude /tank/dec/duplicity --exclude /root/.cache --exclude /usr/local/
Linux minime 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64
/usr/bin/python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609]
=======
Using temporary directory /tmp/duplicity-
Temp has 5762285568 available, backup will use approx 136314880.
Local and Remote metadata are synchronized, no sync needed.
Processing local manifest /tank/dec/
Found 23 volumes in manifest
Processing local manifest /tank/dec/
Found 1134 volumes in manifest
Processing local manifest /tank/dec/
Found 5 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 342 volumes in manifest
Processing local manifest /tank/dec/
Found 3 volumes in manifest
Processing local manifest /tank/dec/
Manifest file '' is corrupt: File count says 261395, File list contains 547668
Found 3659 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 23 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 10 volumes in manifest
Processing local manifest /tank/dec/
Found 4 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 29 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 9 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 1 volumes in manifest
Processing local manifest /tank/dec/
Found 83 volumes in manifest
Added incremental Backupset (start_time: Sat Nov 11 13:43:44 2017 / end_time: Tue Dec 5 20:24:15 2017)
Added incremental Backupset (start_time: Tue Dec 5 20:24:15 2017 / end_time: Wed Dec 6 07:35:04 2017)
Added incremental Backupset (start_time: Wed Dec 6 07:35:04 2017 / end_time: Thu Dec 7 07:35:04 2017)
Added incremental Backupset (start_time: Thu Dec 7 07:35:04 2017 / end_time: Fri Dec 15 07:35:05 2017)
Added incremental Backupset (start_time: Fri Dec 15 07:35:05 2017 / end_time: Sat Dec 23 09:40:05 2017)
Added incremental Backupset (start_time: Sat Dec 23 09:40:05 2017 / end_time: Sun Feb 25 07:35:02 2018)
Added incremental Backupset (start_time: Sun Feb 25 07:35:02 2018 / end_time: Mon Feb 26 07:35:04 2018)
Added incremental Backupset (start_time: Mon Feb 26 07:35:04 2018 / end_time: Wed Feb 28 07:35:02 2018)
Added incremental Backupset (start_time: Wed Feb 28 07:35:02 2018 / end_time: Thu Mar 1 07:35:02 2018)
Added incremental Backupset (start_time: Thu Mar 1 07:35:02 2018 / end_time: Fri Mar 2 07:35:02 2018)
Added incremental Backupset (start_time: Fri Mar 2 07:35:02 2018 / end_time: Fri Mar 2 23:00:15 2018)
Added incremental Backupset (start_time: Fri Mar 2 23:00:15 2018 / end_time: Sat Mar 3 09:04:48 2018)
Added incremental Backupset (start_time: Sat Mar 3 09:04:48 2018 / end_time: Sat Mar 3 23:00:03 2018)
Added incremental Backupset (start_time: Sat Mar 3 23:00:03 2018 / end_time: Sun Mar 4 23:00:03 2018)
Added incremental Backupset (start_time: Sun Mar 4 23:00:03 2018 / end_time: Mon Mar 5 23:00:20 2018)
Added incremental Backupset (start_time: Mon Mar 5 23:00:20 2018 / end_time: Tue Mar 6 23:00:14 2018)
Added incremental Backupset (start_time: Tue Mar 6 23:00:14 2018 / end_time: Wed Mar 7 23:00:03 2018)
Added incremental Backupset (start_time: Wed Mar 7 23:00:03 2018 / end_time: Thu Mar 8 23:00:03 2018)
Added incremental Backupset (start_time: Thu Mar 8 23:00:03 2018 / end_time: Fri Mar 9 23:00:03 2018)
Added incremental Backupset (start_time: Fri Mar 9 23:00:03 2018 / end_time: Sat Mar 10 09:00:08 2018)
Added incremental Backupset (start_time: Sat Mar 10 09:00:08 2018 / end_time: Sat Mar 10 23:00:02 2018)
Added incremental Backupset (start_time: Sat Mar 10 23:00:02 2018 / end_time: Sun Mar 11 23:00:03 2018)
Added incremental Backupset (start_time: Sun Mar 11 23:00:03 2018 / end_time: Mon Mar 12 23:00:03 2018)
Added incremental Backupset (start_time: Mon Mar 12 23:00:03 2018 / end_time: Tue Mar 13 23:00:14 2018)
Added incremental Backupset (start_time: Tue Mar 13 23:00:14 2018 / end_time: Wed Mar 14 23:00:14 2018)
Added incremental Backupset (start_time: Wed Mar 14 23:00:14 2018 / end_time: Thu Mar 15 23:00:14 2018)
Added incremental Backupset (start_time: Thu Mar 15 23:00:14 2018 / end_time: Fri Mar 16 23:00:14 2018)
Added incremental Backupset (start_time: Fri Mar 16 23:00:14 2018 / end_time: Sun Mar 18 23:00:03 2018)
Added incremental Backupset (start_time: Sun Mar 18 23:00:03 2018 / end_time: Mon Mar 19 23:00:05 2018)
Added incremental Backupset (start_time: Mon Mar 19 23:00:05 2018 / end_time: Tue Mar 20 23:00:03 2018)
Added incremental Backupset (start_time: Tue Mar 20 23:00:03 2018 / end_time: Wed Mar 21 23:00:09 2018)
Added incremental Backupset (start_time: Wed Mar 21 23:00:09 2018 / end_time: Thu Mar 22 23:00:15 2018)
Added incremental Backupset (start_time: Thu Mar 22 23:00:15 2018 / end_time: Fri Mar 23 23:00:15 2018)
Added incremental Backupset (start_time: Fri Mar 23 23:00:15 2018 / end_time: Sat Mar 24 08:55:51 2018)
Added incremental Backupset (start_time: Sat Mar 24 08:55:51 2018 / end_time: Tue Mar 27 23:00:09 2018)
Added incremental Backupset (start_time: Tue Mar 27 23:00:09 2018 / end_time: Sat Apr 21 20:45:30 2018)
Last inc backup left a partial set, restarting.
Last full backup date: Sat Nov 11 13:43:44 2017
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /tank/dec/
Found 0 secondary backup chains.
Found primary backup chain with matching signature chain:
-------
Chain start time: Sat Nov 11 13:43:44 2017
Chain end time: Sat Apr 21 20:45:30 2018
Number of contained backup sets: 37
Total number of contained volumes: 5348
Type of backup set: Time: Num volumes:
-------
No orphaned or incomplete backup sets found.
Processing local manifest /tank/dec/
Found 23 volumes in manifest
RESTART: Volumes 22 to 23 failed to upload before termination.
Restarting backup at volume 22.
Deleting /tmp/duplicity-
<deleted body>
AsyncScheduler: running task synchronously (asynchronicity disabled)
Writing duplicity-
Deleting /tmp/duplicity-
AsyncScheduler: task completed successfully
Processed volume 36
Writing duplicity-
Deleting /tank/dec/
Writing duplicity-
Deleting /tank/dec/
--------------[ Backup Statistics ]--------------
StartTime 1524412692.62 (Sun Apr 22 08:58:12 2018)
EndTime 1524429108.67 (Sun Apr 22 13:31:48 2018)
ElapsedTime 16416.06 (4 hours 33 minutes 36.06 seconds)
SourceFiles 1268016
SourceFileSize 562498815668 (524 GB)
NewFiles 54550
NewFileSize 3728199214 (3.47 GB)
DeletedFiles 44996
ChangedFiles 3662
ChangedFileSize 11806997452 (11.0 GB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 103208
RawDeltaSize 4974760234 (4.63 GB)
TotalDestinatio
Errors 0
-------
Changed in duplicity: | |
milestone: | none → 0.7.18 |
Changed in duplicity: | |
milestone: | 0.7.18 → 0.7.19 |
Changed in duplicity: | |
milestone: | 0.7.19 → 0.7.20 |
Changed in duplicity: | |
milestone: | 0.7.20 → 0.8.12 |
Please upgrade to the current version of duplicity. This will assure that any bugs fixed since your release are available and may fix your issue. Please try this and report back to let us know how it went.
NOTE 1: This applies especially to duplicity versions between 0.7.03 and 0.7.14 inclusive. There was a fix in 0.7.15 that reduced memory usage drastically, and will help with memory errors and inability to start new threads. This will also fix the corrupted manifest where file counts are different due to truncation.
NOTE 2: Uninstall duplicity first if it was installed via the distribution repository. For Ubuntu, that would be "sudo apt-get purge duplicity".
There are three options:
Release tarball Install - https:/ /launchpad. net/duplicity/ +download /launchpad. net/~duplicity- team/+archive/ ubuntu/ daily /launchpad. net/~duplicity- team/+archive/ ubuntu/ ppa
Daily duplicity builds - https:/
Stable duplicity builds - https:/