corrupt manifest disallows file restore

Bug #1770200 reported by jazz on 2018-05-09
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Kenneth Loafman

Bug Description

First, thanks for creating duplicity. I hope I can use it to replace my current commercial backup solution.

After 2 months of backing up, with a number of disconnects/resumes, my backup seems to have succeeded. However, on my incremental backups, I see the following in the log file:

Manifest file '' is corrupt: File count says 261395, File list contains 547668

Besides this, I note there are several files in the include path that Duplicity thinks are backed up, i.e. that aren't scheduled for upload on the incremental. However, when I restore these files Duplicity says it cannot find them.

Duplicity version: 0.7.14-0ubuntu0ppa1316~ubuntu16.04.1
Python version: 2.7.12
OS Distro / version: Ubuntu 16.04.4 LTS
Target: B2 backblaze

Log output (deleted some stuff for privacy):
Using archive dir: /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68
Using backup name: a7f18478fedc07bafbb3e532dd999b68
Import of duplicity.backends.acdclibackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Main action: inc
================================================================================
duplicity 0.7.14 ($reldate)
Args: /usr/bin/duplicity --num-retries 100 --archive-dir /tank/dec/duplicity -v8 --volsize 100 --encrypt-key FF05B6E2 --exclude /tank/dec/duplicity --exclude /root/.cache --exclude /usr/local/crashplan --exclude <deleted> --exclude /var/tmp --exclude /var/cache --exclude /var/lib/apt/lists --include <deleted> --include /tank/dec --include /tanky/dec --include /etc --include /home --include /var --include /usr/src --include /root --exclude ** / b2://<deleted>@duplicity-backup-bucket
Linux minime 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64
/usr/bin/python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609]
================================================================================
Using temporary directory /tmp/duplicity-tne17U-tempdir
Temp has 5762285568 available, backup will use approx 136314880.
Local and Remote metadata are synchronized, no sync needed.
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.part (7982)
Found 23 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-full.20171111T214344Z.manifest (148326939)
Found 1134 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171111T214344Z.to.20171206T042415Z.manifest (3516589)
Found 5 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171206T042415Z.to.20171206T153504Z.manifest (13401)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171206T153504Z.to.20171207T153504Z.manifest (3475485)
Found 342 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171207T153504Z.to.20171215T153505Z.manifest (1819905)
Found 3 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171215T153505Z.to.20171223T174005Z.manifest (66889806)
Manifest file '' is corrupt: File count says 261395, File list contains 547668
Found 3659 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171223T174005Z.to.20180225T153502Z.manifest (11793)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180225T153502Z.to.20180226T153504Z.manifest (11513)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180226T153504Z.to.20180228T153502Z.manifest (1690480)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180228T153502Z.to.20180301T153502Z.manifest (16324)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180301T153502Z.to.20180302T153502Z.manifest (17077)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180302T153502Z.to.20180303T070015Z.manifest (7107885)
Found 23 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180303T070015Z.to.20180303T170448Z.manifest (9726)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180303T170448Z.to.20180304T070003Z.manifest (3882284)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180304T070003Z.to.20180305T070003Z.manifest (120149)
Found 10 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180305T070003Z.to.20180306T070020Z.manifest (4030344)
Found 4 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180306T070020Z.to.20180307T070014Z.manifest (16765)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180307T070014Z.to.20180308T070003Z.manifest (19618)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180308T070003Z.to.20180309T070003Z.manifest (21724)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180309T070003Z.to.20180310T070003Z.manifest (20313)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180310T070003Z.to.20180310T170008Z.manifest (9847)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180310T170008Z.to.20180311T070002Z.manifest (16108)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180311T070002Z.to.20180312T060003Z.manifest (14393)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180312T060003Z.to.20180313T060003Z.manifest (1705638)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180313T060003Z.to.20180314T060014Z.manifest (17341)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180314T060014Z.to.20180315T060014Z.manifest (122553)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180315T060014Z.to.20180316T060014Z.manifest (12706)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180316T060014Z.to.20180317T060014Z.manifest (5322447)
Found 29 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180317T060014Z.to.20180319T060003Z.manifest (20739)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180319T060003Z.to.20180320T060005Z.manifest (2324706)
Found 9 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180320T060005Z.to.20180321T060003Z.manifest (1094227)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180321T060003Z.to.20180322T060009Z.manifest (9491)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180322T060009Z.to.20180323T060015Z.manifest (42734)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180323T060015Z.to.20180324T060015Z.manifest (9675)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180324T060015Z.to.20180324T155551Z.manifest (7294)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180324T155551Z.to.20180328T060009Z.manifest (28486817)
Found 83 volumes in manifest
Added incremental Backupset (start_time: Sat Nov 11 13:43:44 2017 / end_time: Tue Dec 5 20:24:15 2017)
Added incremental Backupset (start_time: Tue Dec 5 20:24:15 2017 / end_time: Wed Dec 6 07:35:04 2017)
Added incremental Backupset (start_time: Wed Dec 6 07:35:04 2017 / end_time: Thu Dec 7 07:35:04 2017)
Added incremental Backupset (start_time: Thu Dec 7 07:35:04 2017 / end_time: Fri Dec 15 07:35:05 2017)
Added incremental Backupset (start_time: Fri Dec 15 07:35:05 2017 / end_time: Sat Dec 23 09:40:05 2017)
Added incremental Backupset (start_time: Sat Dec 23 09:40:05 2017 / end_time: Sun Feb 25 07:35:02 2018)
Added incremental Backupset (start_time: Sun Feb 25 07:35:02 2018 / end_time: Mon Feb 26 07:35:04 2018)
Added incremental Backupset (start_time: Mon Feb 26 07:35:04 2018 / end_time: Wed Feb 28 07:35:02 2018)
Added incremental Backupset (start_time: Wed Feb 28 07:35:02 2018 / end_time: Thu Mar 1 07:35:02 2018)
Added incremental Backupset (start_time: Thu Mar 1 07:35:02 2018 / end_time: Fri Mar 2 07:35:02 2018)
Added incremental Backupset (start_time: Fri Mar 2 07:35:02 2018 / end_time: Fri Mar 2 23:00:15 2018)
Added incremental Backupset (start_time: Fri Mar 2 23:00:15 2018 / end_time: Sat Mar 3 09:04:48 2018)
Added incremental Backupset (start_time: Sat Mar 3 09:04:48 2018 / end_time: Sat Mar 3 23:00:03 2018)
Added incremental Backupset (start_time: Sat Mar 3 23:00:03 2018 / end_time: Sun Mar 4 23:00:03 2018)
Added incremental Backupset (start_time: Sun Mar 4 23:00:03 2018 / end_time: Mon Mar 5 23:00:20 2018)
Added incremental Backupset (start_time: Mon Mar 5 23:00:20 2018 / end_time: Tue Mar 6 23:00:14 2018)
Added incremental Backupset (start_time: Tue Mar 6 23:00:14 2018 / end_time: Wed Mar 7 23:00:03 2018)
Added incremental Backupset (start_time: Wed Mar 7 23:00:03 2018 / end_time: Thu Mar 8 23:00:03 2018)
Added incremental Backupset (start_time: Thu Mar 8 23:00:03 2018 / end_time: Fri Mar 9 23:00:03 2018)
Added incremental Backupset (start_time: Fri Mar 9 23:00:03 2018 / end_time: Sat Mar 10 09:00:08 2018)
Added incremental Backupset (start_time: Sat Mar 10 09:00:08 2018 / end_time: Sat Mar 10 23:00:02 2018)
Added incremental Backupset (start_time: Sat Mar 10 23:00:02 2018 / end_time: Sun Mar 11 23:00:03 2018)
Added incremental Backupset (start_time: Sun Mar 11 23:00:03 2018 / end_time: Mon Mar 12 23:00:03 2018)
Added incremental Backupset (start_time: Mon Mar 12 23:00:03 2018 / end_time: Tue Mar 13 23:00:14 2018)
Added incremental Backupset (start_time: Tue Mar 13 23:00:14 2018 / end_time: Wed Mar 14 23:00:14 2018)
Added incremental Backupset (start_time: Wed Mar 14 23:00:14 2018 / end_time: Thu Mar 15 23:00:14 2018)
Added incremental Backupset (start_time: Thu Mar 15 23:00:14 2018 / end_time: Fri Mar 16 23:00:14 2018)
Added incremental Backupset (start_time: Fri Mar 16 23:00:14 2018 / end_time: Sun Mar 18 23:00:03 2018)
Added incremental Backupset (start_time: Sun Mar 18 23:00:03 2018 / end_time: Mon Mar 19 23:00:05 2018)
Added incremental Backupset (start_time: Mon Mar 19 23:00:05 2018 / end_time: Tue Mar 20 23:00:03 2018)
Added incremental Backupset (start_time: Tue Mar 20 23:00:03 2018 / end_time: Wed Mar 21 23:00:09 2018)
Added incremental Backupset (start_time: Wed Mar 21 23:00:09 2018 / end_time: Thu Mar 22 23:00:15 2018)
Added incremental Backupset (start_time: Thu Mar 22 23:00:15 2018 / end_time: Fri Mar 23 23:00:15 2018)
Added incremental Backupset (start_time: Fri Mar 23 23:00:15 2018 / end_time: Sat Mar 24 08:55:51 2018)
Added incremental Backupset (start_time: Sat Mar 24 08:55:51 2018 / end_time: Tue Mar 27 23:00:09 2018)
Added incremental Backupset (start_time: Tue Mar 27 23:00:09 2018 / end_time: Sat Apr 21 20:45:30 2018)
Last inc backup left a partial set, restarting.
Last full backup date: Sat Nov 11 13:43:44 2017
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Sat Nov 11 13:43:44 2017
Chain end time: Sat Apr 21 20:45:30 2018
Number of contained backup sets: 37
Total number of contained volumes: 5348
 Type of backup set: Time: Num volumes:
                Full Sat Nov 11 13:43:44 2017 1134
         Incremental Tue Dec 5 20:24:15 2017 5
         Incremental Wed Dec 6 07:35:04 2017 1
         Incremental Thu Dec 7 07:35:04 2017 342
         Incremental Fri Dec 15 07:35:05 2017 3
         Incremental Sat Dec 23 09:40:05 2017 3659
         Incremental Sun Feb 25 07:35:02 2018 1
         Incremental Mon Feb 26 07:35:04 2018 1
         Incremental Wed Feb 28 07:35:02 2018 1
         Incremental Thu Mar 1 07:35:02 2018 1
         Incremental Fri Mar 2 07:35:02 2018 1
         Incremental Fri Mar 2 23:00:15 2018 23
         Incremental Sat Mar 3 09:04:48 2018 1
         Incremental Sat Mar 3 23:00:03 2018 1
         Incremental Sun Mar 4 23:00:03 2018 10
         Incremental Mon Mar 5 23:00:20 2018 4
         Incremental Tue Mar 6 23:00:14 2018 1
         Incremental Wed Mar 7 23:00:03 2018 1
         Incremental Thu Mar 8 23:00:03 2018 1
         Incremental Fri Mar 9 23:00:03 2018 1
         Incremental Sat Mar 10 09:00:08 2018 1
         Incremental Sat Mar 10 23:00:02 2018 1
         Incremental Sun Mar 11 23:00:03 2018 1
         Incremental Mon Mar 12 23:00:03 2018 1
         Incremental Tue Mar 13 23:00:14 2018 1
         Incremental Wed Mar 14 23:00:14 2018 1
         Incremental Thu Mar 15 23:00:14 2018 1
         Incremental Fri Mar 16 23:00:14 2018 29
         Incremental Sun Mar 18 23:00:03 2018 1
         Incremental Mon Mar 19 23:00:05 2018 9
         Incremental Tue Mar 20 23:00:03 2018 1
         Incremental Wed Mar 21 23:00:09 2018 1
         Incremental Thu Mar 22 23:00:15 2018 1
         Incremental Fri Mar 23 23:00:15 2018 1
         Incremental Sat Mar 24 08:55:51 2018 1
         Incremental Tue Mar 27 23:00:09 2018 83
         Incremental Sat Apr 21 20:45:30 2018 22
-------------------------
No orphaned or incomplete backup sets found.
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.part (7982)
Found 23 volumes in manifest
RESTART: Volumes 22 to 23 failed to upload before termination.
         Restarting backup at volume 22.
Deleting /tmp/duplicity-tne17U-tempdir/mktemp-gLW__x-2

<deleted body>

AsyncScheduler: running task synchronously (asynchronicity disabled)
Writing duplicity-inc.20180328T060009Z.to.20180422T034530Z.vol36.difftar.gpg
Deleting /tmp/duplicity-tne17U-tempdir/mktemp-PgtBD7-17
AsyncScheduler: task completed successfully
Processed volume 36
Writing duplicity-new-signatures.20180328T060009Z.to.20180422T034530Z.sigtar.gpg
Deleting /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-new-signatures.20180328T060009Z.to.20180422T034530Z.sigtar.gpg
Writing duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.gpg
Deleting /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.gpg
--------------[ Backup Statistics ]--------------
StartTime 1524412692.62 (Sun Apr 22 08:58:12 2018)
EndTime 1524429108.67 (Sun Apr 22 13:31:48 2018)
ElapsedTime 16416.06 (4 hours 33 minutes 36.06 seconds)
SourceFiles 1268016
SourceFileSize 562498815668 (524 GB)
NewFiles 54550
NewFileSize 3728199214 (3.47 GB)
DeletedFiles 44996
ChangedFiles 3662
ChangedFileSize 11806997452 (11.0 GB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 103208
RawDeltaSize 4974760234 (4.63 GB)
TotalDestinationSizeChange 1483159528 (1.38 GB)
Errors 0
-------------------------------------------------

Please upgrade to the current version of duplicity. This will assure that any bugs fixed since your release are available and may fix your issue. Please try this and report back to let us know how it went.

NOTE 1: This applies especially to duplicity versions between 0.7.03 and 0.7.14 inclusive. There was a fix in 0.7.15 that reduced memory usage drastically, and will help with memory errors and inability to start new threads. This will also fix the corrupted manifest where file counts are different due to truncation.

NOTE 2: Uninstall duplicity first if it was installed via the distribution repository. For Ubuntu, that would be "sudo apt-get purge duplicity".

There are three options:

Release tarball Install - https://launchpad.net/duplicity/+download
Daily duplicity builds - https://launchpad.net/~duplicity-team/+archive/ubuntu/daily
Stable duplicity builds - https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa

Changed in duplicity:
status: New → In Progress
assignee: nobody → Kenneth Loafman (kenneth-loafman)
importance: Undecided → Medium
  • duplicity Edit (1.0 MiB, text/plain; charset=UTF-8; name="duplicity")
  • duplicity.2 Edit (1.0 MiB, text/plain; charset=UTF-8; name="duplicity.2")

Thanks, Kenneth. I've done the upgrade to 0.7.17-0ubuntu0pp. The problem
persists:

=== experiment 1, restore log : duplicity ===

- I have a certain pdf file that is in the include path. On an
incremental, duplicity thinks it is not modified relative to the full
backup and therefore does not upload it.

- On a restore, duplicity claims the file is not known, with an error
(see below)

I've attached my log file. I note some stuff that is suspicious to my
layman eyes:
- Many backend files are 'not part of a known set'
- Some files in the backend are ignored
- This error message: "Error 'First patch in sequence
[<duplicity.path.ROPath instance at 0x7f9d4d899710>] was a diff'
patching . "

=== experiment 2, restore log: duplicity.2 ===
- I restore the entire directory of the pdf file mentioned in experiment 1.
- I get a similar error message on the specific pdf file:
Error 'First patch in sequence [<duplicity.path.ROPath instance at
0x7f1b1a4927a0>] was a diff' patching <removed>.pdf

- However, I also note that the restore of the entire folder is missing
a lot of files that are also not picked up in an incremental backup

=== conclusion ===
I'm in some error state where changed files are not backed up on an
incremental, but the originals (if any) are also not restoreable.

Jasper

Changed in duplicity:
milestone: none → 0.7.18

Error 'First patch in sequence [<duplicity.path.ROPath instance at
0x7f1b1a4927a0>] was a diff' patching <removed>.pdf

This is telling me that one of the difftar files is corrupted. A lot of disconnects and reconnects can indeed cause this problem.

Please do a full backup and verify. This should fix the problem. Also please note that a long string of incrementals is risky due to possible corruption in one item in the string.

jazz (jazz-launchpad) wrote :

Thanks for the explanation.

Is there some way to repair this backup? It's the result of several
months of uploading (which interferes with my home connection), so I'm
trying to avoid too much data transfer. One thing I can think of is
strip off incrementals from newest to oldest until the restore succeeds,
and then continue with incremental. Perhaps you have a more informed
solution.

Without being constrained by knowledge of the inner workings of
duplicity, I can imagine it would be nice to have some sort of
self-repair, where each incremental strengthens the backup rather than
potentially weakens it; i.e. where corrupted backup structures are fixed
by the next incremental. A weaker form would be to fix the latest
incremental when a disconnect has been detected. Maybe this helps.

Thanks for the help so far.

Jasper.

On 05/28/2018 08:01 AM, Kenneth Loafman wrote:
> Error 'First patch in sequence [<duplicity.path.ROPath instance at
> 0x7f1b1a4927a0>] was a diff' patching <removed>.pdf
>
> This is telling me that one of the difftar files is corrupted. A lot of
> disconnects and reconnects can indeed cause this problem.
>
> Please do a full backup and verify. This should fix the problem. Also
> please note that a long string of incrementals is risky due to possible
> corruption in one item in the string.
>

Good idea about stripping out incremental sets until you find the problem. That may be the best way to proceed. I would move them to another directory instead of deleting just to be safe. Let me know how it goes.

jazz (jazz-launchpad) wrote :

I've stripped out all incrementals until it didn't make sense to strip
out more (since I'd be redoing >90% of the backup), did an incremental
backup (no errors), and ran a verify operation. I'm still getting
several dozens of errors, which I've uniq'd after anonimizing the filenames:

Error 'First patch in sequence [<duplicity.path.ROPath instance at
0x7ff2843fde60>] was a diff' patching <anonimized>
Error 'librsync error 103 while in patch cycle' patching <anonimized>
Error 'Patch sequence isn't regular, but has 2 entries' patching
<anonimized>
python: ERROR: (rs_file_copy_cb) unexpected eof on fd12
python: ERROR: (rs_job_complete) patch job failed: unexpected end of input

My conclusion is I'll have to re-do the backup. Disconnects are a fact
of life on my connection, and since a full backup is >3 months upload
time, my concern is I will have another corrupted backup at the end. Any
suggestions are welcome; thanks for your help so far.

On 06/02/2018 09:30 AM, Kenneth Loafman wrote:
> Good idea about stripping out incremental sets until you find the
> problem. That may be the best way to proceed. I would move them to
> another directory instead of deleting just to be safe. Let me know how
> it goes.
>

I'm not sure how big your backup is, but my most frequent suggestion is to get a USB hard drive and backup to that first using the file:// schema. Then, once you have verified that, you can rsync to whatever remote storage you need. This will be much much faster.

jazz (jazz-launchpad) wrote :
Download full text (20.2 KiB)

Ok, thanks, I can use that as a workaround.

On Fri, Jun 22, 2018, 7:30 AM Kenneth Loafman <email address hidden> wrote:

> I'm not sure how big your backup is, but my most frequent suggestion is
> to get a USB hard drive and backup to that first using the file://
> schema. Then, once you have verified that, you can rsync to whatever
> remote storage you need. This will be much much faster.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1770200
>
> Title:
> corrupt manifest disallows file restore
>
> Status in Duplicity:
> In Progress
>
> Bug description:
> First, thanks for creating duplicity. I hope I can use it to replace
> my current commercial backup solution.
>
> After 2 months of backing up, with a number of disconnects/resumes, my
> backup seems to have succeeded. However, on my incremental backups, I
> see the following in the log file:
>
> Manifest file '' is corrupt: File count says 261395, File list
> contains 547668
>
> Besides this, I note there are several files in the include path that
> Duplicity thinks are backed up, i.e. that aren't scheduled for upload
> on the incremental. However, when I restore these files Duplicity says
> it cannot find them.
>
> Duplicity version: 0.7.14-0ubuntu0ppa1316~ubuntu16.04.1
> Python version: 2.7.12
> OS Distro / version: Ubuntu 16.04.4 LTS
> Target: B2 backblaze
>
> Log output (deleted some stuff for privacy):
> Using archive dir: /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68
> Using backup name: a7f18478fedc07bafbb3e532dd999b68
> Import of duplicity.backends.acdclibackend Succeeded
> Import of duplicity.backends.azurebackend Succeeded
> Import of duplicity.backends.b2backend Succeeded
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.giobackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.hubicbackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.lftpbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.mediafirebackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.multibackend Succeeded
> Import of duplicity.backends.ncftpbackend Succeeded
> Import of duplicity.backends.onedrivebackend Succeeded
> Import of duplicity.backends.par2backend Succeeded
> Import of duplicity.backends.pydrivebackend Succeeded
> Import of duplicity.backends.rsyncbackend Succeeded
> Import of duplicity.backends.ssh_paramiko_backend Succeeded
> Import of duplicity.backends.ssh_pexpect_backend Succeeded
> Import of duplicity.backends.swiftbackend Succeeded
> Import of duplicity.backends.sxbackend Succeeded
> Import of duplicity.backends.tahoebackend Succeeded
> Import of duplicity.backends.webdavbackend Succeeded
> Main action: inc
>
...

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers