Duplicity

corrupt manifest disallows file restore

Bug #1770200 reported by jazz on 2018-05-09

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Duplicity	Won't Fix	Undecided	Unassigned

Bug Description

First, thanks for creating duplicity. I hope I can use it to replace my current commercial backup solution.

After 2 months of backing up, with a number of disconnects/resumes, my backup seems to have succeeded. However, on my incremental backups, I see the following in the log file:

Manifest file '' is corrupt: File count says 261395, File list contains 547668

Besides this, I note there are several files in the include path that Duplicity thinks are backed up, i.e. that aren't scheduled for upload on the incremental. However, when I restore these files Duplicity says it cannot find them.

Duplicity version: 0.7.14-0ubuntu0ppa1316~ubuntu16.04.1
Python version: 2.7.12
OS Distro / version: Ubuntu 16.04.4 LTS
Target: B2 backblaze

Log output (deleted some stuff for privacy):
Using archive dir: /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68
Using backup name: a7f18478fedc07bafbb3e532dd999b68
Import of duplicity.backends.acdclibackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Main action: inc
================================================================================
duplicity 0.7.14 ($reldate)
Args: /usr/bin/duplicity --num-retries 100 --archive-dir /tank/dec/duplicity -v8 --volsize 100 --encrypt-key FF05B6E2 --exclude /tank/dec/duplicity --exclude /root/.cache --exclude /usr/local/crashplan --exclude <deleted> --exclude /var/tmp --exclude /var/cache --exclude /var/lib/apt/lists --include <deleted> --include /tank/dec --include /tanky/dec --include /etc --include /home --include /var --include /usr/src --include /root --exclude ** / b2://<deleted>@duplicity-backup-bucket
Linux minime 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64
/usr/bin/python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609]
================================================================================
Using temporary directory /tmp/duplicity-tne17U-tempdir
Temp has 5762285568 available, backup will use approx 136314880.
Local and Remote metadata are synchronized, no sync needed.
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.part (7982)
Found 23 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-full.20171111T214344Z.manifest (148326939)
Found 1134 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171111T214344Z.to.20171206T042415Z.manifest (3516589)
Found 5 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171206T042415Z.to.20171206T153504Z.manifest (13401)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171206T153504Z.to.20171207T153504Z.manifest (3475485)
Found 342 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171207T153504Z.to.20171215T153505Z.manifest (1819905)
Found 3 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171215T153505Z.to.20171223T174005Z.manifest (66889806)
Manifest file '' is corrupt: File count says 261395, File list contains 547668
Found 3659 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171223T174005Z.to.20180225T153502Z.manifest (11793)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180225T153502Z.to.20180226T153504Z.manifest (11513)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180226T153504Z.to.20180228T153502Z.manifest (1690480)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180228T153502Z.to.20180301T153502Z.manifest (16324)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180301T153502Z.to.20180302T153502Z.manifest (17077)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180302T153502Z.to.20180303T070015Z.manifest (7107885)
Found 23 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180303T070015Z.to.20180303T170448Z.manifest (9726)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180303T170448Z.to.20180304T070003Z.manifest (3882284)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180304T070003Z.to.20180305T070003Z.manifest (120149)
Found 10 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180305T070003Z.to.20180306T070020Z.manifest (4030344)
Found 4 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180306T070020Z.to.20180307T070014Z.manifest (16765)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180307T070014Z.to.20180308T070003Z.manifest (19618)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180308T070003Z.to.20180309T070003Z.manifest (21724)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180309T070003Z.to.20180310T070003Z.manifest (20313)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180310T070003Z.to.20180310T170008Z.manifest (9847)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180310T170008Z.to.20180311T070002Z.manifest (16108)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180311T070002Z.to.20180312T060003Z.manifest (14393)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180312T060003Z.to.20180313T060003Z.manifest (1705638)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180313T060003Z.to.20180314T060014Z.manifest (17341)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180314T060014Z.to.20180315T060014Z.manifest (122553)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180315T060014Z.to.20180316T060014Z.manifest (12706)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180316T060014Z.to.20180317T060014Z.manifest (5322447)
Found 29 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180317T060014Z.to.20180319T060003Z.manifest (20739)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180319T060003Z.to.20180320T060005Z.manifest (2324706)
Found 9 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180320T060005Z.to.20180321T060003Z.manifest (1094227)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180321T060003Z.to.20180322T060009Z.manifest (9491)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180322T060009Z.to.20180323T060015Z.manifest (42734)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180323T060015Z.to.20180324T060015Z.manifest (9675)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180324T060015Z.to.20180324T155551Z.manifest (7294)
Found 1 volumes in manifest
Processing local manifest /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180324T155551Z.to.20180328T060009Z.manifest (28486817)
Found 83 volumes in manifest
Added incremental Backupset (start_time: Sat Nov 11 13:43:44 2017 / end_time: Tue Dec 5 20:24:15 2017)
Added incremental Backupset (start_time: Tue Dec 5 20:24:15 2017 / end_time: Wed Dec 6 07:35:04 2017)
Added incremental Backupset (start_time: Wed Dec 6 07:35:04 2017 / end_time: Thu Dec 7 07:35:04 2017)
Added incremental Backupset (start_time: Thu Dec 7 07:35:04 2017 / end_time: Fri Dec 15 07:35:05 2017)
Added incremental Backupset (start_time: Fri Dec 15 07:35:05 2017 / end_time: Sat Dec 23 09:40:05 2017)
Added incremental Backupset (start_time: Sat Dec 23 09:40:05 2017 / end_time: Sun Feb 25 07:35:02 2018)
Added incremental Backupset (start_time: Sun Feb 25 07:35:02 2018 / end_time: Mon Feb 26 07:35:04 2018)
Added incremental Backupset (start_time: Mon Feb 26 07:35:04 2018 / end_time: Wed Feb 28 07:35:02 2018)
Added incremental Backupset (start_time: Wed Feb 28 07:35:02 2018 / end_time: Thu Mar 1 07:35:02 2018)
Added incremental Backupset (start_time: Thu Mar 1 07:35:02 2018 / end_time: Fri Mar 2 07:35:02 2018)
Added incremental Backupset (start_time: Fri Mar 2 07:35:02 2018 / end_time: Fri Mar 2 23:00:15 2018)
Added incremental Backupset (start_time: Fri Mar 2 23:00:15 2018 / end_time: Sat Mar 3 09:04:48 2018)
Added incremental Backupset (start_time: Sat Mar 3 09:04:48 2018 / end_time: Sat Mar 3 23:00:03 2018)
Added incremental Backupset (start_time: Sat Mar 3 23:00:03 2018 / end_time: Sun Mar 4 23:00:03 2018)
Added incremental Backupset (start_time: Sun Mar 4 23:00:03 2018 / end_time: Mon Mar 5 23:00:20 2018)
Added incremental Backupset (start_time: Mon Mar 5 23:00:20 2018 / end_time: Tue Mar 6 23:00:14 2018)
Added incremental Backupset (start_time: Tue Mar 6 23:00:14 2018 / end_time: Wed Mar 7 23:00:03 2018)
Added incremental Backupset (start_time: Wed Mar 7 23:00:03 2018 / end_time: Thu Mar 8 23:00:03 2018)
Added incremental Backupset (start_time: Thu Mar 8 23:00:03 2018 / end_time: Fri Mar 9 23:00:03 2018)
Added incremental Backupset (start_time: Fri Mar 9 23:00:03 2018 / end_time: Sat Mar 10 09:00:08 2018)
Added incremental Backupset (start_time: Sat Mar 10 09:00:08 2018 / end_time: Sat Mar 10 23:00:02 2018)
Added incremental Backupset (start_time: Sat Mar 10 23:00:02 2018 / end_time: Sun Mar 11 23:00:03 2018)
Added incremental Backupset (start_time: Sun Mar 11 23:00:03 2018 / end_time: Mon Mar 12 23:00:03 2018)
Added incremental Backupset (start_time: Mon Mar 12 23:00:03 2018 / end_time: Tue Mar 13 23:00:14 2018)
Added incremental Backupset (start_time: Tue Mar 13 23:00:14 2018 / end_time: Wed Mar 14 23:00:14 2018)
Added incremental Backupset (start_time: Wed Mar 14 23:00:14 2018 / end_time: Thu Mar 15 23:00:14 2018)
Added incremental Backupset (start_time: Thu Mar 15 23:00:14 2018 / end_time: Fri Mar 16 23:00:14 2018)
Added incremental Backupset (start_time: Fri Mar 16 23:00:14 2018 / end_time: Sun Mar 18 23:00:03 2018)
Added incremental Backupset (start_time: Sun Mar 18 23:00:03 2018 / end_time: Mon Mar 19 23:00:05 2018)
Added incremental Backupset (start_time: Mon Mar 19 23:00:05 2018 / end_time: Tue Mar 20 23:00:03 2018)
Added incremental Backupset (start_time: Tue Mar 20 23:00:03 2018 / end_time: Wed Mar 21 23:00:09 2018)
Added incremental Backupset (start_time: Wed Mar 21 23:00:09 2018 / end_time: Thu Mar 22 23:00:15 2018)
Added incremental Backupset (start_time: Thu Mar 22 23:00:15 2018 / end_time: Fri Mar 23 23:00:15 2018)
Added incremental Backupset (start_time: Fri Mar 23 23:00:15 2018 / end_time: Sat Mar 24 08:55:51 2018)
Added incremental Backupset (start_time: Sat Mar 24 08:55:51 2018 / end_time: Tue Mar 27 23:00:09 2018)
Added incremental Backupset (start_time: Tue Mar 27 23:00:09 2018 / end_time: Sat Apr 21 20:45:30 2018)
Last inc backup left a partial set, restarting.
Last full backup date: Sat Nov 11 13:43:44 2017
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Sat Nov 11 13:43:44 2017
Chain end time: Sat Apr 21 20:45:30 2018
Number of contained backup sets: 37
Total number of contained volumes: 5348
Type of backup set: Time: Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental Incremental -------------------------
No orphaned or incomplete backup sets found.
Processing local manifest /tank/dec/duplicity/a7f18478fedc07b RESTART: Volumes 22 to 23 failed to upload before termination.
Restarting Deleting /tmp/duplicity-tne17U-tempdir/mktemp- ; Full Sat Nov 11 13:43:44 2017 1134
Tue Dec 5 20:24:15 2017 5
Wed Dec 6 07:35:04 2017 1
Thu Dec 7 07:35:04 2017 342
Fri Dec 15 07:35:05 2017 3
Sat Dec 23 09:40:05 2017 3659
Sun Feb 25 07:35:02 2018 1
Mon Feb 26 07:35:04 2018 1
Wed Feb 28 07:35:02 2018 1
Thu Mar 1 07:35:02 2018 1
Fri Mar 2 07:35:02 2018 1
Fri Mar 2 23:00:15 2018 23
Sat Mar 3 09:04:48 2018 1
Sat Mar 3 23:00:03 2018 1
Sun Mar 4 23:00:03 2018 10
Mon Mar 5 23:00:20 2018 4
Tue Mar 6 23:00:14 2018 1
Wed Mar 7 23:00:03 2018 1
Thu Mar 8 23:00:03 2018 1
Fri Mar 9 23:00:03 2018 1
Sat Mar 10 09:00:08 2018 1
Sat Mar 10 23:00:02 2018 1
Sun Mar 11 23:00:03 2018 1
Mon Mar 12 23:00:03 2018 1
Tue Mar 13 23:00:14 2018 1
Wed Mar 14 23:00:14 2018 1
Thu Mar 15 23:00:14 2018 1
Fri Mar 16 23:00:14 2018 29
Sun Mar 18 23:00:03 2018 1
Mon Mar 19 23:00:05 2018 9
Tue Mar 20 23:00:03 2018 1
Wed Mar 21 23:00:09 2018 1
Thu Mar 22 23:00:15 2018 1
Fri Mar 23 23:00:15 2018 1
Sat Mar 24 08:55:51 2018 1
Tue Mar 27 23:00:09 2018 83
Sat Apr 21 20:45:30 2018 22
/>afbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.part (7982)
backup at volume 22.
/>gLW__x-2

AsyncScheduler: running task synchronously (asynchronicity disabled)
Writing duplicity-inc.20180328T060009Z.to.20180422T034530Z.vol36.difftar.gpg
Deleting /tmp/duplicity-tne17U-tempdir/mktemp-PgtBD7-17
AsyncScheduler: task completed successfully
Processed volume 36
Writing duplicity-new-signatures.20180328T060009Z.to.20180422T034530Z.sigtar.gpg
Deleting /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-new-signatures.20180328T060009Z.to.20180422T034530Z.sigtar.gpg
Writing duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.gpg
Deleting /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.gpg
--------------[ Backup Statistics ]--------------
StartTime 1524412692.62 (Sun Apr 22 08:58:12 2018)
EndTime 1524429108.67 (Sun Apr 22 13:31:48 2018)
ElapsedTime 16416.06 (4 hours 33 minutes 36.06 seconds)
SourceFiles 1268016
SourceFileSize 562498815668 (524 GB)
NewFiles 54550
NewFileSize 3728199214 (3.47 GB)
DeletedFiles 44996
ChangedFiles 3662
ChangedFileSize 11806997452 (11.0 GB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 103208
RawDeltaSize 4974760234 (4.63 GB)
TotalDestinationSizeChange 1483159528 (1.38 GB)
Errors 0
-------------------------------------------------

Revision history for this message

Kenneth Loafman (kenneth-loafman) wrote on 2018-05-10:

Please upgrade to the current version of duplicity. This will assure that any bugs fixed since your release are available and may fix your issue. Please try this and report back to let us know how it went.

NOTE 1: This applies especially to duplicity versions between 0.7.03 and 0.7.14 inclusive. There was a fix in 0.7.15 that reduced memory usage drastically, and will help with memory errors and inability to start new threads. This will also fix the corrupted manifest where file counts are different due to truncation.

NOTE 2: Uninstall duplicity first if it was installed via the distribution repository. For Ubuntu, that would be "sudo apt-get purge duplicity".

There are three options:

Release tarball Install - https://launchpad.net/duplicity/+download
Daily duplicity builds - https://launchpad.net/~duplicity-team/+archive/ubuntu/daily
Stable duplicity builds - https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa

Changed in duplicity:
status:	New → In Progress
assignee:	nobody → Kenneth Loafman (kenneth-loafman)
importance:	Undecided → Medium

Revision history for this message

jazz (jazz-launchpad) wrote on 2018-05-20: Re: [Bug 1770200] Re: corrupt manifest disallows file restore

duplicity Edit (1.0 MiB, text/plain; charset=UTF-8; name="duplicity")
duplicity.2 Edit (1.0 MiB, text/plain; charset=UTF-8; name="duplicity.2")

Thanks, Kenneth. I've done the upgrade to 0.7.17-0ubuntu0pp. The problem
persists:

=== experiment 1, restore log : duplicity ===

- I have a certain pdf file that is in the include path. On an
incremental, duplicity thinks it is not modified relative to the full
backup and therefore does not upload it.

- On a restore, duplicity claims the file is not known, with an error
(see below)

I've attached my log file. I note some stuff that is suspicious to my
layman eyes:
- Many backend files are 'not part of a known set'
- Some files in the backend are ignored
- This error message: "Error 'First patch in sequence
[<duplicity.path.ROPath instance at 0x7f9d4d899710>] was a diff'
patching . "

=== experiment 2, restore log: duplicity.2 ===
- I restore the entire directory of the pdf file mentioned in experiment 1.
- I get a similar error message on the specific pdf file:
Error 'First patch in sequence [<duplicity.path.ROPath instance at
0x7f1b1a4927a0>] was a diff' patching <removed>.pdf

- However, I also note that the restore of the entire folder is missing
a lot of files that are also not picked up in an incremental backup

=== conclusion ===
I'm in some error state where changed files are not backed up on an
incremental, but the originals (if any) are also not restoreable.

Jasper

Kenneth Loafman (kenneth-loafman) on 2018-05-24

Changed in duplicity:
milestone:	none → 0.7.18

Revision history for this message

Kenneth Loafman (kenneth-loafman) wrote on 2018-05-28:

Error 'First patch in sequence [<duplicity.path.ROPath instance at
0x7f1b1a4927a0>] was a diff' patching <removed>.pdf

This is telling me that one of the difftar files is corrupted. A lot of disconnects and reconnects can indeed cause this problem.

Please do a full backup and verify. This should fix the problem. Also please note that a long string of incrementals is risky due to possible corruption in one item in the string.

Revision history for this message

jazz (jazz-launchpad) wrote on 2018-05-29:

Thanks for the explanation.

Is there some way to repair this backup? It's the result of several
months of uploading (which interferes with my home connection), so I'm
trying to avoid too much data transfer. One thing I can think of is
strip off incrementals from newest to oldest until the restore succeeds,
and then continue with incremental. Perhaps you have a more informed
solution.

Without being constrained by knowledge of the inner workings of
duplicity, I can imagine it would be nice to have some sort of
self-repair, where each incremental strengthens the backup rather than
potentially weakens it; i.e. where corrupted backup structures are fixed
by the next incremental. A weaker form would be to fix the latest
incremental when a disconnect has been detected. Maybe this helps.

Thanks for the help so far.

Jasper.

On 05/28/2018 08:01 AM, Kenneth Loafman wrote:
> Error 'First patch in sequence [<duplicity.path.ROPath instance at
> 0x7f1b1a4927a0>] was a diff' patching <removed>.pdf
>
> This is telling me that one of the difftar files is corrupted. A lot of
> disconnects and reconnects can indeed cause this problem.
>
> Please do a full backup and verify. This should fix the problem. Also
> please note that a long string of incrementals is risky due to possible
> corruption in one item in the string.
>

Revision history for this message

Kenneth Loafman (kenneth-loafman) wrote on 2018-06-02:

Good idea about stripping out incremental sets until you find the problem. That may be the best way to proceed. I would move them to another directory instead of deleting just to be safe. Let me know how it goes.

Revision history for this message

jazz (jazz-launchpad) wrote on 2018-06-21:

I've stripped out all incrementals until it didn't make sense to strip
out more (since I'd be redoing >90% of the backup), did an incremental
backup (no errors), and ran a verify operation. I'm still getting
several dozens of errors, which I've uniq'd after anonimizing the filenames:

Error 'First patch in sequence [<duplicity.path.ROPath instance at
0x7ff2843fde60>] was a diff' patching <anonimized>
Error 'librsync error 103 while in patch cycle' patching <anonimized>
Error 'Patch sequence isn't regular, but has 2 entries' patching
<anonimized>
python: ERROR: (rs_file_copy_cb) unexpected eof on fd12
python: ERROR: (rs_job_complete) patch job failed: unexpected end of input

My conclusion is I'll have to re-do the backup. Disconnects are a fact
of life on my connection, and since a full backup is >3 months upload
time, my concern is I will have another corrupted backup at the end. Any
suggestions are welcome; thanks for your help so far.

On 06/02/2018 09:30 AM, Kenneth Loafman wrote:
> Good idea about stripping out incremental sets until you find the
> problem. That may be the best way to proceed. I would move them to
> another directory instead of deleting just to be safe. Let me know how
> it goes.
>

Revision history for this message

Kenneth Loafman (kenneth-loafman) wrote on 2018-06-22:

I'm not sure how big your backup is, but my most frequent suggestion is to get a USB hard drive and backup to that first using the file:// schema. Then, once you have verified that, you can rsync to whatever remote storage you need. This will be much much faster.

Revision history for this message

jazz (jazz-launchpad) wrote on 2018-06-22:

Download full text (20.2 KiB)

Ok, thanks, I can use that as a workaround.

On Fri, Jun 22, 2018, 7:30 AM Kenneth Loafman <email address hidden> wrote:

Ok, thanks, I can use that as a workaround.

On Fri, Jun 22, 2018, 7:30 AM Kenneth Loafman <kenneth@loafman.com> wrote:

> I'm not sure how big your backup is, but my most frequent suggestion is
> to get a USB hard drive and backup to that first using the file://
> schema.  Then, once you have verified that, you can rsync to whatever
> remote storage you need.  This will be much much faster.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1770200
>
> Title:
>   corrupt manifest disallows file restore
>
> Status in Duplicity:
>   In Progress
>
> Bug description:
>   First, thanks for creating duplicity. I hope I can use it to replace
>   my current commercial backup solution.
>
>   After 2 months of backing up, with a number of disconnects/resumes, my
>   backup seems to have succeeded. However, on my incremental backups, I
>   see the following in the log file:
>
>   Manifest file '' is corrupt: File count says 261395, File list
>   contains 547668
>
>   Besides this, I note there are several files in the include path that
>   Duplicity thinks are backed up, i.e. that aren't scheduled for upload
>   on the incremental. However, when I restore these files Duplicity says
>   it cannot find them.
>
>   Duplicity version: 0.7.14-0ubuntu0ppa1316~ubuntu16.04.1
>   Python version:  2.7.12
>   OS Distro / version:  Ubuntu 16.04.4 LTS
>   Target: B2 backblaze
>
>   Log output (deleted some stuff for privacy):
>   Using archive dir: /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68
>   Using backup name: a7f18478fedc07bafbb3e532dd999b68
>   Import of duplicity.backends.acdclibackend Succeeded
>   Import of duplicity.backends.azurebackend Succeeded
>   Import of duplicity.backends.b2backend Succeeded
>   Import of duplicity.backends.botobackend Succeeded
>   Import of duplicity.backends.cfbackend Succeeded
>   Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
>   Import of duplicity.backends.gdocsbackend Succeeded
>   Import of duplicity.backends.giobackend Succeeded
>   Import of duplicity.backends.hsibackend Succeeded
>   Import of duplicity.backends.hubicbackend Succeeded
>   Import of duplicity.backends.imapbackend Succeeded
>   Import of duplicity.backends.lftpbackend Succeeded
>   Import of duplicity.backends.localbackend Succeeded
>   Import of duplicity.backends.mediafirebackend Succeeded
>   Import of duplicity.backends.megabackend Succeeded
>   Import of duplicity.backends.multibackend Succeeded
>   Import of duplicity.backends.ncftpbackend Succeeded
>   Import of duplicity.backends.onedrivebackend Succeeded
>   Import of duplicity.backends.par2backend Succeeded
>   Import of duplicity.backends.pydrivebackend Succeeded
>   Import of duplicity.backends.rsyncbackend Succeeded
>   Import of duplicity.backends.ssh_paramiko_backend Succeeded
>   Import of duplicity.backends.ssh_pexpect_backend Succeeded
>   Import of duplicity.backends.swiftbackend Succeeded
>   Import of duplicity.backends.sxbackend Succeeded
>   Import of duplicity.backends.tahoebackend Succeeded
>   Import of duplicity.backends.webdavbackend Succeeded
>   Main action: inc
>
> ================================================================================
>   duplicity 0.7.14 ($reldate)
>   Args: /usr/bin/duplicity --num-retries 100 --archive-dir
> /tank/dec/duplicity -v8 --volsize 100 --encrypt-key FF05B6E2 --exclude
> /tank/dec/duplicity --exclude /root/.cache --exclude /usr/local/crashplan
> --exclude <deleted> --exclude /var/tmp --exclude /var/cache --exclude
> /var/lib/apt/lists --include <deleted> --include /tank/dec --include
> /tanky/dec --include /etc --include /home --include /var --include /usr/src
> --include /root --exclude ** / b2://<deleted>@duplicity-backup-bucket
>   Linux minime 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC
> 2018 x86_64 x86_64
>   /usr/bin/python 2.7.12 (default, Dec  4 2017, 14:50:18)
>   [GCC 5.4.0 20160609]
>
> ================================================================================
>   Using temporary directory /tmp/duplicity-tne17U-tempdir
>   Temp has 5762285568 available, backup will use approx 136314880.
>   Local and Remote metadata are synchronized, no sync needed.
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.part
> (7982)
>   Found 23 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-full.20171111T214344Z.manifest
> (148326939)
>   Found 1134 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171111T214344Z.to.20171206T042415Z.manifest
> (3516589)
>   Found 5 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171206T042415Z.to.20171206T153504Z.manifest
> (13401)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171206T153504Z.to.20171207T153504Z.manifest
> (3475485)
>   Found 342 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171207T153504Z.to.20171215T153505Z.manifest
> (1819905)
>   Found 3 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171215T153505Z.to.20171223T174005Z.manifest
> (66889806)
>   Manifest file '' is corrupt: File count says 261395, File list contains
> 547668
>   Found 3659 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171223T174005Z.to.20180225T153502Z.manifest
> (11793)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180225T153502Z.to.20180226T153504Z.manifest
> (11513)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180226T153504Z.to.20180228T153502Z.manifest
> (1690480)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180228T153502Z.to.20180301T153502Z.manifest
> (16324)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180301T153502Z.to.20180302T153502Z.manifest
> (17077)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180302T153502Z.to.20180303T070015Z.manifest
> (7107885)
>   Found 23 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180303T070015Z.to.20180303T170448Z.manifest
> (9726)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180303T170448Z.to.20180304T070003Z.manifest
> (3882284)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180304T070003Z.to.20180305T070003Z.manifest
> (120149)
>   Found 10 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180305T070003Z.to.20180306T070020Z.manifest
> (4030344)
>   Found 4 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180306T070020Z.to.20180307T070014Z.manifest
> (16765)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180307T070014Z.to.20180308T070003Z.manifest
> (19618)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180308T070003Z.to.20180309T070003Z.manifest
> (21724)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180309T070003Z.to.20180310T070003Z.manifest
> (20313)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180310T070003Z.to.20180310T170008Z.manifest
> (9847)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180310T170008Z.to.20180311T070002Z.manifest
> (16108)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180311T070002Z.to.20180312T060003Z.manifest
> (14393)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180312T060003Z.to.20180313T060003Z.manifest
> (1705638)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180313T060003Z.to.20180314T060014Z.manifest
> (17341)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180314T060014Z.to.20180315T060014Z.manifest
> (122553)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180315T060014Z.to.20180316T060014Z.manifest
> (12706)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180316T060014Z.to.20180317T060014Z.manifest
> (5322447)
>   Found 29 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180317T060014Z.to.20180319T060003Z.manifest
> (20739)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180319T060003Z.to.20180320T060005Z.manifest
> (2324706)
>   Found 9 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180320T060005Z.to.20180321T060003Z.manifest
> (1094227)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180321T060003Z.to.20180322T060009Z.manifest
> (9491)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180322T060009Z.to.20180323T060015Z.manifest
> (42734)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180323T060015Z.to.20180324T060015Z.manifest
> (9675)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180324T060015Z.to.20180324T155551Z.manifest
> (7294)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180324T155551Z.to.20180328T060009Z.manifest
> (28486817)
>   Found 83 volumes in manifest
>   Added incremental Backupset (start_time: Sat Nov 11 13:43:44 2017 /
> end_time: Tue Dec  5 20:24:15 2017)
>   Added incremental Backupset (start_time: Tue Dec  5 20:24:15 2017 /
> end_time: Wed Dec  6 07:35:04 2017)
>   Added incremental Backupset (start_time: Wed Dec  6 07:35:04 2017 /
> end_time: Thu Dec  7 07:35:04 2017)
>   Added incremental Backupset (start_time: Thu Dec  7 07:35:04 2017 /
> end_time: Fri Dec 15 07:35:05 2017)
>   Added incremental Backupset (start_time: Fri Dec 15 07:35:05 2017 /
> end_time: Sat Dec 23 09:40:05 2017)
>   Added incremental Backupset (start_time: Sat Dec 23 09:40:05 2017 /
> end_time: Sun Feb 25 07:35:02 2018)
>   Added incremental Backupset (start_time: Sun Feb 25 07:35:02 2018 /
> end_time: Mon Feb 26 07:35:04 2018)
>   Added incremental Backupset (start_time: Mon Feb 26 07:35:04 2018 /
> end_time: Wed Feb 28 07:35:02 2018)
>   Added incremental Backupset (start_time: Wed Feb 28 07:35:02 2018 /
> end_time: Thu Mar  1 07:35:02 2018)
>   Added incremental Backupset (start_time: Thu Mar  1 07:35:02 2018 /
> end_time: Fri Mar  2 07:35:02 2018)
>   Added incremental Backupset (start_time: Fri Mar  2 07:35:02 2018 /
> end_time: Fri Mar  2 23:00:15 2018)
>   Added incremental Backupset (start_time: Fri Mar  2 23:00:15 2018 /
> end_time: Sat Mar  3 09:04:48 2018)
>   Added incremental Backupset (start_time: Sat Mar  3 09:04:48 2018 /
> end_time: Sat Mar  3 23:00:03 2018)
>   Added incremental Backupset (start_time: Sat Mar  3 23:00:03 2018 /
> end_time: Sun Mar  4 23:00:03 2018)
>   Added incremental Backupset (start_time: Sun Mar  4 23:00:03 2018 /
> end_time: Mon Mar  5 23:00:20 2018)
>   Added incremental Backupset (start_time: Mon Mar  5 23:00:20 2018 /
> end_time: Tue Mar  6 23:00:14 2018)
>   Added incremental Backupset (start_time: Tue Mar  6 23:00:14 2018 /
> end_time: Wed Mar  7 23:00:03 2018)
>   Added incremental Backupset (start_time: Wed Mar  7 23:00:03 2018 /
> end_time: Thu Mar  8 23:00:03 2018)
>   Added incremental Backupset (start_time: Thu Mar  8 23:00:03 2018 /
> end_time: Fri Mar  9 23:00:03 2018)
>   Added incremental Backupset (start_time: Fri Mar  9 23:00:03 2018 /
> end_time: Sat Mar 10 09:00:08 2018)
>   Added incremental Backupset (start_time: Sat Mar 10 09:00:08 2018 /
> end_time: Sat Mar 10 23:00:02 2018)
>   Added incremental Backupset (start_time: Sat Mar 10 23:00:02 2018 /
> end_time: Sun Mar 11 23:00:03 2018)
>   Added incremental Backupset (start_time: Sun Mar 11 23:00:03 2018 /
> end_time: Mon Mar 12 23:00:03 2018)
>   Added incremental Backupset (start_time: Mon Mar 12 23:00:03 2018 /
> end_time: Tue Mar 13 23:00:14 2018)
>   Added incremental Backupset (start_time: Tue Mar 13 23:00:14 2018 /
> end_time: Wed Mar 14 23:00:14 2018)
>   Added incremental Backupset (start_time: Wed Mar 14 23:00:14 2018 /
> end_time: Thu Mar 15 23:00:14 2018)
>   Added incremental Backupset (start_time: Thu Mar 15 23:00:14 2018 /
> end_time: Fri Mar 16 23:00:14 2018)
>   Added incremental Backupset (start_time: Fri Mar 16 23:00:14 2018 /
> end_time: Sun Mar 18 23:00:03 2018)
>   Added incremental Backupset (start_time: Sun Mar 18 23:00:03 2018 /
> end_time: Mon Mar 19 23:00:05 2018)
>   Added incremental Backupset (start_time: Mon Mar 19 23:00:05 2018 /
> end_time: Tue Mar 20 23:00:03 2018)
>   Added incremental Backupset (start_time: Tue Mar 20 23:00:03 2018 /
> end_time: Wed Mar 21 23:00:09 2018)
>   Added incremental Backupset (start_time: Wed Mar 21 23:00:09 2018 /
> end_time: Thu Mar 22 23:00:15 2018)
>   Added incremental Backupset (start_time: Thu Mar 22 23:00:15 2018 /
> end_time: Fri Mar 23 23:00:15 2018)
>   Added incremental Backupset (start_time: Fri Mar 23 23:00:15 2018 /
> end_time: Sat Mar 24 08:55:51 2018)
>   Added incremental Backupset (start_time: Sat Mar 24 08:55:51 2018 /
> end_time: Tue Mar 27 23:00:09 2018)
>   Added incremental Backupset (start_time: Tue Mar 27 23:00:09 2018 /
> end_time: Sat Apr 21 20:45:30 2018)
>   Last inc backup left a partial set, restarting.
>   Last full backup date: Sat Nov 11 13:43:44 2017
>   Collection Status
>   -----------------
>   Connecting with backend: BackendWrapper
>   Archive dir: /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68
>
>   Found 0 secondary backup chains.
>
>   Found primary backup chain with matching signature chain:
>   -------------------------
>   Chain start time: Sat Nov 11 13:43:44 2017
>   Chain end time: Sat Apr 21 20:45:30 2018
>   Number of contained backup sets: 37
>   Total number of contained volumes: 5348
>    Type of backup set:                            Time:      Num volumes:
>                   Full         Sat Nov 11 13:43:44 2017              1134
>            Incremental         Tue Dec  5 20:24:15 2017                 5
>            Incremental         Wed Dec  6 07:35:04 2017                 1
>            Incremental         Thu Dec  7 07:35:04 2017               342
>            Incremental         Fri Dec 15 07:35:05 2017                 3
>            Incremental         Sat Dec 23 09:40:05 2017              3659
>            Incremental         Sun Feb 25 07:35:02 2018                 1
>            Incremental         Mon Feb 26 07:35:04 2018                 1
>            Incremental         Wed Feb 28 07:35:02 2018                 1
>            Incremental         Thu Mar  1 07:35:02 2018                 1
>            Incremental         Fri Mar  2 07:35:02 2018                 1
>            Incremental         Fri Mar  2 23:00:15 2018                23
>            Incremental         Sat Mar  3 09:04:48 2018                 1
>            Incremental         Sat Mar  3 23:00:03 2018                 1
>            Incremental         Sun Mar  4 23:00:03 2018                10
>            Incremental         Mon Mar  5 23:00:20 2018                 4
>            Incremental         Tue Mar  6 23:00:14 2018                 1
>            Incremental         Wed Mar  7 23:00:03 2018                 1
>            Incremental         Thu Mar  8 23:00:03 2018                 1
>            Incremental         Fri Mar  9 23:00:03 2018                 1
>            Incremental         Sat Mar 10 09:00:08 2018                 1
>            Incremental         Sat Mar 10 23:00:02 2018                 1
>            Incremental         Sun Mar 11 23:00:03 2018                 1
>            Incremental         Mon Mar 12 23:00:03 2018                 1
>            Incremental         Tue Mar 13 23:00:14 2018                 1
>            Incremental         Wed Mar 14 23:00:14 2018                 1
>            Incremental         Thu Mar 15 23:00:14 2018                 1
>            Incremental         Fri Mar 16 23:00:14 2018                29
>            Incremental         Sun Mar 18 23:00:03 2018                 1
>            Incremental         Mon Mar 19 23:00:05 2018                 9
>            Incremental         Tue Mar 20 23:00:03 2018                 1
>            Incremental         Wed Mar 21 23:00:09 2018                 1
>            Incremental         Thu Mar 22 23:00:15 2018                 1
>            Incremental         Fri Mar 23 23:00:15 2018                 1
>            Incremental         Sat Mar 24 08:55:51 2018                 1
>            Incremental         Tue Mar 27 23:00:09 2018                83
>            Incremental         Sat Apr 21 20:45:30 2018                22
>   -------------------------
>   No orphaned or incomplete backup sets found.
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.part
> (7982)
>   Found 23 volumes in manifest
>   RESTART: Volumes 22 to 23 failed to upload before termination.
>            Restarting backup at volume 22.
>   Deleting /tmp/duplicity-tne17U-tempdir/mktemp-gLW__x-2
>
>   <deleted body>
>
>   AsyncScheduler: running task synchronously (asynchronicity disabled)
>   Writing
> duplicity-inc.20180328T060009Z.to.20180422T034530Z.vol36.difftar.gpg
>   Deleting /tmp/duplicity-tne17U-tempdir/mktemp-PgtBD7-17
>   AsyncScheduler: task completed successfully
>   Processed volume 36
>   Writing
> duplicity-new-signatures.20180328T060009Z.to.20180422T034530Z.sigtar.gpg
>   Deleting
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-new-signatures.20180328T060009Z.to.20180422T034530Z.sigtar.gpg
>   Writing duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.gpg
>   Deleting
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.gpg
>   --------------[ Backup Statistics ]--------------
>   StartTime 1524412692.62 (Sun Apr 22 08:58:12 2018)
>   EndTime 1524429108.67 (Sun Apr 22 13:31:48 2018)
>   ElapsedTime 16416.06 (4 hours 33 minutes 36.06 seconds)
>   SourceFiles 1268016
>   SourceFileSize 562498815668 (524 GB)
>   NewFiles 54550
>   NewFileSize 3728199214 (3.47 GB)
>   DeletedFiles 44996
>   ChangedFiles 3662
>   ChangedFileSize 11806997452 (11.0 GB)
>   ChangedDeltaSize 0 (0 bytes)
>   DeltaEntries 103208
>   RawDeltaSize 4974760234 (4.63 GB)
>   TotalDestinationSizeChange 1483159528 (1.38 GB)
>   Errors 0
>   -------------------------------------------------
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1770200/+subscriptions
>

Kenneth Loafman (kenneth-loafman) on 2018-08-21

Changed in duplicity:
milestone:	0.7.18 → 0.7.19

Kenneth Loafman (kenneth-loafman) on 2019-04-29

Changed in duplicity:
milestone:	0.7.19 → 0.7.20

Kenneth Loafman (kenneth-loafman) on 2020-02-09

Changed in duplicity:
milestone:	0.7.20 → 0.8.12

Revision history for this message

Kenneth Loafman (kenneth-loafman) wrote on 2020-02-25:

Corrupt files are a fact of life in flaky networks. Use par2 or similar to help correct this problem.

Changed in duplicity:
assignee:	Kenneth Loafman (kenneth-loafman) → nobody
importance:	Medium → Undecided
milestone:	0.8.12 → none
status:	In Progress → Won't Fix

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.