Ok, thanks, I can use that as a workaround.



On Fri, Jun 22, 2018, 7:30 AM Kenneth Loafman <email address hidden> wrote:

> I'm not sure how big your backup is, but my most frequent suggestion is
> to get a USB hard drive and backup to that first using the file://
> schema.  Then, once you have verified that, you can rsync to whatever
> remote storage you need.  This will be much much faster.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1770200
>
> Title:
>   corrupt manifest disallows file restore
>
> Status in Duplicity:
>   In Progress
>
> Bug description:
>   First, thanks for creating duplicity. I hope I can use it to replace
>   my current commercial backup solution.
>
>   After 2 months of backing up, with a number of disconnects/resumes, my
>   backup seems to have succeeded. However, on my incremental backups, I
>   see the following in the log file:
>
>   Manifest file '' is corrupt: File count says 261395, File list
>   contains 547668
>
>   Besides this, I note there are several files in the include path that
>   Duplicity thinks are backed up, i.e. that aren't scheduled for upload
>   on the incremental. However, when I restore these files Duplicity says
>   it cannot find them.
>
>   Duplicity version: 0.7.14-0ubuntu0ppa1316~ubuntu16.04.1
>   Python version:  2.7.12
>   OS Distro / version:  Ubuntu 16.04.4 LTS
>   Target: B2 backblaze
>
>   Log output (deleted some stuff for privacy):
>   Using archive dir: /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68
>   Using backup name: a7f18478fedc07bafbb3e532dd999b68
>   Import of duplicity.backends.acdclibackend Succeeded
>   Import of duplicity.backends.azurebackend Succeeded
>   Import of duplicity.backends.b2backend Succeeded
>   Import of duplicity.backends.botobackend Succeeded
>   Import of duplicity.backends.cfbackend Succeeded
>   Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
>   Import of duplicity.backends.gdocsbackend Succeeded
>   Import of duplicity.backends.giobackend Succeeded
>   Import of duplicity.backends.hsibackend Succeeded
>   Import of duplicity.backends.hubicbackend Succeeded
>   Import of duplicity.backends.imapbackend Succeeded
>   Import of duplicity.backends.lftpbackend Succeeded
>   Import of duplicity.backends.localbackend Succeeded
>   Import of duplicity.backends.mediafirebackend Succeeded
>   Import of duplicity.backends.megabackend Succeeded
>   Import of duplicity.backends.multibackend Succeeded
>   Import of duplicity.backends.ncftpbackend Succeeded
>   Import of duplicity.backends.onedrivebackend Succeeded
>   Import of duplicity.backends.par2backend Succeeded
>   Import of duplicity.backends.pydrivebackend Succeeded
>   Import of duplicity.backends.rsyncbackend Succeeded
>   Import of duplicity.backends.ssh_paramiko_backend Succeeded
>   Import of duplicity.backends.ssh_pexpect_backend Succeeded
>   Import of duplicity.backends.swiftbackend Succeeded
>   Import of duplicity.backends.sxbackend Succeeded
>   Import of duplicity.backends.tahoebackend Succeeded
>   Import of duplicity.backends.webdavbackend Succeeded
>   Main action: inc
>
> ================================================================================
>   duplicity 0.7.14 ($reldate)
>   Args: /usr/bin/duplicity --num-retries 100 --archive-dir
> /tank/dec/duplicity -v8 --volsize 100 --encrypt-key FF05B6E2 --exclude
> /tank/dec/duplicity --exclude /root/.cache --exclude /usr/local/crashplan
> --exclude <deleted> --exclude /var/tmp --exclude /var/cache --exclude
> /var/lib/apt/lists --include <deleted> --include /tank/dec --include
> /tanky/dec --include /etc --include /home --include /var --include /usr/src
> --include /root --exclude ** / b2://<deleted>@duplicity-backup-bucket
>   Linux minime 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC
> 2018 x86_64 x86_64
>   /usr/bin/python 2.7.12 (default, Dec  4 2017, 14:50:18)
>   [GCC 5.4.0 20160609]
>
> ================================================================================
>   Using temporary directory /tmp/duplicity-tne17U-tempdir
>   Temp has 5762285568 available, backup will use approx 136314880.
>   Local and Remote metadata are synchronized, no sync needed.
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.part
> (7982)
>   Found 23 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-full.20171111T214344Z.manifest
> (148326939)
>   Found 1134 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171111T214344Z.to.20171206T042415Z.manifest
> (3516589)
>   Found 5 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171206T042415Z.to.20171206T153504Z.manifest
> (13401)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171206T153504Z.to.20171207T153504Z.manifest
> (3475485)
>   Found 342 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171207T153504Z.to.20171215T153505Z.manifest
> (1819905)
>   Found 3 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171215T153505Z.to.20171223T174005Z.manifest
> (66889806)
>   Manifest file '' is corrupt: File count says 261395, File list contains
> 547668
>   Found 3659 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20171223T174005Z.to.20180225T153502Z.manifest
> (11793)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180225T153502Z.to.20180226T153504Z.manifest
> (11513)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180226T153504Z.to.20180228T153502Z.manifest
> (1690480)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180228T153502Z.to.20180301T153502Z.manifest
> (16324)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180301T153502Z.to.20180302T153502Z.manifest
> (17077)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180302T153502Z.to.20180303T070015Z.manifest
> (7107885)
>   Found 23 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180303T070015Z.to.20180303T170448Z.manifest
> (9726)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180303T170448Z.to.20180304T070003Z.manifest
> (3882284)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180304T070003Z.to.20180305T070003Z.manifest
> (120149)
>   Found 10 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180305T070003Z.to.20180306T070020Z.manifest
> (4030344)
>   Found 4 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180306T070020Z.to.20180307T070014Z.manifest
> (16765)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180307T070014Z.to.20180308T070003Z.manifest
> (19618)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180308T070003Z.to.20180309T070003Z.manifest
> (21724)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180309T070003Z.to.20180310T070003Z.manifest
> (20313)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180310T070003Z.to.20180310T170008Z.manifest
> (9847)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180310T170008Z.to.20180311T070002Z.manifest
> (16108)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180311T070002Z.to.20180312T060003Z.manifest
> (14393)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180312T060003Z.to.20180313T060003Z.manifest
> (1705638)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180313T060003Z.to.20180314T060014Z.manifest
> (17341)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180314T060014Z.to.20180315T060014Z.manifest
> (122553)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180315T060014Z.to.20180316T060014Z.manifest
> (12706)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180316T060014Z.to.20180317T060014Z.manifest
> (5322447)
>   Found 29 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180317T060014Z.to.20180319T060003Z.manifest
> (20739)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180319T060003Z.to.20180320T060005Z.manifest
> (2324706)
>   Found 9 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180320T060005Z.to.20180321T060003Z.manifest
> (1094227)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180321T060003Z.to.20180322T060009Z.manifest
> (9491)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180322T060009Z.to.20180323T060015Z.manifest
> (42734)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180323T060015Z.to.20180324T060015Z.manifest
> (9675)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180324T060015Z.to.20180324T155551Z.manifest
> (7294)
>   Found 1 volumes in manifest
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180324T155551Z.to.20180328T060009Z.manifest
> (28486817)
>   Found 83 volumes in manifest
>   Added incremental Backupset (start_time: Sat Nov 11 13:43:44 2017 /
> end_time: Tue Dec  5 20:24:15 2017)
>   Added incremental Backupset (start_time: Tue Dec  5 20:24:15 2017 /
> end_time: Wed Dec  6 07:35:04 2017)
>   Added incremental Backupset (start_time: Wed Dec  6 07:35:04 2017 /
> end_time: Thu Dec  7 07:35:04 2017)
>   Added incremental Backupset (start_time: Thu Dec  7 07:35:04 2017 /
> end_time: Fri Dec 15 07:35:05 2017)
>   Added incremental Backupset (start_time: Fri Dec 15 07:35:05 2017 /
> end_time: Sat Dec 23 09:40:05 2017)
>   Added incremental Backupset (start_time: Sat Dec 23 09:40:05 2017 /
> end_time: Sun Feb 25 07:35:02 2018)
>   Added incremental Backupset (start_time: Sun Feb 25 07:35:02 2018 /
> end_time: Mon Feb 26 07:35:04 2018)
>   Added incremental Backupset (start_time: Mon Feb 26 07:35:04 2018 /
> end_time: Wed Feb 28 07:35:02 2018)
>   Added incremental Backupset (start_time: Wed Feb 28 07:35:02 2018 /
> end_time: Thu Mar  1 07:35:02 2018)
>   Added incremental Backupset (start_time: Thu Mar  1 07:35:02 2018 /
> end_time: Fri Mar  2 07:35:02 2018)
>   Added incremental Backupset (start_time: Fri Mar  2 07:35:02 2018 /
> end_time: Fri Mar  2 23:00:15 2018)
>   Added incremental Backupset (start_time: Fri Mar  2 23:00:15 2018 /
> end_time: Sat Mar  3 09:04:48 2018)
>   Added incremental Backupset (start_time: Sat Mar  3 09:04:48 2018 /
> end_time: Sat Mar  3 23:00:03 2018)
>   Added incremental Backupset (start_time: Sat Mar  3 23:00:03 2018 /
> end_time: Sun Mar  4 23:00:03 2018)
>   Added incremental Backupset (start_time: Sun Mar  4 23:00:03 2018 /
> end_time: Mon Mar  5 23:00:20 2018)
>   Added incremental Backupset (start_time: Mon Mar  5 23:00:20 2018 /
> end_time: Tue Mar  6 23:00:14 2018)
>   Added incremental Backupset (start_time: Tue Mar  6 23:00:14 2018 /
> end_time: Wed Mar  7 23:00:03 2018)
>   Added incremental Backupset (start_time: Wed Mar  7 23:00:03 2018 /
> end_time: Thu Mar  8 23:00:03 2018)
>   Added incremental Backupset (start_time: Thu Mar  8 23:00:03 2018 /
> end_time: Fri Mar  9 23:00:03 2018)
>   Added incremental Backupset (start_time: Fri Mar  9 23:00:03 2018 /
> end_time: Sat Mar 10 09:00:08 2018)
>   Added incremental Backupset (start_time: Sat Mar 10 09:00:08 2018 /
> end_time: Sat Mar 10 23:00:02 2018)
>   Added incremental Backupset (start_time: Sat Mar 10 23:00:02 2018 /
> end_time: Sun Mar 11 23:00:03 2018)
>   Added incremental Backupset (start_time: Sun Mar 11 23:00:03 2018 /
> end_time: Mon Mar 12 23:00:03 2018)
>   Added incremental Backupset (start_time: Mon Mar 12 23:00:03 2018 /
> end_time: Tue Mar 13 23:00:14 2018)
>   Added incremental Backupset (start_time: Tue Mar 13 23:00:14 2018 /
> end_time: Wed Mar 14 23:00:14 2018)
>   Added incremental Backupset (start_time: Wed Mar 14 23:00:14 2018 /
> end_time: Thu Mar 15 23:00:14 2018)
>   Added incremental Backupset (start_time: Thu Mar 15 23:00:14 2018 /
> end_time: Fri Mar 16 23:00:14 2018)
>   Added incremental Backupset (start_time: Fri Mar 16 23:00:14 2018 /
> end_time: Sun Mar 18 23:00:03 2018)
>   Added incremental Backupset (start_time: Sun Mar 18 23:00:03 2018 /
> end_time: Mon Mar 19 23:00:05 2018)
>   Added incremental Backupset (start_time: Mon Mar 19 23:00:05 2018 /
> end_time: Tue Mar 20 23:00:03 2018)
>   Added incremental Backupset (start_time: Tue Mar 20 23:00:03 2018 /
> end_time: Wed Mar 21 23:00:09 2018)
>   Added incremental Backupset (start_time: Wed Mar 21 23:00:09 2018 /
> end_time: Thu Mar 22 23:00:15 2018)
>   Added incremental Backupset (start_time: Thu Mar 22 23:00:15 2018 /
> end_time: Fri Mar 23 23:00:15 2018)
>   Added incremental Backupset (start_time: Fri Mar 23 23:00:15 2018 /
> end_time: Sat Mar 24 08:55:51 2018)
>   Added incremental Backupset (start_time: Sat Mar 24 08:55:51 2018 /
> end_time: Tue Mar 27 23:00:09 2018)
>   Added incremental Backupset (start_time: Tue Mar 27 23:00:09 2018 /
> end_time: Sat Apr 21 20:45:30 2018)
>   Last inc backup left a partial set, restarting.
>   Last full backup date: Sat Nov 11 13:43:44 2017
>   Collection Status
>   -----------------
>   Connecting with backend: BackendWrapper
>   Archive dir: /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68
>
>   Found 0 secondary backup chains.
>
>   Found primary backup chain with matching signature chain:
>   -------------------------
>   Chain start time: Sat Nov 11 13:43:44 2017
>   Chain end time: Sat Apr 21 20:45:30 2018
>   Number of contained backup sets: 37
>   Total number of contained volumes: 5348
>    Type of backup set:                            Time:      Num volumes:
>                   Full         Sat Nov 11 13:43:44 2017              1134
>            Incremental         Tue Dec  5 20:24:15 2017                 5
>            Incremental         Wed Dec  6 07:35:04 2017                 1
>            Incremental         Thu Dec  7 07:35:04 2017               342
>            Incremental         Fri Dec 15 07:35:05 2017                 3
>            Incremental         Sat Dec 23 09:40:05 2017              3659
>            Incremental         Sun Feb 25 07:35:02 2018                 1
>            Incremental         Mon Feb 26 07:35:04 2018                 1
>            Incremental         Wed Feb 28 07:35:02 2018                 1
>            Incremental         Thu Mar  1 07:35:02 2018                 1
>            Incremental         Fri Mar  2 07:35:02 2018                 1
>            Incremental         Fri Mar  2 23:00:15 2018                23
>            Incremental         Sat Mar  3 09:04:48 2018                 1
>            Incremental         Sat Mar  3 23:00:03 2018                 1
>            Incremental         Sun Mar  4 23:00:03 2018                10
>            Incremental         Mon Mar  5 23:00:20 2018                 4
>            Incremental         Tue Mar  6 23:00:14 2018                 1
>            Incremental         Wed Mar  7 23:00:03 2018                 1
>            Incremental         Thu Mar  8 23:00:03 2018                 1
>            Incremental         Fri Mar  9 23:00:03 2018                 1
>            Incremental         Sat Mar 10 09:00:08 2018                 1
>            Incremental         Sat Mar 10 23:00:02 2018                 1
>            Incremental         Sun Mar 11 23:00:03 2018                 1
>            Incremental         Mon Mar 12 23:00:03 2018                 1
>            Incremental         Tue Mar 13 23:00:14 2018                 1
>            Incremental         Wed Mar 14 23:00:14 2018                 1
>            Incremental         Thu Mar 15 23:00:14 2018                 1
>            Incremental         Fri Mar 16 23:00:14 2018                29
>            Incremental         Sun Mar 18 23:00:03 2018                 1
>            Incremental         Mon Mar 19 23:00:05 2018                 9
>            Incremental         Tue Mar 20 23:00:03 2018                 1
>            Incremental         Wed Mar 21 23:00:09 2018                 1
>            Incremental         Thu Mar 22 23:00:15 2018                 1
>            Incremental         Fri Mar 23 23:00:15 2018                 1
>            Incremental         Sat Mar 24 08:55:51 2018                 1
>            Incremental         Tue Mar 27 23:00:09 2018                83
>            Incremental         Sat Apr 21 20:45:30 2018                22
>   -------------------------
>   No orphaned or incomplete backup sets found.
>   Processing local manifest
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.part
> (7982)
>   Found 23 volumes in manifest
>   RESTART: Volumes 22 to 23 failed to upload before termination.
>            Restarting backup at volume 22.
>   Deleting /tmp/duplicity-tne17U-tempdir/mktemp-gLW__x-2
>
>   <deleted body>
>
>   AsyncScheduler: running task synchronously (asynchronicity disabled)
>   Writing
> duplicity-inc.20180328T060009Z.to.20180422T034530Z.vol36.difftar.gpg
>   Deleting /tmp/duplicity-tne17U-tempdir/mktemp-PgtBD7-17
>   AsyncScheduler: task completed successfully
>   Processed volume 36
>   Writing
> duplicity-new-signatures.20180328T060009Z.to.20180422T034530Z.sigtar.gpg
>   Deleting
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-new-signatures.20180328T060009Z.to.20180422T034530Z.sigtar.gpg
>   Writing duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.gpg
>   Deleting
> /tank/dec/duplicity/a7f18478fedc07bafbb3e532dd999b68/duplicity-inc.20180328T060009Z.to.20180422T034530Z.manifest.gpg
>   --------------[ Backup Statistics ]--------------
>   StartTime 1524412692.62 (Sun Apr 22 08:58:12 2018)
>   EndTime 1524429108.67 (Sun Apr 22 13:31:48 2018)
>   ElapsedTime 16416.06 (4 hours 33 minutes 36.06 seconds)
>   SourceFiles 1268016
>   SourceFileSize 562498815668 (524 GB)
>   NewFiles 54550
>   NewFileSize 3728199214 (3.47 GB)
>   DeletedFiles 44996
>   ChangedFiles 3662
>   ChangedFileSize 11806997452 (11.0 GB)
>   ChangedDeltaSize 0 (0 bytes)
>   DeltaEntries 103208
>   RawDeltaSize 4974760234 (4.63 GB)
>   TotalDestinationSizeChange 1483159528 (1.38 GB)
>   Errors 0
>   -------------------------------------------------
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1770200/+subscriptions
>