Duplicity crashes when verifying

[Mattias Månsson]

Done a backup on my server, it's mostly photos. First I just did a full backup of some folder, then added the rest with incremental. After completion I tried a verify on my pictures folder just to check that it worked. After a long time I get this error:

# duplicity verify --file-to-restore pub/pictures dpbx:/// /pub/pictures

Duplicity 0.6 series is being deprecated:
See http://www.nongnu.org/duplicity/

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Apr 14 21:34:46 2015
GnuPG passphrase:
Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 1509, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1503, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1352, in main
    do_backup(action)
  File "/usr/local/bin/duplicity", line 1439, in do_backup
    verify(col_stats)
  File "/usr/local/bin/duplicity", line 827, in verify
    for backup_ropath, current_path in collated:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 271, in collate2iters
    relem1 = riter1.next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 499, in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 379, in yield_tuples
    setrorps( overflow, elems )
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 368, in setrorps
    elems[i] = iter_list[i].next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 99, in filter_path_iter
    for path in path_iter:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 119, in difftar2path_iter
    multivol_fileobj.close() # aborting in middle of multivol
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 239, in close
    if not self.addtobuffer():
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 227, in addtobuffer
    self.tarinfo_list[0] = self.tar_iter.next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 334, in next
    self.set_tarfile()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 323, in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "/usr/local/bin/duplicity", line 729, in get_fileobj_iter
    backup_set.volume_name_dict[vol_num],
KeyError: 636

Duplicity 0.6.25
Python 2.7.3
Ubuntu Server 12.04.5 x86_64
Filesystem: ext4 on mdraid

[macho (macho)]

Duplicity just crashed for me on the seemingly equivalent line in version 2.7.10. This is with Python 2.7.10 in Ubuntu Desktop 15.10. In my case this happened after making an initial full backup without any subsequent incremental backups.

Mattias, did this problem continue for you?

Here are the two invocations I ran and their associated output:

# duplicity --full-if-older-than 2W --exclude-other-filesystems --exclude="$home_source" $root_source $root_target
<snip some lines>
--------------[ Backup Statistics ]--------------
StartTime 1451684678.34 (Fri Jan 1 16:44:38 2016)
EndTime 1451707485.17 (Fri Jan 1 23:04:45 2016)
ElapsedTime 22806.83 (6 hours 20 minutes 6.83 seconds)
SourceFiles 450744
SourceFileSize 11054488128 (10.3 GB)
NewFiles 450744
NewFileSize 11051545499 (10.3 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 450744
RawDeltaSize 10805130919 (10.1 GB)
TotalDestinationSizeChange 5788143896 (5.39 GB)
Errors 0
-------------------------------------------------

# duplicity verify $root_target $root_source
Synchronizing remote metadata to local cache...
Deleting local /root/.cache/duplicity/2b0a3987b2b7b2bac8c5f83e77ca0f11/duplicity-full-signatures.20160101T214438Z.sigtar.gz (not authoritative at backend).
Last full backup date: Fri Jan 1 16:44:38 2016
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1519, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1513, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1370, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1449, in do_backup
    verify(col_stats)
  File "/usr/bin/duplicity", line 852, in verify
    for backup_ropath, current_path in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 276, in collate2iters
    relem1 = riter1.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 516, in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 388, in yield_tuples
    setrorps(overflow, elems)
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 377, in setrorps
    elems[i] = iter_list[i].next()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 127, in difftar2path_iter
    multivol_fileobj.close() # aborting in middle of multivol
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 248, in close
    if not self.addtobuffer():
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 236, in addtobuffer
    self.tarinfo_list[0] = self.tar_iter.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 343, in next
    self.set_tarfile()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 332, in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "/usr/bin/duplicity", line 751, in get_fileobj_iter
    backup_set.volume_name_dict[vol_num],
KeyError: 4

[Mattias Månsson (mattias-mansson)]

I actually gave up on duplicity because I didn't feel I could trust it in combination with Dropbox. I switched to paying for Crashplan instead and is very happy with that. Hope you solve your issues.

[Steve Madsen (sjmadsen)]

I think bug #1632858 is a duplicate of this one.

I just started hitting this problem today, after upgrading a Debian server from jessie to stretch yesterday. Duplicity version 0.7.11.

I tried moving the old archive directory out of the way to force it to re-sync from the remote. The error still occurs. I'm using the ssh backend.