Duplicity is failing with an assertion error when looking for a non-existent manifest file

[Pete Goodall]

Binary package hint: duplicity

Dupliciity refuses to back my files and throws up an assertion error where it appears to be trying to use a non-existent manifest.part file. That file doesn't exist and I cannot find where it is referenced. Here is the traceback:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1251, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1244, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1149, in main
    globals.archive_dir).set_values()
  File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 676, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 799, in get_backup_chains
    map(add_to_sets, filename_list)
  File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 789, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 89, in add_filename
    self.set_manifest(filename)
  File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 118, in set_manifest
    remote_filename)
AssertionError: ('duplicity-full.20100817T193647Z.manifest.part', 'duplicity-full.20100817T193647Z.manifest')

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: duplicity 0.6.09-0ubuntu2
ProcVersionSignature: Ubuntu 2.6.35-20.29-generic 2.6.35.4
Uname: Linux 2.6.35-20-generic i686
Architecture: i386
Date: Wed Sep 8 12:18:29 2010
InstallationMedia: Ubuntu-Netbook 10.10 "Maverick Meerkat" - Alpha i386 (20100803.1)
ProcEnviron:
 PATH=(custom, user)
 LANG=en_GB.utf8
 SHELL=/bin/bash
SourcePackage: duplicity

[Olivier Berger (oberger)]

Similar problem here (Debian stable with 0.6.08b-1+b1), with the following trace :

Info: File duplicity-inc.20101225T001425Z.to.20101226T001506Z.manifest.part is not part of a known set; creating new set
Info: File duplicity-full-signatures.20101222T161821Z.sigtar.gpg is not part of a known set; creating new set
Info: Ignoring file (rejected by backup set) 'duplicity-full-signatures.20101222T161821Z.sigtar.gpg'
Info: File duplicity-full.20101222T161821Z.manifest.gpg is not part of a known set; creating new set
Info: File duplicity-full.20101222T161821Z.vol1.difftar.gpg is part of known set
Info: File duplicity-full.20101222T161821Z.vol2.difftar.gpg is part of known set
Info: File duplicity-full.20101222T161821Z.vol3.difftar.gpg is part of known set
Info: File duplicity-inc.20101222T161821Z.to.20101223T002132Z.manifest.gpg is not part of a known set; creating new set
Info: File duplicity-inc.20101222T161821Z.to.20101223T002132Z.vol1.difftar.gpg is part of known set
Info: File duplicity-inc.20101223T002132Z.to.20101224T001431Z.manifest.gpg is not part of a known set; creating new set
Info: File duplicity-inc.20101223T002132Z.to.20101224T001431Z.vol1.difftar.gpg is part of known set
Info: File duplicity-inc.20101224T001431Z.to.20101225T001425Z.manifest.gpg is not part of a known set; creating new set
Info: File duplicity-inc.20101224T001431Z.to.20101225T001425Z.vol1.difftar.gpg is part of known set
Info: Removing still remembered temporary file /tmp/duplicity-g1LQZX-tempdir/mkstemp-zwaTeY-1
Info: Traceback (most recent call last):
Info: File "/usr/bin/duplicity", line 1251, in <module>
Info: with_tempdir(main)
Info: File "/usr/bin/duplicity", line 1244, in with_tempdir
Info: fn()
Info: File "/usr/bin/duplicity", line 1149, in main
Info: globals.archive_dir).set_values()
Info: File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 676, in set_values
Info: self.get_backup_chains(partials + backend_filename_list)
Info: File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 799, in get_backup_chains
Info: map(add_to_sets, filename_list)
Info: File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 789, in add_to_sets
Info: if set.add_filename(filename):
Info: File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 89, in add_filename
Info: self.set_manifest(filename)
Info: File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 118, in set_manifest
Info: remote_filename)
Info: AssertionError: ('duplicity-inc.20101225T001425Z.to.20101226T001506Z.manifest.part', 'duplicity-inc.20101225T001425Z.to.20101226T001506Z.manifest.gpg')
Fatal: Duplicity failed.

I noticed that the file in the local cache was more or less normal (size > 0), whereas the distant one was of size 0.

I guess that situation has happened as the file was transferred once on the distant server, which was over quota, so the file was created with size 0... then the resulting failures on next run.

The situation may be a bit different from the other reporter's, but duplicity should try and handle such events better or at least explain such failures in more details.

Hope this helps.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[skorqa (skorqa)]

Hi everyone,

I'm using duplicity: 0.6.18-0ubuntu0ppa14~lucid1 and am experiencing the same bug.

The ~/.cache/deja-dup has been erased.
Deja-dup / duplicity has been working for more than 6 months and is failing since March.

The same bug seems to have been reported here:
https://answers.launchpad.net/duplicity/+question/141176
but with no clue.

Please find the terminal output of
DEJA_DUP_DEBUG=1 deja-dup --backup

in the attached file. (dejaduplog20120425_2_stdout.txt.zip)

You'll also find the 'ls -a' of the archive directory (files_20120425.txt.zip)

Cheers
F

[skorqa (skorqa)]

Hello,

These clues could help.
The presence of the 'manifest' file without '.gpg' extension seemed odd.

I tried to:
* rename it to 'duplicity-full.20120301T071617Z.manifest.txt' . The assertion became :
AssertionError: ('duplicity-full.20120301T071617Z.manifest.gpg', 'duplicity-full.20120301T071617Z.manifest.txt')

* move it to another location
The backup worked!

F

[Peter Meier (peter-meier)]

I'm having the same problem from time to time:

Local and Remote metadata are synchronized, no sync needed.
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1377, in ?
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1370, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1250, in main
    globals.archive_dir).set_values()
  File "/usr/lib64/python2.4/site-packages/duplicity/collections.py", line 694, in set_values
    (backup_chains, self.orphaned_backup_sets,
  File "/usr/lib64/python2.4/site-packages/duplicity/collections.py", line 819, in get_backup_chains
    map(add_to_sets, filename_list)
  File "/usr/lib64/python2.4/site-packages/duplicity/collections.py", line 809, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib64/python2.4/site-packages/duplicity/collections.py", line 93, in add_filename
    self.set_manifest(filename)
  File "/usr/lib64/python2.4/site-packages/duplicity/collections.py", line 123, in set_manifest
    assert not self.remote_manifest_name, (self.remote_manifest_name,
AssertionError: ('duplicity-inc.20120507T053251Z.to.20120509T042329Z.manifest.part', 'duplicity-inc.20120507T053251Z.to.20120509T042329Z.manifest.gpg')

Removing the file in question on the remote side usually fixes it.

[Ian Chard (ian-chard)]

Same problem here, using duplicity 0.6.18. This was from a 'collection-status' command:

Local and Remote metadata are synchronized, no sync needed.
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1404, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1397, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1277, in main
    globals.archive_dir).set_values()
  File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 691, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 814, in get_backup_chains
    map(add_to_sets, filename_list)
  File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 804, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 93, in add_filename
    self.set_manifest(filename)
  File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 124, in set_manifest
    remote_filename)
AssertionError: ('duplicity-inc.20121213T042623Z.to.20121214T051524Z.manifest.part', 'duplicity-inc.20121213T042623Z.to.20121214T051524Z.manifest.gpg')

As above, removing the file makes the problem go away.

[Phill (phill.l)]

I get this occasionally over an SFTP interface to a remote block storage system from time-to-time. Due to their implementation there is an occasional and variable but short delay in newly written files becoming accessible in subsequent requests. As its a question of timing it usually doesn't cause a problem, but sometimes it does and then duplicity is unable to perform subsequent backups without manually wiping the local cache.

Note the first run terminated with the error `Giving up after 1 attempts. IOError: [Errno 2] No such file`. Followed by the assertion errors described above for all subsequent runs until the cache is cleared.

We may take the view that it is reasonable for the program to give up when during the original error on the remote, but I think it should to handle corruption in its own cache without manual intervention.

[Juergen Meixner (juergen-sly)]

Duplicity fails since a few weeks, have updated my desktop to Ubuntu 15.04 and cleaned up everything, de-installed Duplicity and re-installed it, cleaned the harddrive from all old duplicity files but still won't work.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1500, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1494, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1343, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1376, in do_backup
    globals.archive_dir).set_values()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 698, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 821, in get_backup_chains
    add_to_sets(f)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 810, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 102, in add_filename
    (self.volume_name_dict, filename)
AssertionError: ({2: 'duplicity-inc.20161110T115644Z.to.20161120T105000Z.vol2.difftar.gpg', 3: 'duplicity-inc.20161110T115644Z.to.20161120T105000Z.vol3.difftar.gpg'}, 'duplicity-inc.20161110T115644Z.to.20161120T105000Z.vol3.difftar.gpg')

No clue what this means, do I need a different backup program now?.