0.6.06, archive dir: cache desynchronization caused by remove*

[az]

note: this applies to 0.6.06 only, 0.6.05 works fine.

setup: i'm running duplicity gpg-encrypt-only to an rsync backend and the backup job has no access to
the private key - so any cache desync that requires stuff to be recopied from the backend and then
decrypted causes that backup run to fail. up to 0.6.05 that works perfectly on multiple machines.

0.6.06 does reject numerous duplicity-owned files on the backend (that's what -v 9 says, but the relevant
code doesn't tell me anything useful why the rejection happens): neither remove-if-older nor remove-all-but...
cleans them up, but they do get removed from the local cache (apparently every second run) - which causes
my resync problem (and which made the problem visible).

here's some evidence: this is the file listing on the backend after a remove-all-but-n-full 1:
duplicity-full-signatures.20091210T170421Z.sigtar.gpg
duplicity-full-signatures.20091214T170422Z.sigtar.gpg
duplicity-full.20091214T170422Z.manifest.gpg
duplicity-full.20091214T170422Z.vol1.difftar.gpg
...snipped to vol 67...
duplicity-inc.20091214T170422Z.to.20091216T010123Z.manifest.gpg
duplicity-inc.20091214T170422Z.to.20091216T010123Z.vol1.difftar.gpg
duplicity-inc.20091216T010123Z.to.20091216T010714Z.manifest.gpg
duplicity-inc.20091216T010123Z.to.20091216T010714Z.vol1.difftar.gpg
duplicity-new-signatures.20091210T170421Z.to.20091212T005914Z.sigtar.gpg
duplicity-new-signatures.20091212T005914Z.to.20091212T010626Z.sigtar.gpg
duplicity-new-signatures.20091212T010626Z.to.20091212T170426Z.sigtar.gpg
duplicity-new-signatures.20091212T170426Z.to.20091213T170404Z.sigtar.gpg
duplicity-new-signatures.20091214T170422Z.to.20091216T010123Z.sigtar.gpg
duplicity-new-signatures.20091216T010123Z.to.20091216T010714Z.sigtar.gpg

none of the 2009-12-10 stuff should be present anymore.

i'm attaching a redacted (minus key info and hostnames) logfile of collection-status -v 9 which
mentions the rejected older files and confirms that the one and only full backup is the 20091214 one (my timezone is GMT+10). the logfile then continues with a cleanup -v 9, which shows the same rejects and failure to clean
things up.

regards
az

regards
az

[az (az-debian)]

i've spent some more effort on tedious debugging but have found the problematic issue (and i've also got a patch):

the remove-* ops use BackupSet.delete() in collections.py, which has direct access to the names of the remote files to
remove, but does some guesswork for cached files in the archive dir; it (re)parses the filenames, then decides
which ones to remove purely based on time.

since 0.6.06 the removal of old stuff is less stringent, and old signatures are supposed to stay around - but BackupSet.delete()
hasn't been changed: it does, indeed, remove files of type new-sig - whereas the rest of delete() doesn't remove new-sig stuff
from the remote archive anymore.

this causes the cache to desynchronize, the next backup operation requires a resync, and that fails miserably if one uses --encrypt-key and runs the backups non-interactively.

the solution is simple: delete() needs to ignore files in the cache that are of type new-sig.

regards
az