remove-* commands don't remove signature-files

[Gnaddelwarz]

Duplicity version: 0.6.06 (debian: 0.6.06-3)
Python version: 2.5.2 (debian: 2.5.2-15+lenny1
OS Distro and version: Debian Lenny + packages from Testing

When working with duplicity for some time, the archive-dir grows and grows. (One user on the mailinglist had a 19GiB archive-dir)
The problem seems to be, that duplicity does not cleanup signature-files
when removing old backups, so the signatures occupy more and more space.

To reproduce, I did some backups (locally), some with a forced full
backup and some incementals. Afterwards the backup looks like this:

$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 2 secondary backup chains.
Secondary chain 1 of 2:
-------------------------
Chain start time: Wed Feb 10 16:50:16 2010
Chain end time: Wed Feb 10 16:50:44 2010
Number of contained backup sets: 4
Total number of contained volumes: 4
 Type of backup set: Time: Num volumes:
                Full Wed Feb 10 16:50:16 2010 1
         Incremental Wed Feb 10 16:50:36 2010 1
         Incremental Wed Feb 10 16:50:41 2010 1
         Incremental Wed Feb 10 16:50:44 2010 1
-------------------------

Secondary chain 2 of 2:
-------------------------
Chain start time: Wed Feb 10 16:51:01 2010
Chain end time: Wed Feb 10 16:51:39 2010
Number of contained backup sets: 4
Total number of contained volumes: 4
 Type of backup set: Time: Num volumes:
                Full Wed Feb 10 16:51:01 2010 1
         Incremental Wed Feb 10 16:51:36 2010 1
         Incremental Wed Feb 10 16:51:38 2010 1
         Incremental Wed Feb 10 16:51:39 2010 1
-------------------------

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set: Time: Num volumes:
                Full Wed Feb 10 16:51:46 2010 1
         Incremental Wed Feb 10 16:52:04 2010 1
-------------------------
No orphaned or incomplete backup sets found.

$ ls /tmp/duptest/
duplicity-full.20100210T155016Z.manifest.gpg
duplicity-full.20100210T155016Z.vol1.difftar.gpg
duplicity-full.20100210T155101Z.manifest.gpg
duplicity-full.20100210T155101Z.vol1.difftar.gpg
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg

Now I remove the old backups:

$ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
file:///tmp/duptest
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Feb 10 16:51:46 2010
Deleting backup sets at times:
Wed Feb 10 16:50:16 2010
Wed Feb 10 16:50:36 2010
Wed Feb 10 16:50:41 2010
Wed Feb 10 16:50:44 2010
Wed Feb 10 16:51:01 2010
Wed Feb 10 16:51:36 2010
Wed Feb 10 16:51:38 2010
Wed Feb 10 16:51:39 2010
Warning, found the following local orphaned signature files:
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz

Note the "locally orphaned files".
When I look at the status:

$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set: Time: Num volumes:
                Full Wed Feb 10 16:51:46 2010 1
         Incremental Wed Feb 10 16:52:04 2010 1
-------------------------
No orphaned or incomplete backup sets found.

Note that duplicity did fetch the signatures of the 2 old (deleted!)
full-backups

When looking at the target, it becomes clear that "duplicity remove..."
did only remove the data (*vol*difftar-files), not the obsoleted signatures:

$ ls /tmp/duptest/
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg

These files are also present (in decrypted form) in the cache-dir. And
they stay there (or are refetched from the backup-target).
Only workaround: delete the unneeded signature-files on the target, then
duplicity also removes them from the cache (but be carefull to only
remove old and unneeded files):
$ cd /tmp/duptest/
:/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-new-signatures.20100210T1550*
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T15513*
:/tmp/duptest$ ls
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
:/tmp/duptest$ cd -
~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Synchronizing remote metadata to local cache...
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
(not authoritative at backend).
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set: Time: Num volumes:
                Full Wed Feb 10 16:51:46 2010 1
         Incremental Wed Feb 10 16:52:04 2010 1
-------------------------
No orphaned or incomplete backup sets found.

By doing this, I could shrink the archive-dir (from my real backup) from over 3 GiB back to 1 GiB

[Gnaddelwarz (t-riemenschneider)]

this is probably related to / the cause for
https://bugs.launchpad.net/duplicity/+bug/497243

[Olivier Berger (olivierberger)]

See the context of this report in discussion on duplicity-talk list : http://lists.gnu.org/archive/html/duplicity-talk/2010-02/msg00010.html

[Ed Blackman (ed-edgewood)]

I experience this defect as well.

I found the --extra-clean parameter from the discussion at http://lists.gnu.org/archive/html/duplicity-talk/2010-02/msg00012.html but that doesn't seem to do anything when supplied as a parameter to remove-all-but-n-full, and removes current signature files (or seems like it would, I didn't use --force) when supplied to cleanup.

[Kenneth Loafman (kenneth-loafman)]

"duplicity cleanup --extra-clean --force $target" should do the trick, for now.

[Joshua Jensen (joshua-joshuajensen)]

--extra-clean does nothing to solve this issue. We need a real fix

[Morten Lied Johansen (mortenjo)]

In what way is this issue even close to "Fixed"?

I see several recent reports about this problem has been closed as duplicates of this bug, and this one is closed with a status of "Fix Released", which is clearly not the case.

The only solution mentioned is a workaround (--extra-clean when using cleanup), which doesn't work.

[derp herp (junkmail-trash)]

Still seeing this is 0.6.17

[Steffen (steffen-weber)]

Me, too.

[Jens Braeuer (jens-braeuer)]

I still see this issue in 0.6.19. Problem persists even with "--force --extra-clean"

[Steffen (steffen-weber)]

Same for me! :-(

[Mechanical snail (replicator-snail)]

Just confirming that `duplicity remove-all-but-n-full` is still fetching all the obsolete .sigtar.gpg files from the backend, using 0.6.19 from the Ubuntu PPA. Same behavior on Fedora (I think that was 0.6.18).

[iceflatline (iceflatline)]

Confirming same problem. Commands in current test script used in order as follows:

duplicity --full-if-older-than 60m --encrypt-key="<phrase>" /home/foo/local/ scp://foo@target/remote/

duplicity remove-all-but-n-full 4 --force --encrypt-key="<phrase>" scp://foo@target/remote/

duplicity cleanup --extra-clean --force --encrypt-key="<phrase>" scp://foo@target/remote/

Using FreeBSD 9.0-Release; duplicity v0.6.19_2 compiled from ports.

[Anton Kornexl (anton-kornexl)]

duplicity -V
 duplicity 0.6.20

duplicity remove-older-than 1M --force ftp://ixxx@yyyy/usr

But many old signature from 2011 are still on the backup-Store

NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/).
Connecting to ...
FTP server ready.
Logging in...

Login successful.
Logged in to .
Current remote directory is /usr.
ncftp /usr > ll duplicity-full-signatures.201105
    20110506T051048Z.sigtar.gpg 20110521T052258Z.sigtar.gpg
    20110513T051832Z.sigtar.gpg 20110529T052206Z.sigtar.gpg
ncftp /usr > ll duplicity-full-signatures.20110*
-rw------- 1 85285 2000 34183227 Apr 28 2011 duplicity-full-signatures.20110428T065520Z.sigtar.gpg
-rw------- 1 85285 2000 35780951 May 6 2011 duplicity-full-signatures.20110506T051048Z.sigtar.gpg
-rw------- 1 85285 2000 36017596 May 13 2011 duplicity-full-signatures.20110513T051832Z.sigtar.gpg
-rw------- 1 85285 2000 36017612 May 21 2011 duplicity-full-signatures.20110521T052258Z.sigtar.gpg
-rw------- 1 85285 2000 36318629 May 29 2011 duplicity-full-signatures.20110529T052206Z.sigtar.gpg
-rw------- 1 85285 2000 36318696 Jun 6 2011 duplicity-full-signatures.20110606T051741Z.sigtar.gpg
-rw------- 1 85285 2000 36318442 Jun 13 2011 duplicity-full-signatures.20110613T051930Z.sigtar.gpg
-rw------- 1 85285 2000 36318437 Jun 21 2011 duplicity-full-signatures.20110621T051846Z.sigtar.gpg
-rw------- 1 85285 2000 36319150 Jun 28 2011 duplicity-full-signatures.20110628T052016Z.sigtar.gpg
-rw------- 1 85285 2000 36319231 Jul 6 2011 duplicity-full-signatures.20110706T052756Z.sigtar.gpg
.....

duplicity 0.6.20 remove does not remove old signatures

[prl77 (prl77)]

Confirming findings detailed in post #13 above.
Old sigtar files remain in both local cache and backend after running the same procedure Anton detailed.

[iceflatline (iceflatline)]

Confirming that 0.6.20 has fixed the problem for me. Example commands in current production script used in order as follows:

duplicity cleanup --extra-clean --force --encrypt-key=<phrase> file:///mnt/files/weekly/

duplicity remove-older-than 30D --extra-clean --force --encrypt-key=<phrase> file:///mnt/files/weekly/

duplicity --full-if-older-than 7D --include /mnt/files/backup/ --exclude '**' --encrypt-key=<phrase> /mnt/files/ file:///mnt/files/weekly/

FreeBSD 9.0-RELEASE and 9.1-RELEASE; duplicity-0.6.20 compiled from ports.

[Alexander Mette (mail-amette)]

Yep, confirming that 0.6.20 with the command

"duplicity remove-all-but-n-full x --force"

removes unneeded signatures as well. Yay! Thanks a bunch! :)