remove-* commands don't remove signature-files

Bug #519948 reported by Gnaddelwarz on 2010-02-10
132
This bug affects 27 people
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Unassigned
Debian
Fix Released
Unknown

Bug Description

Duplicity version: 0.6.06 (debian: 0.6.06-3)
Python version: 2.5.2 (debian: 2.5.2-15+lenny1
OS Distro and version: Debian Lenny + packages from Testing

When working with duplicity for some time, the archive-dir grows and grows. (One user on the mailinglist had a 19GiB archive-dir)
The problem seems to be, that duplicity does not cleanup signature-files
when removing old backups, so the signatures occupy more and more space.

To reproduce, I did some backups (locally), some with a forced full
backup and some incementals. Afterwards the backup looks like this:

$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 2 secondary backup chains.
Secondary chain 1 of 2:
-------------------------
Chain start time: Wed Feb 10 16:50:16 2010
Chain end time: Wed Feb 10 16:50:44 2010
Number of contained backup sets: 4
Total number of contained volumes: 4
 Type of backup set: Time: Num volumes:
                Full Wed Feb 10 16:50:16 2010 1
         Incremental Wed Feb 10 16:50:36 2010 1
         Incremental Wed Feb 10 16:50:41 2010 1
         Incremental Wed Feb 10 16:50:44 2010 1
-------------------------

Secondary chain 2 of 2:
-------------------------
Chain start time: Wed Feb 10 16:51:01 2010
Chain end time: Wed Feb 10 16:51:39 2010
Number of contained backup sets: 4
Total number of contained volumes: 4
 Type of backup set: Time: Num volumes:
                Full Wed Feb 10 16:51:01 2010 1
         Incremental Wed Feb 10 16:51:36 2010 1
         Incremental Wed Feb 10 16:51:38 2010 1
         Incremental Wed Feb 10 16:51:39 2010 1
-------------------------

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set: Time: Num volumes:
                Full Wed Feb 10 16:51:46 2010 1
         Incremental Wed Feb 10 16:52:04 2010 1
-------------------------
No orphaned or incomplete backup sets found.

$ ls /tmp/duptest/
duplicity-full.20100210T155016Z.manifest.gpg
duplicity-full.20100210T155016Z.vol1.difftar.gpg
duplicity-full.20100210T155101Z.manifest.gpg
duplicity-full.20100210T155101Z.vol1.difftar.gpg
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg

Now I remove the old backups:

$ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
file:///tmp/duptest
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Feb 10 16:51:46 2010
Deleting backup sets at times:
Wed Feb 10 16:50:16 2010
Wed Feb 10 16:50:36 2010
Wed Feb 10 16:50:41 2010
Wed Feb 10 16:50:44 2010
Wed Feb 10 16:51:01 2010
Wed Feb 10 16:51:36 2010
Wed Feb 10 16:51:38 2010
Wed Feb 10 16:51:39 2010
Warning, found the following local orphaned signature files:
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz

Note the "locally orphaned files".
When I look at the status:

$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set: Time: Num volumes:
                Full Wed Feb 10 16:51:46 2010 1
         Incremental Wed Feb 10 16:52:04 2010 1
-------------------------
No orphaned or incomplete backup sets found.

Note that duplicity did fetch the signatures of the 2 old (deleted!)
full-backups

When looking at the target, it becomes clear that "duplicity remove..."
did only remove the data (*vol*difftar-files), not the obsoleted signatures:

$ ls /tmp/duptest/
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg

These files are also present (in decrypted form) in the cache-dir. And
they stay there (or are refetched from the backup-target).
Only workaround: delete the unneeded signature-files on the target, then
duplicity also removes them from the cache (but be carefull to only
remove old and unneeded files):
$ cd /tmp/duptest/
:/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-new-signatures.20100210T1550*
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T15513*
:/tmp/duptest$ ls
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
:/tmp/duptest$ cd -
~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Synchronizing remote metadata to local cache...
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
(not authoritative at backend).
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set: Time: Num volumes:
                Full Wed Feb 10 16:51:46 2010 1
         Incremental Wed Feb 10 16:52:04 2010 1
-------------------------
No orphaned or incomplete backup sets found.

By doing this, I could shrink the archive-dir (from my real backup) from over 3 GiB back to 1 GiB

Gnaddelwarz (t-riemenschneider) wrote :

this is probably related to / the cause for
https://bugs.launchpad.net/duplicity/+bug/497243

Olivier Berger (olivierberger) wrote :

See the context of this report in discussion on duplicity-talk list : http://lists.gnu.org/archive/html/duplicity-talk/2010-02/msg00010.html

Ed Blackman (ed-edgewood) wrote :

I experience this defect as well.

I found the --extra-clean parameter from the discussion at http://lists.gnu.org/archive/html/duplicity-talk/2010-02/msg00012.html but that doesn't seem to do anything when supplied as a parameter to remove-all-but-n-full, and removes current signature files (or seems like it would, I didn't use --force) when supplied to cleanup.

"duplicity cleanup --extra-clean --force $target" should do the trick, for now.

Changed in debian:
status: Unknown → Fix Released
Changed in duplicity:
status: New → Fix Released

--extra-clean does nothing to solve this issue. We need a real fix

Morten Lied Johansen (mortenjo) wrote :

In what way is this issue even close to "Fixed"?

I see several recent reports about this problem has been closed as duplicates of this bug, and this one is closed with a status of "Fix Released", which is clearly not the case.

The only solution mentioned is a workaround (--extra-clean when using cleanup), which doesn't work.

derp herp (junkmail-trash) wrote :

Still seeing this is 0.6.17

Steffen (steffen-weber) wrote :

Me, too.

Jens Braeuer (jens-braeuer) wrote :

I still see this issue in 0.6.19. Problem persists even with "--force --extra-clean"

Steffen (steffen-weber) wrote :

Same for me! :-(

Changed in duplicity:
status: Fix Released → Confirmed
importance: Undecided → Medium

Just confirming that `duplicity remove-all-but-n-full` is still fetching all the obsolete .sigtar.gpg files from the backend, using 0.6.19 from the Ubuntu PPA. Same behavior on Fedora (I think that was 0.6.18).

iceflatline (iceflatline) wrote :

Confirming same problem. Commands in current test script used in order as follows:

duplicity --full-if-older-than 60m --encrypt-key="<phrase>" /home/foo/local/ scp://foo@target/remote/

duplicity remove-all-but-n-full 4 --force --encrypt-key="<phrase>" scp://foo@target/remote/

duplicity cleanup --extra-clean --force --encrypt-key="<phrase>" scp://foo@target/remote/

Using FreeBSD 9.0-Release; duplicity v0.6.19_2 compiled from ports.

Changed in duplicity:
milestone: none → 0.6.20
status: Confirmed → Fix Committed
Changed in duplicity:
status: Fix Committed → Fix Released
Anton Kornexl (anton-kornexl) wrote :

duplicity -V
 duplicity 0.6.20

duplicity remove-older-than 1M --force ftp://ixxx@yyyy/usr

But many old signature from 2011 are still on the backup-Store

NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/).
Connecting to ...
FTP server ready.
Logging in...

Login successful.
Logged in to .
Current remote directory is /usr.
ncftp /usr > ll duplicity-full-signatures.201105
    20110506T051048Z.sigtar.gpg 20110521T052258Z.sigtar.gpg
    20110513T051832Z.sigtar.gpg 20110529T052206Z.sigtar.gpg
ncftp /usr > ll duplicity-full-signatures.20110*
-rw------- 1 85285 2000 34183227 Apr 28 2011 duplicity-full-signatures.20110428T065520Z.sigtar.gpg
-rw------- 1 85285 2000 35780951 May 6 2011 duplicity-full-signatures.20110506T051048Z.sigtar.gpg
-rw------- 1 85285 2000 36017596 May 13 2011 duplicity-full-signatures.20110513T051832Z.sigtar.gpg
-rw------- 1 85285 2000 36017612 May 21 2011 duplicity-full-signatures.20110521T052258Z.sigtar.gpg
-rw------- 1 85285 2000 36318629 May 29 2011 duplicity-full-signatures.20110529T052206Z.sigtar.gpg
-rw------- 1 85285 2000 36318696 Jun 6 2011 duplicity-full-signatures.20110606T051741Z.sigtar.gpg
-rw------- 1 85285 2000 36318442 Jun 13 2011 duplicity-full-signatures.20110613T051930Z.sigtar.gpg
-rw------- 1 85285 2000 36318437 Jun 21 2011 duplicity-full-signatures.20110621T051846Z.sigtar.gpg
-rw------- 1 85285 2000 36319150 Jun 28 2011 duplicity-full-signatures.20110628T052016Z.sigtar.gpg
-rw------- 1 85285 2000 36319231 Jul 6 2011 duplicity-full-signatures.20110706T052756Z.sigtar.gpg
.....

duplicity 0.6.20 remove does not remove old signatures

prl77 (peterlecki) wrote :

Confirming findings detailed in post #13 above.
Old sigtar files remain in both local cache and backend after running the same procedure Anton detailed.

iceflatline (iceflatline) wrote :

Confirming that 0.6.20 has fixed the problem for me. Example commands in current production script used in order as follows:

duplicity cleanup --extra-clean --force --encrypt-key=<phrase> file:///mnt/files/weekly/

duplicity remove-older-than 30D --extra-clean --force --encrypt-key=<phrase> file:///mnt/files/weekly/

duplicity --full-if-older-than 7D --include /mnt/files/backup/ --exclude '**' --encrypt-key=<phrase> /mnt/files/ file:///mnt/files/weekly/

FreeBSD 9.0-RELEASE and 9.1-RELEASE; duplicity-0.6.20 compiled from ports.

Alexander Mette (mail-amette) wrote :

Yep, confirming that 0.6.20 with the command

"duplicity remove-all-but-n-full x --force"

removes unneeded signatures as well. Yay! Thanks a bunch! :)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.