cleanup broken

[esskov]

The cleanup action no longer works correctly with duplicity 0.6.19 and an scp:// target. It has worked fine with some earlier 0.6.1? version, so this seems to be a regression.

Duplicity correctly detects the files to be deleted, but actually only deletes the first of them - even though no error is reported.
Please note that this type of backend behaviour could also explain Bug #1031269.

================================================================================
duplicity 0.6.19 ($reldate)
Args: /usr/bin/duplicity cleanup --verbosity 9 --ssh-askpass --force scp://xxx@xxx/backup/xxx
Linux xxx 3.0.0-23-generic-pae #39-Ubuntu SMP Thu Jul 19 19:39:19 UTC 2012 i686 i686
/usr/bin/python2.7 2.7.2+ (default, Oct 4 2011, 20:03:08)
[GCC 4.6.1]
================================================================================
Local and Remote metadata are synchronized, no sync needed.
9998 files exist on backend
4 files exist in cache
Extracting backup chains from list of files: ['duplicity-full.20110630T230528Z.vol847.difftar.gpg', 'duplicity-full.20110630T230528Z.vol4585.difftar.gpg', ...]
...
File duplicity-full.20120730T192908Z.vol3386.difftar.gpg is part of known set
File duplicity-full.20120730T192908Z.vol4692.difftar.gpg is part of known set
Found backup chain [Mon Jul 30 21:29:08 2012]-[Mon Jul 30 21:29:08 2012]
Warning, found incomplete backup sets, probably left from aborted session
Last full backup date: Mon Jul 30 21:29:08 2012
Collection Status
-----------------
Connecting with backend: SSHParamikoBackend
Archive dir: /home/xxx/.cache/duplicity/xxx

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Mon Jul 30 21:29:08 2012
Chain end time: Mon Jul 30 21:29:08 2012
Number of contained backup sets: 1
Total number of contained volumes: 4964
 Type of backup set: Time: Num volumes:
                Full Mon Jul 30 21:29:08 2012 4964
-------------------------
Also found 0 backup sets not part of any chain,
and 2 incomplete backup sets.
These may be deleted by running duplicity with the "cleanup" command.
Deleting these files from backend:
duplicity-full.20110630T230528Z.vol2.difftar.gpg
duplicity-full.20110630T230528Z.vol3.difftar.gpg
duplicity-full.20110630T230528Z.vol4.difftar.gpg
...

[Matteo Brancaleoni (mbrancaleoni)]

I think the issue is in _ssh_paramiko.py, into delete function.

the backend deletes the first file of the list passed to delete() but then a return is called after the real delete... this skips all the remaining files in list.

I think that removing the return statements in delete fn (should be line 363 and 367) will fix that.

[Kevin M. (kevin-meyer)]

Hi,

same or similar error here.

command:

duplicity remove-all-but-n-full 2 --force sftp://${user}:${pw}@${target}

and i get

Deleting complete signature chain Wed Jan 16 11:30:41 2013
Deleting complete signature chain Wed Jan 16 11:30:41 2013 !!!!!!!!!!!!!!!!!!!! WHY, You damn *** :-/
sftp rm duplicity-full-signatures.20130116T103041Z.sigtar.gpg failed: [Errno 2] No such file (Try 2 of 5) Will retry in 10 seconds.
sftp rm duplicity-full-signatures.20130116T103041Z.sigtar.gpg failed: [Errno 2] No such file (Try 3 of 5) Will retry in 10 seconds.
sftp rm duplicity-full-signatures.20130116T103041Z.sigtar.gpg failed: [Errno 2] No such file (Try 4 of 5) Will retry in 10 seconds.
sftp rm duplicity-full-signatures.20130116T103041Z.sigtar.gpg failed: [Errno 2] No such file

duplicity-full-signatures.20130116T103041Z.sigtar.gpg deleted. So you can not find annother one, you ...

But i can not find any "return" in delete() function, which i could remove.

--------------------------------------------------------------------------------------------------------------------
def delete(self, filename_list):
        """deletes all files in the list on the remote side. In scp mode unavoidable quoting issues
        will cause failures if filenames containing single quotes are encountered."""
        for n in range(1, globals.num_retries+1):
            if n > 1:
                # sleep before retry
                time.sleep(self.retry_delay)
            try:
                for fn in filename_list:
                    if (globals.use_scp):
                        self.runremote("rm '%s/%s'" % (self.remote_dir,fn),False,"scp rm ")
                    else:
                        try:
                            self.sftp.remove(fn)
                        except Exception, e:
                            raise BackendException("sftp rm %s failed: %s" % (fn,e))
            except Exception, e:
                if n == globals.num_retries:
                    log.FatalError(str(e), log.ErrorCode.backend_error)
                else:
                    log.Warn("%s (Try %d of %d) Will retry in %d seconds." % (e,n,globals.num_retries,self.retry_delay))
------------------------------------------------------------------------------------------------------------------------

I am using duplicity version 0.6.20, python 2.7.3, gpg 1.4.11

[Dehalion (mail-launchpad)]

I also have this [Errno 2] No such file (Try 2 of 5) when running remove-all-but-n-full.
I used duplicity version 0.6.21.

The only workaround I found is downgrading to 0.6.18 where it works just fine...

[Die Antwort (b-reg)]

also affects me, like the previous comment pointed out:
(it is a
remove-all-but-n-full 2
call)

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Mar 26 00:10:03 2013
Deleting backup chains at times:
Wed Feb 6 16:01:04 2013
Sat Mar 9 00:10:01 2013
Deleting complete signature chain Sat Mar 9 00:10:01 2013
Deleting complete signature chain Sat Mar 9 00:10:01 2013
sftp rm duplicity-new-signatures.20130307T231003Z.to.20130308T231001Z.sigtar.gpg failed: [Errno 2] No such file (Try 2 of 5) Will retry in 10 seconds.
etc...

[Andreas Nüßlein (nutz)]

I actually found the problem and fixed it:

https://bugs.launchpad.net/duplicity/+bug/1115715

[Die Antwort (b-reg)]

Confirming that the temporary fix of
https://bugs.launchpad.net/duplicity/+bug/1115715
(setting --num-retries 1) solves the problem.

[Horst Schirmeier (horst)]

Confirming that the patch from Andreas Nüßlein in https://bugs.launchpad.net/duplicity/+bug/1115715 solves the "sftp rm [...].sigtar.gpg failed: [Errno 2] No such file (Try 2 of 5) Will retry in 10 seconds." problem.