Try overwriting file with an 0-byte file on delete failure (workaround for Hetzner's backup backend)

[Daniel Hahler]

I've run into an issue with the backup backend provided by Hetzner (a German provider):
when the backup space is full, you cannot remove any files, but have to overwrite the file with a 0-byte file of the same name. Only then you can delete the original file.

In duplicity the failure looks like this:
Running 'sftp -oServerAliveInterval=15 -oServerAliveCountMax=2 <email address hidden>' (attempt #1)
State = sftp, Before = 'Connected to u123.your-backup.de.'
sftp command: 'cd "moby/"'
State = sftp, Before = 'cd "moby/"'
sftp command: 'rm "duplicity-full.20110105T145659Z.vol1.difftar.gpg"'
State = sftp, Before = 'rm "duplicity-full.20110105T145659Z.vol1.difftar.gpg"
Removing /moby/duplicity-full.20110105T145659Z.vol1.difftar.gpg'
Could not delete file in command='sftp -oServerAliveInterval=15 -oServerAliveCountMax=2 <email address hidden>'
Running 'sftp -oServerAliveInterval=15 -oServerAliveCountMax=2 <email address hidden>' failed (attempt #1)

Replaying this via "sftp" looks like this:
sftp> rm "duplicity-full.20110105T145659Z.vol1.difftar.gpg"
Removing /moby/duplicity-full.20110105T145659Z.vol1.difftar.gpg
Couldn't delete file: Failure

There are two issues here:
1. The error "Couldn't delete file: Failure" should get logged. At least with verbosity=9, but also with something like 4 (default?!).
2. At least in case of this exact error ("Couldn't delete file: Failure") it should try the following procedure:
upload a 0-byte file, overwriting the file that should get deleted. If this works, try the "rm" again.

[Kenneth Loafman (kenneth-loafman)]

Have you tried this yet? Did it work?

[Daniel Hahler (blueyed)]

Yes, it did work.
I have this patch laying around in my source tree at the server, but ran into some merge conflicts and gave up the last time I've tried to extract it.

I plan to get back to this during the next year, and will provide a branch to merge here.

[Röbke Geenen (rgeenen-lp-oops)]

On a related note, the Stack platform of the Dutch TransIP also takes the size of deleted files into account when determining space usage, unless the files are permanently deleted via the web interface. A workaround is to overwrite the files with zero-byte files before deleting them.

It would be great to have an option in duplicity to overwrite all files to be deleted by zero-byte files first.

Should I make a separate feature request, or is it related closely enough to this bug?