Slow transfer over sftp connection

[Anton]

I noticed that backups were taking much longer than I expected. Using scp to the same server, I get about 20MB/s (which is about my max HD throughput). However, sbackup writes at about only 2MB/s, a factor of 10 (!) slower (as reported by iotop and network traffic and manual measurement of file growth). It isn't the compression, because that only runs at only 20% CPU. So there seems room for a 5-fold improvement here.

This is a serious issue, because a full backup now can take up to 5 hours or so (in stead of about one hour), which significantly increases the odds that the backup does not finish before I shutdown my laptop, or take it offline.

I have tracked the problem to what I think may be the origin; a bug in glib/gvfs:
https://bugs.launchpad.net/ubuntu/+source/glib2.0/+bug/250517
In short, all writes on a gvfs volume are acknowledged, causing a network-latency dependent bottleneck, which may be quite severe even on networks with a high (intrinsic) bandwidth. In my case, it's 1G from my laptop all the way to the server, but with several routers in between (university setup).

The proper solution would be to have gvfs fixed, however according to the bug that isn't going to be any time soon.

An alternative may be to see if a reasonable workaround fix can be implemented for sbackup. Any ideas?

(I tried to use the 'fuse' plugin in stead of 'gvfs' for sbackup to see if the issue exists with fuse as well, but fuse will not connect without password, and does not accept my password since it has a '?' in it. Will get back on this if I found time to try with a different password.)

[Jean-Peer Lorenz (peer.loz)]

Still valid? How is the status?

[Anton (feenstra)]

I'm running Natty at the moment (gvfs 1.9.0-0ubuntu1, glib 2.28.6-0ubuntu1), and it does seem to run much faster now, although I haven't done any proper timings now. However, the glib bug I refer to above is still open, so I don't straight away see how things may have approved (maybe the latency in our university network went down?).

[Launchpad Janitor (janitor)]

[Expired for sbackup because there has been no activity for 60 days.]

[Anton (feenstra)]

Hi Jean-Peer,
Unfortunately, on Trusty, this problem is still there. The other day a full backup took almost the whole working day (this is a university laptop, and I've tweaked scripts so its starts a backup as soon as it comes online when I arrive at work).
As a workaround, I'd be happy to patch in a backend that uses an ssh pipe (something like tar | ssh user@remote dd of=/where/my/backups/are.tgz). But I don't understand the structure of the file backend that sbackup uses. Is this worth looking in to (I'd certainly like to have this feature - it would also allow me to do full backups from home over fiber optics - high bandwith but much longer latencies).
Please give me some pointers to where I could put this into sbackup. I'll try to do it nicely enough - preferably as a separate backend - so that you may be inclined to include it in trunk.