Allow bandwidth throttling

[Michael Terry]

If doing a network backup, we may want to allow the user to throttle the bandwidth. Maybe using trickle?

[Exquisite Dead Guy (ben-forlent)]

Affects me as well. I have a slower DSL connection and backing up the S3 makes everything else unusable (web pages take like 30 seconds+ to load). I'd love a feature somewhere where I can limit it to say 20Kb/sec.

Related to this, a pause button would be very helpful too. If a backup is running and I need full connection speed for an hour or two, I'd love to be able to click pause and have it resume right where it left off. I can click resume later, but it seems to start over from the beginning, so if I was at 5% it would start over from 0%. I've been trying to make my initial backup for two months now and have never gotten past 5% due to these problems.

[Saiyine (saiyine)]

DSL users like us need this badly.

I'm a new DejaDup/Duplicity user, EDG, do you mean uploads can't be paused? If that's true I can stop trying to use this software right now, as my DSL upload speed would take my backup months to be completely FTPed :(

[Exquisite Dead Guy (ben-forlent)]

Saiyine: as far as I can tell, there's no way to pause. There's "Resume Later" but if you click this, when the backup resumes in a few hours, it starts over from the beginning.

[Michael Terry (mterry)]

No pause button no. Note that if you "Resume Later", it shouldn't be actually starting over. It may look like it, because it rescans files and starts progress at 0%, but it will be building on previous progress and should move to 100% faster because there is less work to do.

The problem here is that there doesn't seem to be very good ways of throttling bandwidth. Like a network nice or something. Trickle is the closest I've seen, but it's not a very common tool, not sure how good it is.

[Rahul Sundaram (metherid)]

I think you could use cgroups to throttle bandwidth

[scw (shaun-walbridge)]

In my usage of it, Trickle's worked well, and I can imagine the calls out to the duplicity command could just be wrapped in it. During the upload steps, my latency jumps to three seconds, so I have to manually choose times for backing up instead of the regularly scheduled intervals.

[Matthew Leggett (leggett-matthew)]

I think upload throttling would be great addition. I too have a rather unsteady DSL connection, some days faster than others and on the slow days (when the Mrs. is using the internet also) I get complaints of my backups hogging what bandwidth is available.

Is this something that could be implemented?

Would be fantastic if you could tie in the deja dup upload speed with the U1 client so it adheres to those throttling settings and save on duplication of settings etc.

[Gustavo L (gustavo-lapido)]

"Allow bandwidth throttling" should be a core requirement for any application that aims to send/receive client data through the Internet.
Not only because of those who have slow DSL connections, but only of those who have relatively slow Upload speeds, something that applies to many broadband users.
Running the first backup upload can be hard in regards to the overall connection experience, since Deja Dup seems to automatically max out upload speed.

[Thibault Nélis (thibn)]

Another thing to consider is that when saturating the outbound bandwidth, TCP ACKs of other connections will be delayed, which is one of the reasons why maxing out upload will negatively affect download elsewhere as well. This also applies to any kind of outbound interaction that typically expects relatively low latency. Throttling even a little (to leave room for these low-bandwidth communications) improves overall latency a great deal.

Some users heavily affected by bufferbloat may also wish to throttle more aggressively to better map the real end-to-end link bandwidth (as opposed to filling up buffers midway then blocking entirely for several seconds until they clear up, creating visible high-lows in transfer rates).

I don't think Déjà Dup should be directly responsible for throttling per se, though duplicity probably shouldn't either. I believe the suggestion of composing duplicity with another *nix-y utility is a good one. Be it trickle or another program (possibly a new tool if Déjà Dup has special needs, though I would imagine not).

[Vicente Bolea (vbolea)]

A doable way to solve this with Trickle it is to create desktop file that overrides the default desktop file for DejaDup Monitor:

$ cp /etc/xdg/autostart/org.gnome.DejaDup.Monitor.desktop ~/.config/autostart

and edit the lines:

+Exec=sh -c "ulimit -v 1000000; exec /usr/lib/deja-dup/deja-dup-monitor"
-Exec=sh -c "ulimit -v 1000000; exec /usr/bin/trickle -s -d 50 -u 100 /usr/lib/deja-dup/deja-dup-monitor"

I hope you can find this useful.

[Vicente Bolea (vbolea)]

I must add that the previous comment is not a working example, since deja-dup-monitor does not actually send/recv data.

I found that an easier way is to create the following file at the following path:

On /usr/local/bin/duplicity:

/usr/bin/trickle -s -u 50 -d 50 /usr/bin/duplicity $@

Then:

chmod +x /usr/local/bin/duplicity