Bandwidth limit is not correctly enforced: Transmission delays are inserted between data chunk writes (of arbitrary sizes)

[Zygmunt Krynicki]

GENERAL DESCRIPTION
*******************

Throttling implementation today is not working as users might expect, often saturating the link.

STEPS FOR REPRODUCING
*********************

1. Ensure the Ubuntu One client is installed and running

2. In Ubuntu One client, limit its bandwidth to a small number (for example 20 KB)

3. Copy a big file to a folder configured to be synced by Ubuntu One

4. Open the System Monitor

4. Wait one minute

5. In the Terminal, run the following command: ping google.com -c20

EXPECTED BEHAVIOUR
******************

The monitored latency and throughput to be stable.

REAL BEHAVIOUR
**************

The monitored latency and the throughput to be unstable.

RELEVANT DETAILS
****************

- A screen-cast shows this issue with more detail: http://people.canonical.com/~zyga/Maverick%20Movie004-1.m4v.

- The actual limit during this video was around 10-20KB.

WORKAROUND
**********

Saturation can partially be mitigated by doing the following:

1. By contracting an Internet Service which follows the International Computer Science Institute quality standards, testable at http://netalyzr.icsi.berkeley.edu. Many times bandwidth saturation is caused by an improper upload buffering from the Internet Service Provider, which will affect not only the Ubuntu One Client but any service that uploads to the Internet.

2. If you are pretending to use a wireless network, only use routers which are IEEE 802.11n compliant. This standard uses more advance algorithms than its predecessors for assigning bandwidth to links.

3. If you are using a home router, disable any feature which doesn't belong to any standard but are specific from the router's manufacturer. These generally are the firewall and the Quality of Service features, except Wireless Multimedia Extensions (WME or WMM). These specific functions trend to dramatically increase network latency when the link gets saturated.

4. If any application being used give an option to limit bandwidth, limit it to the 80% of the contracted one. This will ensure constant throughput for that program and faster initialisation for other services while bandwidth is being negotiated between them.

[Facundo Batista (facundo)]

this is an effect of the following (examples for "write", the same applies for "read")... in the client.py, ThrottlingStorageClient has...

    def write(self, data):
        """Transport API to capture bytes written."""
        if self.factory.client is self:
            self.factory.registerWritten(len(data))
        StorageClient.write(self, data)

...that writes whatever data it got, and just tells the throttler the amount of data written. So, if the limit is X bytes per second, and twisted wants to write 10*X, it just writes that (probably saturating the tcp link), and then the throttler waits 9 seconds more to be below the limit.

The way to fix this is not write the whole amount at once, but mix that with the throttling scheme.

[tolbier (ltoyos)]

I'm using Ubuntu 11.04 , and I suffer from this issue. I will attach my apport-collect, if it could be interesting for a rapid solution for this bug

[Stuart Bishop (stub)]

Please consider bumping the priority of this. I can currently only enable file synchronization when nothing else needs the Internet in the house due to a large pending upload - with my ISP, the link gets saturated and new TCP/IP connections fail for even simple things like DNS lookups and web browsing.

[John Lenton (chipaca)]

On Wed, 04 May 2011 07:33:23 -0000, Stuart Bishop <email address hidden> wrote:
> Please consider bumping the priority of this. I can currently only
> enable file synchronization when nothing else needs the Internet in the
> house due to a large pending upload - with my ISP, the link gets
> saturated and new TCP/IP connections fail for even simple things like
> DNS lookups and web browsing.

while not wanting to detract from the urgency and priority of the bug,
uplink killing the adsl link is a known problem of that kind of link,
and you probably should be using wondershaper (and this would mitigate
the issue for you; consider it a temporary workaround if you would).

[Zygmunt Krynicki (zyga)]

I can only stress the same as comment #3. Ubuntu one is unusable without resolving this issue. It should be the primary feature "synchronizes files without you noticing" and I'm sure it lowers the perceived quality of our services by a great amount.

[Stuart Bishop (stub)]

As a work around, I'm finding wondershaper is working wonderfully in my situation.

[Alberto Salvia Novella (es20490446e)]

Can you test if you continue having this issue by setting the bandwidth limits to the 30% of the maximum?

[AJenbo (ajenbo)]

The really ironic part is that the U1 client will some times fail to work because it can't contact the server, because of the upload, in this case i don't think you would even be able to set a bandwith limit x(

[Alberto Salvia Novella (es20490446e)]

I thought it wasn't but the bug is still there for me too.

[Rick McBride (rmcbride)]

I've seen this as well, uplink saturation brings other network operations on a 20/2 cable connection to an unusable state. I was able to enable default QoS rules on my router, which kept the traffic class U1 utilizes from using more than 80% of the uplink. This smoothed things out considerably.

[chrisfaron (chrisfaron)]

I have the same problem with 11.04 using all my bandwidth, can someone suggest a workaround for now (similar to the router Qos solution)?

[Aang (aang-aero)]

@chrisfaron - try this: http://aangaero.wordpress.com/2011/03/13/ubuntu-one-maximum-upload-speed/

[Alberto Salvia Novella (es20490446e)]

I have just substituted my old router by a new one which is Wireless Multimedia Extensions compliant, and then Ubuntu One stopped saturating the link. Do you experience link saturation with a Wireless Multimedia Extensions compliant router?

[Danielsinisi (danielsinisi)]

Helo Alberto
    Suddenly Ubuntu One started to work fine, without hardware changes (same
router , same computer).
    I think it was an automatic update.

Daniel Sinisi

----- Original Message -----
From: "Alberto Salvia Novella" <email address hidden>
To: <email address hidden>
Sent: Wednesday, October 05, 2011 12:59 PM
Subject: [Bug 720707] Re: Bandwidth limit is not correctly
enforced:Transmission delays are inserted between data chunk writes
(ofarbitrary sizes)

I have just substituted my old router by a new one which is Wireless
Multimedia Extensions compliant, and then Ubuntu One stopped saturating
the link. Do you experience link saturation with a Wireless Multimedia
Extensions compliant router?

--
You received this bug notification because you are subscribed to a
duplicate bug report (769899).
https://bugs.launchpad.net/bugs/720707

Title:
  Bandwidth limit is not correctly enforced: Transmission delays are
  inserted between data chunk writes (of arbitrary sizes)

Status in Ubuntu One storage protocol:
  Confirmed

Bug description:
  Throttling implementation today is not working as users might expect,
  often saturating the link.

  A screencast shows this issue with more detail:
  http://people.canonical.com/~zyga/Maverick%20Movie004-1.m4v

  The actual limit during that video was around 10-20KB

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntuone-storage-protocol/+bug/720707/+subscriptions

[Alberto Salvia Novella (es20490446e)]

Okay then

[Dedeco (dedeco)]

Another (but easier) possible workaround:

Just set the upload limit to 1KiB/s!!

The bandwidth limit isn't absent ("...is not working as users might expect..."), so I thought... in what scenario will it work?

My maximum upload was around 70KiB/s, and setting upload limit to 30, 20, and 10 was useless (just like for other people, internet for other programs become unusable). Then I tried the lowest possible value. And now it doesn't wreck internet. :) I don't need to keep connecting and disconnecting Ubuntu One when I need or don't need internet.

[Dedeco (dedeco)]

However, what I have said above does not always work. Right now I cannot do it again. I can't use Ubuntu One because it will make internet for a whole room of computers useless. :|

[Alberto Salvia Novella (es20490446e)]

Sometimes I ask myself if this issue is caused only because of Ubuntu One itself, because when I start downloading a single file using Firefox the hole network ping increases by a lot. This doesn't happen when using the same browser in MacOS or Windows, even by switching off the QoS service.

It is possible than my ISP and Ubuntu QoS technologies are unable to understand each other. The ISP temporarily leases bandwidth for a sort period of time when the network becames unresponsible; and then seems that Ubuntu chooses to use more bandwidth than what is natural to the network, making permanent congestion.

[Alberto Salvia Novella (es20490446e)]

After running the ICSI Netalyzr (http://netalyzr.icsi.berkeley.edu/) I get the following diagnostic:

"We estimate your uplink as having 1700 msec of buffering. This is quite high, and you may experience substantial disruption to your network performance when performing interactive tasks such as web-surfing while simultaneously conducting large uploads. With such a buffer, real-time applications such as games or audio chat can work quite poorly when conducting large uploads at the same time. "

(http://n2.netalyzr.icsi.berkeley.edu/summary/id=43ca208a-7536-efed7842-f872-4384-b6bf)

I have experienced this problem by using different computers, different network equipment and different distros. It also happens by attaching the computer directly to two different modems provided by the ISP.

It seems there's some type of missconfiguration by my ISP, and I will run some more test to make this clear. Please, run the ICSI Netalyzr (http://netalyzr.icsi.berkeley.edu/) and tell if you're experiencing problems in your uplink buffering.

[Stuart Bishop (stub)]

Back to comment #1, wouldn't this implementation solve the bug?

def write(self, data):
    """Transport API to capture bytes written."""
    chunk_size = 1024*1024
    data_len = len(data)
    index = 0
    while index < data_len:
        if self.factory.client is self:
            self.factory.registerWritten(len(data[index:index+chunk_size]))
        StorageClient.write(self, data[index:index + chunk_size])
        index += chunk_size

[Zygmunt Krynicki (zyga)]

With chunk size of 1M it's not going to make any difference. On my link it takes around 2 minutes to upload one megabyte. While that happens nothing works. I'd like to set upload cap to 5KB and still be able to use the (slow, capped) network.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 7/2/2012 1:11 PM, Zygmunt Krynicki wrote:
> With chunk size of 1M it's not going to make any difference. On my
> link it takes around 2 minutes to upload one megabyte. While that
> happens nothing works. I'd like to set upload cap to 5KB and still
> be able to use the (slow, capped) network.
>

It seems like it could be made into:
 chunk_size = min(1024*1024, cap_size / 2)

Or something along those lines.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/xlBUACgkQJdeBCYSNAAOfJwCZAaGkqmQ2eC8bVQR0Ua0LNxlC
RvwAoMRNFw4pN7+Jh3W248hkvjxm3j1j
=6zS+
-----END PGP SIGNATURE-----

[Zygmunt Krynicki (zyga)]

Could we please bump the priority of this bug. It prevents me from running ubuntu one on this machine/network.

[Zygmunt Krynicki (zyga)]

Hi, could we please bump this bug, it's really preventing me from using u1 as a service and looking at some of the comments above I'm not unique in this aspect. If u1 cannot reliably sync files without making me micro-manage network all the time what is it useful for?

[Gerhard Aigner (gerhard-aigner)]

I'm thinking about canceling my ubuntu one storage plan because of this bug. The service is useless if you can't do anything else while uploading.

[Alberto Salvia Novella (es20490446e)]

Saturation can partially be mitigated by doing the following, deeply tested by myself:

1. By contracting an Internet Service which follows the International Computer Science Institute quality standards, testable at http://netalyzr.icsi.berkeley.edu. Many times bandwidth saturation is caused by an improper upload buffering from the Internet Service Provider, which will affect not only the Ubuntu One Client but any service that uploads to the Internet.

2. If you are pretending to use a wireless network, only use routers which are IEEE 802.11n compliant. This standard uses more advance algorithms than its predecessors for assigning bandwidth to links.

3. If you are using a home router, disable any feature which doesn't belong to any standard but are specific from the router's manufacturer. These generally are the firewall and the Quality of Service features, except Wireless Multimedia Extensions (WME or WMM). These specific functions trend to dramatically increase network latency when the link gets saturated.

4. If any application being used give an option to limit bandwidth, limit it to the 80% of the contracted one. This will ensure constant throughput for that program and faster initialisation for other services while bandwidth is being negotiated between them.

[Zygmunt Krynicki (zyga)]

Hello Alberto

While I commend you for listing workarounds I doubt any of the non highly-technical users can ever make use of this. This bug is real, it breaks all of the experience, it should be ESCALATED and FIXED.

The [bandwidth limiting] feature, as implemented, DOES NOT WORK. This bug is not about fixing the internet, buffering, ADSL saturation behavior .This bug is about the broken implementation of throttling, in ubuntuone-syncdaemon, that makes no sense at all and should have never been released in this form.

[Zygmunt Krynicki (zyga)]

For completeness, the feature is implemented that the average upload speed is indeed limited.

The implementation is broken because the period over which it is averaged arbitrary and in practice it's equal to sending everything in one go and then waiting, once everything is sent "long enough" to make the "average" correct.

This obviously makes no sense in practice.

[Alberto Salvia Novella (es20490446e)]

What I meant is this bug is capable of triggering other network deficiencies and make it work much worse.

I understand this bug should be fixed itself and I knew exactly what was its behaviour, but in the end if the link gets saturated is in first stage because network misconfiguration. No network properly configured will notice Ubuntu One is running, but not taking only one of the steps listed in the workaround can increase network latency from 250ms to 1000ms. Try yourself: the last two points are easy to test, and I did it with very different equipment.

[mspanc (mspanc)]

Alberto, even if what you write is technically true, it is totally irrelevant to average user's use case. Most users in the world are using, and will be using shitty cheap routers, misconfigured software and hardware and so on.

Most software that have bandwith limiting feature implement it in correct way, so most users expect that it will work correctly. They don't care if this triggers any other problems, they want fixed this feature.

To sum up: calling this a workaround in task description is a bad joke. It's technically true, but it is so far from ubuntu attempt to be linux for humans, not geeks that treating this even as temporary solution is embarassing.

[Greg Grossmeier (greg.grossmeier)]

All: Launchpad Bug Reports are not a place for discussion. If you want to discuss certain features, bugs, or workarounds without actively working on the problem there are other, better, venues for that.

This venue is for diagnosing and addressing the problem at hand, not general discussion area.

It is pretty clear what the issue is here and how to attack it. A developer on the Ubuntu One team will need to address it, or someone from the community will need to submit a patch for review. Stuart Bishop did just that and has proposed a merge here: https://code.launchpad.net/~stub/ubuntuone-storage-protocol/devel/+merge/143261

Please leave the discussion off this bug in the future.

Thanks.

[Alberto Salvia Novella (es20490446e)]

If any of you feel have any important point of view to share about this topic, feel free to email me.

[unelement (unelement)]

Can't believe this hasn't been addressed after years.

If limits don't work as they should and this is not going to be solved, just remove the option altogether so users don't waste their time checking a feature that doesn't work and clogs their networks.

A file sync system without bandwidth limits is a joke but at least is honest, this is just incredible. And the workarounds... C'mon guys!

[Alberto Salvia Novella (es20490446e)]

When I wrote these workarounds I knew they were a total crap, but I did because I had wanted to find them myself the first time I read this bug report. They describe better the origin of the bug and in the end they work just fine.

But yes, this is a holi-bullsit™ software (and report).

And not to mention some files just disappear when syncing between devices... better you don't format the computer where the file has originally been...

Look into the toilet, into its waters, and perhaps you will notice Ubuntu One being there.

🙈🙉🙊

[John Lenton (chipaca)]

Ouch!

This bug seems to have slipped through the cracks. I'll have somebody look at it today.

[Alberto Salvia Novella (es20490446e)]

Excuse the tone of my last comment for been inadequate.

[John Lenton (chipaca)]

Alberto, no need to excuse yourself; it's entirely understandable to get frustrated when something like this isn't addressed appropriately.

[John Lenton (chipaca)]

An update for those of you following along at home: a modified version of this patch is being real-life tested right now; we then need to change it slightly to get it ready for 12.04, at which point we'll SRU it to that, and 12.10, and 13.04.

If time permits the symmetric patch will also be done, to make download rate limiting effective, but I don't want to delay the upload fix (and testing the download fix is slightly harder).

Have a great weekend!

[Diogo Baeder (diogobaeder)]

I tried to reproduce this problem, functionally, but couldn't - uploading with throttling, and using wondershaper, still allowed me to navigate the internet and ping servers by hostname with almost no perceivable difference -.

However, we found out by our logs that throttling indeed doesn't work; Take these log samples, for example:
2013-05-09 17:53:06,045 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,045 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,046 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,046 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,047 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,047 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,047 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,048 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,048 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,048 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,049 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,049 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,049 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,050 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,050 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,050 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,051 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,051 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,052 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,052 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,052 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,053 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,053 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,053 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,054 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,054 - ubuntuone.SyncDaemon.StorageClient - TRACE - start - sendMessage: id: 25, type: BYTES
2013-05-09 17:53:06,054 - ubuntuone.SyncDaemon.StorageClient - TRACE - end - sendMessage: id: 25
2013-05-09 17:53:06,055 - ubuntuone.SyncDaemon.Sto...