Problem with Linux Kernel Traffic Control

[Vincent Autefage]

Hi,

The last main versions of QEMU (0.14.1, 0.15 and 1.0) have an important problem when running on a Linux distribution which running itself a Traffic Control (TC) instance.
Indeed, when TC is configured with a Token Bucket Filter (TBF) with a particular rate, the effective rate is very slower than the desired one.

For instance, lets consider the following configuration :

# tc qdisc add dev eth0 root tbf rate 20mbit burst 20k latency 50ms

The effective rate will be about 100kbit/s ! (verified with iperf)
I've encountered this problem on versions 0.14.1, 0.15 and 1.0 but not with the 0.14.0...
In the 0.14.0, we have a rate of 19.2 mbit/s which is quiet normal.

I've done the experimentation on several hosts :

- Debian 32bit core i7, 4GB RAM
- Debian 64bit core i7, 8GB RAM
- 3 different high performance servers : Ubuntu 64 bits, 48 AMD Opteron, 128GB of RAM

The problem is always the same... The problem is also seen with a Class Based Queuing (CBQ) in TC.

Thanks

[Stefan Hajnoczi (stefanha)]

Hi Vincent,
Please give steps to reproduce the problem including the QEMU
command-lines you used and what commands need to be run inside the
guest and on the host.

Stefan

[Vincent Autefage (autefage)]

Hi,

So, the host command lines are :
*$ qemu -name A -sdl -m 512 -enable-kvm -localtime -k fr -hda
debian1.img -net nic,macaddr=a0:00:00:00:00:01 -net
socket,mcast=230.0.0.1:7000*

The second is
*$ qemu -name B -sdl -m 512 -enable-kvm -localtime -k fr -hda
debian2.img -net nic,macaddr=a0:00:00:00:00:02 -net
socket,mcast=230.0.0.1:7000*

On virual machines :

*root@A# ifconfig eth0 192.168.0.1*
*root@A# tc qdisc add dev eth0 root tbf rate 20mbit burst 20480 latency
50ms*

*root@B# **ifconfig eth0 192.168.0.2*

Then if we check with /Iperf/, the real rate will be about 100kbit/s :

*root@B# iperf -s*

*root@A# iperf -c 192.168.0.1*

Vincent

Le 02/12/2011 14:34, Stefan Hajnoczi a écrit :
> Hi Vincent,
> Please give steps to reproduce the problem including the QEMU
> command-lines you used and what commands need to be run inside the
> guest and on the host.
>
> Stefan
>

[Stefan Hajnoczi (stefanha)]

On Fri, Dec 2, 2011 at 2:42 PM, Vincent Autefage
<email address hidden> wrote:
> *root@A# tc qdisc add dev eth0 root tbf rate 20mbit burst 20480 latency
> 50ms*
>
> *root@B# **ifconfig eth0 192.168.0.2*
>
> Then if we check with /Iperf/, the real rate will be about 100kbit/s :

What is the iperf result without tc? It's worth checking what rate
the unlimited interface saturates at before applying tc. Perhaps this
setup is just performing very poorly and it has nothing to do with tc.

Stefan

[Vincent Autefage (autefage)]

The result without TC is about 120 Mbit/s.
I check the bandwidth with lot of programs (not only with Iperf) and the
result is also the same....

However, if I use the same raw image and the same TC configuration with
the version 0.14.0 of QEMU or with some real physical hosts, the result
with TC is about 19.2 Mbit/s what is the desired result...

Vincent

Le 03/12/2011 19:48, Stefan Hajnoczi a écrit :
> On Fri, Dec 2, 2011 at 2:42 PM, Vincent Autefage
> <email address hidden> wrote:
>> *root@A# tc qdisc add dev eth0 root tbf rate 20mbit burst 20480 latency
>> 50ms*
>>
>> *root@B# **ifconfig eth0 192.168.0.2*
>>
>> Then if we check with /Iperf/, the real rate will be about 100kbit/s :
> What is the iperf result without tc? It's worth checking what rate
> the unlimited interface saturates at before applying tc. Perhaps this
> setup is just performing very poorly and it has nothing to do with tc.
>
> Stefan
>

[Stefan Hajnoczi (stefanha)]

On Sun, Dec 04, 2011 at 03:54:12PM -0000, Vincent Autefage wrote:
> The result without TC is about 120 Mbit/s.
> I check the bandwidth with lot of programs (not only with Iperf) and the
> result is also the same....
>
> However, if I use the same raw image and the same TC configuration with
> the version 0.14.0 of QEMU or with some real physical hosts, the result
> with TC is about 19.2 Mbit/s what is the desired result...

Thanks for checking if tc is involved in this bug.

Git bisect can identify which commit introduced the bug between QEMU
0.14.0 and 0.14.1. The following steps show how to do this:

Clone the QEMU git repository:
$ git clone git://git.qemu.org/qemu.git
$ cd qemu

Double-check that 0.14.1 has the bug:
$ git checkout v0.14.1
$ make distclean
$ ./configure --target-list=x86_64-softmmu
$ make
$ # test x86_64-softmmu/qemu-system-x86_64 binary

Double-check that 0.14.0 does *not* have the bug:
$ git checkout v0.14.0
$ make distclean
$ ./configure --target-list=x86_64-softmmu
$ make
$ # test x86_64-softmmu/qemu-system-x86_64 binary

Now you can be confident that 0.14.0 and 0.14.1 do indeed behave
differently when built from source. It's time to perform the bisect,
you can read more about what this does in the git-bisect(1) man page.

Find the commit that introduced the bug:
$ git bisect start v0.14.1 0.14.0
$ make distclean
$ ./configure --target-list=x86_64-softmmu
$ make
$ # test x86_64-softmmu/qemu-system-x86_64 binary

If tc achieves ~20 Mbit/s:
$ git bisect good

Otherwise:
$ git bisect bad

Git bisect will keep splitting the commit history in half until it
reaches the point where QEMU's behavior changes from good to bad. So
you typically need to build and test a couple of times until the guilty
commit has been identified.

Stefan

[Vincent Autefage (autefage)]

Hi,

So we have another problem...
The thing is that the 0.14.0 (and all 0.14.0 rc) built from GIT has the
same problem.
However, the package 0.14.0 from Ubuntu does not has this bug...

Le 05/12/2011 09:26, Stefan Hajnoczi a écrit :
> On Sun, Dec 04, 2011 at 03:54:12PM -0000, Vincent Autefage wrote:
>> The result without TC is about 120 Mbit/s.
>> I check the bandwidth with lot of programs (not only with Iperf) and the
>> result is also the same....
>>
>> However, if I use the same raw image and the same TC configuration with
>> the version 0.14.0 of QEMU or with some real physical hosts, the result
>> with TC is about 19.2 Mbit/s what is the desired result...
> Thanks for checking if tc is involved in this bug.
>
> Git bisect can identify which commit introduced the bug between QEMU
> 0.14.0 and 0.14.1. The following steps show how to do this:
>
> Clone the QEMU git repository:
> $ git clone git://git.qemu.org/qemu.git
> $ cd qemu
>
> Double-check that 0.14.1 has the bug:
> $ git checkout v0.14.1
> $ make distclean
> $ ./configure --target-list=x86_64-softmmu
> $ make
> $ # test x86_64-softmmu/qemu-system-x86_64 binary
>
> Double-check that 0.14.0 does *not* have the bug:
> $ git checkout v0.14.0
> $ make distclean
> $ ./configure --target-list=x86_64-softmmu
> $ make
> $ # test x86_64-softmmu/qemu-system-x86_64 binary
>
> Now you can be confident that 0.14.0 and 0.14.1 do indeed behave
> differently when built from source. It's time to perform the bisect,
> you can read more about what this does in the git-bisect(1) man page.
>
> Find the commit that introduced the bug:
> $ git bisect start v0.14.1 0.14.0
> $ make distclean
> $ ./configure --target-list=x86_64-softmmu
> $ make
> $ # test x86_64-softmmu/qemu-system-x86_64 binary
>
> If tc achieves ~20 Mbit/s:
> $ git bisect good
>
> Otherwise:
> $ git bisect bad
>
> Git bisect will keep splitting the commit history in half until it
> reaches the point where QEMU's behavior changes from good to bad. So
> you typically need to build and test a couple of times until the guilty
> commit has been identified.
>
> Stefan
>

[Stefan Hajnoczi (stefanha)]

On Mon, Dec 5, 2011 at 10:45 AM, Vincent Autefage
<email address hidden> wrote:
> So we have another problem...
> The thing is that the 0.14.0 (and all 0.14.0 rc) built from GIT has the
> same problem.
> However, the package 0.14.0 from Ubuntu does not has this bug...

Okay, that's actually a good thing because the issue is now isolated
to two similar builds: 0.14.0 from source and 0.14.0 from Ubuntu.

Either there is an environmental difference in the build configuration
or Ubuntu has applied patches on top of vanilla 0.14.0.

I think the next step is to grab the Ubuntu 0.14.0 source package and
rebuild it to confirm that it does *not* have the bug.

Then it's just a matter of figuring out what the difference is by a
(manual) bisection.

Are you using qemu-kvm? I found Ubuntu's 0.14.0-based package here:
http://packages.ubuntu.com/natty/qemu-kvm

Stefan

[Vincent Autefage (autefage)]

Yes this is the package that seems to not include the bug.
I'm going to check sources of this package.

Vincent Autefage

Le 05/12/2011 12:11, Stefan Hajnoczi a écrit :
> On Mon, Dec 5, 2011 at 10:45 AM, Vincent Autefage
> <email address hidden> wrote:
>> So we have another problem...
>> The thing is that the 0.14.0 (and all 0.14.0 rc) built from GIT has the
>> same problem.
>> However, the package 0.14.0 from Ubuntu does not has this bug...
> Okay, that's actually a good thing because the issue is now isolated
> to two similar builds: 0.14.0 from source and 0.14.0 from Ubuntu.
>
> Either there is an environmental difference in the build configuration
> or Ubuntu has applied patches on top of vanilla 0.14.0.
>
> I think the next step is to grab the Ubuntu 0.14.0 source package and
> rebuild it to confirm that it does *not* have the bug.
>
> Then it's just a matter of figuring out what the difference is by a
> (manual) bisection.
>
> Are you using qemu-kvm? I found Ubuntu's 0.14.0-based package here:
> http://packages.ubuntu.com/natty/qemu-kvm
>
> Stefan
>

[Vincent Autefage (autefage)]

Well....

I've compiled the ubuntu package.
When I've launched qemu, I've got this :
*
*$ *qemu-system-x86_64 -hda debian.img -m 512
qemu: could not load PC BIOS 'bios.bin'
**
*I've checked the content of the *pc-bios* directory and no bios are
generated but I've got strange file like :
**.bin
*.dtb
openbios-*
*
I think that the *configure* interprets the *** as a base character...
Therefore, I've copied the content of*pc-bios* directory of 0.15.1 in
the *pc-bios* directory of 0.14.0

Finally, the bug of rate have disappeared !!
*Iperf* gave me a rate of 19mbit which is the desired rate.

Vincent

Le 05/12/2011 12:11, Stefan Hajnoczi a écrit :
> On Mon, Dec 5, 2011 at 10:45 AM, Vincent Autefage
> <email address hidden> wrote:
>> So we have another problem...
>> The thing is that the 0.14.0 (and all 0.14.0 rc) built from GIT has the
>> same problem.
>> However, the package 0.14.0 from Ubuntu does not has this bug...
> Okay, that's actually a good thing because the issue is now isolated
> to two similar builds: 0.14.0 from source and 0.14.0 from Ubuntu.
>
> Either there is an environmental difference in the build configuration
> or Ubuntu has applied patches on top of vanilla 0.14.0.
>
> I think the next step is to grab the Ubuntu 0.14.0 source package and
> rebuild it to confirm that it does *not* have the bug.
>
> Then it's just a matter of figuring out what the difference is by a
> (manual) bisection.
>
> Are you using qemu-kvm? I found Ubuntu's 0.14.0-based package here:
> http://packages.ubuntu.com/natty/qemu-kvm
>
> Stefan
>

[Vincent Autefage (autefage)]

Well,

I have checked differences between the GIT repository (V0.14.0) and the
Ubuntu version (V0.14.0) and generated patch diff file.
The patch contains about 5000 lines...

What's the next step ?

Vincent

Le 05/12/2011 12:11, Stefan Hajnoczi a écrit :
> On Mon, Dec 5, 2011 at 10:45 AM, Vincent Autefage
> <email address hidden> wrote:
>> So we have another problem...
>> The thing is that the 0.14.0 (and all 0.14.0 rc) built from GIT has the
>> same problem.
>> However, the package 0.14.0 from Ubuntu does not has this bug...
> Okay, that's actually a good thing because the issue is now isolated
> to two similar builds: 0.14.0 from source and 0.14.0 from Ubuntu.
>
> Either there is an environmental difference in the build configuration
> or Ubuntu has applied patches on top of vanilla 0.14.0.
>
> I think the next step is to grab the Ubuntu 0.14.0 source package and
> rebuild it to confirm that it does *not* have the bug.
>
> Then it's just a matter of figuring out what the difference is by a
> (manual) bisection.
>
> Are you using qemu-kvm? I found Ubuntu's 0.14.0-based package here:
> http://packages.ubuntu.com/natty/qemu-kvm
>
> Stefan
>

[Vincent Autefage (autefage)]

I've just checked the problem with a *ne2k_pci* instead of the default
e1000 and the problem does not exist with the *ne2k_pci*... (Version
0.14-1 of qemu)

I'm going to check other cards right now

Vincent

Le 05/12/2011 12:11, Stefan Hajnoczi a écrit :
> On Mon, Dec 5, 2011 at 10:45 AM, Vincent Autefage
> <email address hidden> wrote:
>> So we have another problem...
>> The thing is that the 0.14.0 (and all 0.14.0 rc) built from GIT has the
>> same problem.
>> However, the package 0.14.0 from Ubuntu does not has this bug...
> Okay, that's actually a good thing because the issue is now isolated
> to two similar builds: 0.14.0 from source and 0.14.0 from Ubuntu.
>
> Either there is an environmental difference in the build configuration
> or Ubuntu has applied patches on top of vanilla 0.14.0.
>
> I think the next step is to grab the Ubuntu 0.14.0 source package and
> rebuild it to confirm that it does *not* have the bug.
>
> Then it's just a matter of figuring out what the difference is by a
> (manual) bisection.
>
> Are you using qemu-kvm? I found Ubuntu's 0.14.0-based package here:
> http://packages.ubuntu.com/natty/qemu-kvm
>
> Stefan
>

[Stefan Hajnoczi (stefanha)]

On Wed, Dec 14, 2011 at 1:36 PM, Vincent Autefage
<email address hidden> wrote:
> I have checked differences between the GIT repository (V0.14.0) and the
> Ubuntu version (V0.14.0) and generated patch diff file.
> The patch contains about 5000 lines...
>
> What's the next step ?

Okay, so when you rebuild the Ubuntu package from source the bug is
not present and the largish diff suggests they have added patches on
top of vanilla 0.14.0.

If the Ubuntu source ships with a number of .diff/.patch files that
get applied during the build then you could manually bisect this.
That means rerunning the Ubuntu build but with only the first half of
the list of patches applied. If that has the bug then split the
untested patches in half too and continue the test cycle. If the bug
is not present then split the patches in half and continue testing
until you reach the point where the bug goes from present to fixed.

Stefan

[Vincent Autefage (autefage)]

Ok so the *Intel e1000* seems the only card which is impacted by the bug.

Vincent

Le 05/12/2011 12:11, Stefan Hajnoczi a écrit :
> On Mon, Dec 5, 2011 at 10:45 AM, Vincent Autefage
> <email address hidden> wrote:
>> So we have another problem...
>> The thing is that the 0.14.0 (and all 0.14.0 rc) built from GIT has the
>> same problem.
>> However, the package 0.14.0 from Ubuntu does not has this bug...
> Okay, that's actually a good thing because the issue is now isolated
> to two similar builds: 0.14.0 from source and 0.14.0 from Ubuntu.
>
> Either there is an environmental difference in the build configuration
> or Ubuntu has applied patches on top of vanilla 0.14.0.
>
> I think the next step is to grab the Ubuntu 0.14.0 source package and
> rebuild it to confirm that it does *not* have the bug.
>
> Then it's just a matter of figuring out what the difference is by a
> (manual) bisection.
>
> Are you using qemu-kvm? I found Ubuntu's 0.14.0-based package here:
> http://packages.ubuntu.com/natty/qemu-kvm
>
> Stefan
>

[Stefan Hajnoczi (stefanha)]

On Wed, Dec 14, 2011 at 02:42:12PM -0000, Vincent Autefage wrote:
> Ok so the *Intel e1000* seems the only card which is impacted by the
> bug.

Let me recap with a summary of your debugging:

QEMU 0.14.0, 0.15.0, and 1.0 built from source all have poor network
performance below a 20 Mbit/s limit set with tc inside the guest.

Ubuntu's 0.14.0 QEMU package does not have poor network performance.

This problem only occurs with the emulated e1000 device. All other
emulated NICs operate correctly.

Now you could diff the e1000 emulation code to get the code changes
between vanilla and Ubuntu:

 $ diff -u qemu-0.14.0-vanilla/hw/e1000.c qemu-0.14.0-ubuntu/hw/e1000.c

(It's possible that there are no significant changes and this bug is
caused by something outside e1000.c but this is place to check first.)

Or you could even try copying Ubuntu's e1000.c into the vanilla QEMU
source tree and retesting to see if the behavior changes.

Stefan

[Vincent Autefage (autefage)]

Ok,

So the e1000.c and the e1000_hw.h have absolutely no difference between
the original and the ubuntu version...
The only differences witch refers to the *Intel e1000* in the wall
sources is this one :

diff -ru qemu//hw/pc_piix.c qemu-kvm-0.14.0+noroms//hw/pc_piix.c
--- qemu//hw/pc_piix.c 2011-12-15 15:37:28.539290000 +0100
+++ qemu-kvm-0.14.0+noroms//hw/pc_piix.c 2011-02-22
14:34:38.000000000 +0100

@@ -123,7 +141,7 @@
          if (!pci_enabled || (nd->model && strcmp(nd->model,
"ne2k_isa") == 0))
              pc_init_ne2k_isa(nd);
          else
- pci_nic_init_nofail(nd, "e1000", NULL);
+ pci_nic_init_nofail(nd, "rtl8139", NULL);
      }

      if (drive_get_max_bus(IF_IDE) >= MAX_IDE_BUS) {

Vincent

Le 15/12/2011 09:07, Stefan Hajnoczi a écrit :
> On Wed, Dec 14, 2011 at 02:42:12PM -0000, Vincent Autefage wrote:
>> Ok so the *Intel e1000* seems the only card which is impacted by the
>> bug.
> Let me recap with a summary of your debugging:
>
> QEMU 0.14.0, 0.15.0, and 1.0 built from source all have poor network
> performance below a 20 Mbit/s limit set with tc inside the guest.
>
> Ubuntu's 0.14.0 QEMU package does not have poor network performance.
>
> This problem only occurs with the emulated e1000 device. All other
> emulated NICs operate correctly.
>
> Now you could diff the e1000 emulation code to get the code changes
> between vanilla and Ubuntu:
>
> $ diff -u qemu-0.14.0-vanilla/hw/e1000.c qemu-0.14.0-ubuntu/hw/e1000.c
>
> (It's possible that there are no significant changes and this bug is
> caused by something outside e1000.c but this is place to check first.)
>
> Or you could even try copying Ubuntu's e1000.c into the vanilla QEMU
> source tree and retesting to see if the behavior changes.
>
> Stefan
>

[Stefan Hajnoczi (stefanha)]

On Thu, Dec 15, 2011 at 3:03 PM, Vincent Autefage
<email address hidden> wrote:
> Ok,
>
> So the e1000.c and the e1000_hw.h have absolutely no difference between
> the original and the ubuntu version...
> The only differences witch refers to the *Intel e1000* in the wall
> sources is this one :
>
>
> diff -ru qemu//hw/pc_piix.c qemu-kvm-0.14.0+noroms//hw/pc_piix.c
> --- qemu//hw/pc_piix.c  2011-12-15 15:37:28.539290000 +0100
> +++ qemu-kvm-0.14.0+noroms//hw/pc_piix.c        2011-02-22
> 14:34:38.000000000 +0100
>
> @@ -123,7 +141,7 @@
>          if (!pci_enabled || (nd->model && strcmp(nd->model,
> "ne2k_isa") == 0))
>              pc_init_ne2k_isa(nd);
>          else
> -            pci_nic_init_nofail(nd, "e1000", NULL);
> +            pci_nic_init_nofail(nd, "rtl8139", NULL);
>      }
>
>      if (drive_get_max_bus(IF_IDE) >= MAX_IDE_BUS) {

That looks like it is only changing the default NIC from e1000 to rtl8139.

Stefan

[Stefan Hajnoczi (stefanha)]

On Thu, Dec 15, 2011 at 4:09 PM, Stefan Hajnoczi <email address hidden> wrote:
> On Thu, Dec 15, 2011 at 3:03 PM, Vincent Autefage
> <email address hidden> wrote:
>> Ok,
>>
>> So the e1000.c and the e1000_hw.h have absolutely no difference between
>> the original and the ubuntu version...
>> The only differences witch refers to the *Intel e1000* in the wall
>> sources is this one :
>>
>>
>> diff -ru qemu//hw/pc_piix.c qemu-kvm-0.14.0+noroms//hw/pc_piix.c
>> --- qemu//hw/pc_piix.c  2011-12-15 15:37:28.539290000 +0100
>> +++ qemu-kvm-0.14.0+noroms//hw/pc_piix.c        2011-02-22
>> 14:34:38.000000000 +0100
>>
>> @@ -123,7 +141,7 @@
>>          if (!pci_enabled || (nd->model && strcmp(nd->model,
>> "ne2k_isa") == 0))
>>              pc_init_ne2k_isa(nd);
>>          else
>> -            pci_nic_init_nofail(nd, "e1000", NULL);
>> +            pci_nic_init_nofail(nd, "rtl8139", NULL);
>>      }
>>
>>      if (drive_get_max_bus(IF_IDE) >= MAX_IDE_BUS) {
>
> That looks like it is only changing the default NIC from e1000 to rtl8139.

Perhaps you can stop Ubuntu from applying its patches on top of
vanilla QEMU but still use the same build process. In other words,
try building the vanilla QEMU source but using Ubuntu's method (not
sure if you are using dpkg build tools here). If it turns out the
binary does not have the bug then we know it's an environmental issue
like a ./configure difference or similar.

Stefan

[Vincent Autefage (autefage)]

Here is the problem !

The Ubuntu version works only because it not uses an *Intel e1000* but a
*rtl8139*.
Therefore, the problem about the e1000 is present in *all* version
(original or ubuntu ones).

Thus, the file *e1000.c* must contain some instructions which imply the
bad TC behavior.

Vincent

Le 15/12/2011 17:09, Stefan Hajnoczi a écrit :
> On Thu, Dec 15, 2011 at 3:03 PM, Vincent Autefage
> <email address hidden> wrote:
>> Ok,
>>
>> So the e1000.c and the e1000_hw.h have absolutely no difference between
>> the original and the ubuntu version...
>> The only differences witch refers to the *Intel e1000* in the wall
>> sources is this one :
>>
>>
>> diff -ru qemu//hw/pc_piix.c qemu-kvm-0.14.0+noroms//hw/pc_piix.c
>> --- qemu//hw/pc_piix.c 2011-12-15 15:37:28.539290000 +0100
>> +++ qemu-kvm-0.14.0+noroms//hw/pc_piix.c 2011-02-22
>> 14:34:38.000000000 +0100
>>
>> @@ -123,7 +141,7 @@
>> if (!pci_enabled || (nd->model&& strcmp(nd->model,
>> "ne2k_isa") == 0))
>> pc_init_ne2k_isa(nd);
>> else
>> - pci_nic_init_nofail(nd, "e1000", NULL);
>> + pci_nic_init_nofail(nd, "rtl8139", NULL);
>> }
>>
>> if (drive_get_max_bus(IF_IDE)>= MAX_IDE_BUS) {
> That looks like it is only changing the default NIC from e1000 to
> rtl8139.
>
> Stefan
>

[Stefan Hajnoczi (stefanha)]

On Thu, Dec 15, 2011 at 04:48:13PM -0000, Vincent Autefage wrote:
> Here is the problem !
>
> The Ubuntu version works only because it not uses an *Intel e1000* but a
> *rtl8139*.
> Therefore, the problem about the e1000 is present in *all* version
> (original or ubuntu ones).
>
> Thus, the file *e1000.c* must contain some instructions which imply the
> bad TC behavior.

You are right! Looking back at your QEMU command-line you are not
explicitly specifying the NIC model so the default will take effect.

Now we're back to square one: e1000.c performs poorly when the tc
command you posted is used. We don't know why yet.

Michael: Have you ever encountered unexpectedly low throughput when tc
is used inside the guest?

# tc qdisc add dev eth0 root tbf rate 20mbit burst 20k latency 50ms

The observed throughput from iperf is only 100kbit/s, not around
20mbit/s as expected. When tc is not run inside the guest then the NIC
saturates 20mbit/s easily.

Stefan

[Henrique Rodrigues (henriquesilvar)]

Hi guys,

I'm having the same problem with a ubuntu 11.04 (natty) host. I tried to set the rate controllers using tc both at the host and inside the guest i.e.:

tc qdisc add vnic0 root tbf rate 20mbit burst 20480 latency 50ms (host - to control the traffic going to the guest vm) and
tc qdisc add eth0 root tbf rate 20mbit burst 20480 latency 50ms (guest)

And the results are the same reported initially: ~140kbit/sec. I also tried to use policing filters at the host but I got the same results.

However, if I use htb I can get reasonable throughputs (~20mbit). I used these commands (both for host and guest):

tc qdisc add dev <DEV> root handle 1: htb default 255
tc class add dev <DEV> parent 1: classid 1:1 htb rate 20mbit burst 20480
tc filter add dev <DEV> parent 1: prio 255 proto ip u32 match ip src 0.0.0.0/0 flowid 1:1

It seems that the problem is related with the root qdisc only. Have you guys found an answer for this?

[Vincent Autefage (autefage)]

Hi,

The problem seems to come from the implementation of the Intel e1000
network cards (which is the default one used by QEMU).
If you use another one, the problem does not appear ;)

Vince

Le 29/01/2012 05:49, Henrique Rodrigues a écrit :
> Hi guys,
>
> I'm having the same problem with a ubuntu 11.04 (natty) host. I tried to
> set the rate controllers using tc both at the host and inside the guest
> i.e.:
>
> tc qdisc add vnic0 root tbf rate 20mbit burst 20480 latency 50ms (host - to control the traffic going to the guest vm) and
> tc qdisc add eth0 root tbf rate 20mbit burst 20480 latency 50ms (guest)
>
> And the results are the same reported initially: ~140kbit/sec. I also
> tried to use policing filters at the host but I got the same results.
>
> However, if I use htb I can get reasonable throughputs (~20mbit). I used
> these commands (both for host and guest):
>
> tc qdisc add dev<DEV> root handle 1: htb default 255
> tc class add dev<DEV> parent 1: classid 1:1 htb rate 20mbit burst 20480
> tc filter add dev<DEV> parent 1: prio 255 proto ip u32 match ip src 0.0.0.0/0 flowid 1:1
>
> It seems that the problem is related with the root qdisc only. Have you
> guys found an answer for this?
>

[Henrique Rodrigues (henriquesilvar)]

Hi,

I figured out what was the problem. It seems that the pkts generated by each guest iperf command is bigger than the default qdisc mtu of 2kb (the pkts have length of 65k). If you set a higher qdisc mtu (=65k) the traffic should be controlled as expected.

Henrique

[Henrique Rodrigues (henriquesilvar)]

Vincent,

Have you tried to change the mtu of the tbf qdisc? The traffic control should work well if you set it to 65kb.
I believe that this is happening due to the napi gro functionality. I'm still not sure, but the problem seems to be related to that.

Henrique

[Vincent Autefage (autefage)]

Hi,

No I don't try, i will :)
The probleme is not present with another NIC so I use another one for
the moment.

Vincent

Le 09/02/2012 20:05, Henrique Rodrigues a écrit :
> Vincent,
>
> Have you tried to change the mtu of the tbf qdisc? The traffic control should work well if you set it to 65kb.
> I believe that this is happening due to the napi gro functionality. I'm still not sure, but the problem seems to be related to that.
>
> Henrique
>

[Thomas Huth (th-huth)]

Can you still reproduce this issue with the latest version of QEMU (currently v2.8), or could we close this ticket nowadays?

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]