virtio-net performance in 10Gbs lan

[Walt Corey]

Is this a bug or configuration?

virtio_net performance
In the course of helping a fellow UEC user he, and consequently we, hit a potential bug. In the case of a system with an e1000 or even a Realtek NIK or simulated NIC performance sucks with the emulated driver in an instance. But changing the eucalyptus.conf settings for virtio_* to 1 the generated kvm command line uses virtio where possible. This brings performance up substancially. In the case of an e1000 or Realtek NIC performance is right about 800Mbs (very close to hardware speed. This is wonderful and way better than using a fully virtualized driver.

------------------------------------------------------------
Client connecting to 172.19.1.8, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 172.19.1.7 port 60596 connected with 172.19.1.8 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 67.4 MBytes 566 Mbits/sec
[ 3] 1.0- 2.0 sec 90.0 MBytes 755 Mbits/sec
[ 3] 2.0- 3.0 sec 90.5 MBytes 759 Mbits/sec
[ 3] 3.0- 4.0 sec 98.4 MBytes 826 Mbits/sec
[ 3] 4.0- 5.0 sec 91.9 MBytes 771 Mbits/sec
[ 3] 5.0- 6.0 sec 92.9 MBytes 779 Mbits/sec
[ 3] 6.0- 7.0 sec 94.0 MBytes 789 Mbits/sec
[ 3] 7.0- 8.0 sec 97.1 MBytes 815 Mbits/sec
[ 3] 8.0- 9.0 sec 103 MBytes 860 Mbits/sec
[ 3] 9.0-10.0 sec 95.7 MBytes 803 Mbits/sec
[ 3] 0.0-10.0 sec 921 MBytes 772 Mbits/sec

However for people running 10GB NIC cards by specifying virtio_net only brings their speed up to just over 1GB/s.

ubuntu@172:~$ ./nuttcp 172.19.1.4 <- virtual machine speed
1283.1875 MB / 10.00 sec = 1076.4544 Mbps 39 %TX 45 %RX 0 retrans 1.49 msRTT

Same test at the base install level.... <- real measured machine speed

hoot@cloud1:~$ ./nuttcp 10.10.10.3
11048.6250 MB / 10.00 sec = 9265.0703 Mbps 38 %TX 87 %RX 0 retrans 0.22 msRTT

From just anecdotal hits on Google this appears to be an introduced bug in recent versions of either KVM or Linux Kernel. I saw a thread where if one were to recompile and install a different kvm-qemu the problem is rectified. I open this thread in order to provide a central place to separate fact from fiction and, hopefully, provide closure on this issue.

If such a fix exists, will, or can, it be included in U11.04?

References:
http://www.mail-archive.com/kvm@vger.../msg42690.html
http://forum.proxmox.com/threads/500...et-performance

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug and helping to make Ubuntu better.

Unfortunately your reference links got abbreviated so I was unable to follow them.

Could you please provide a little more information about your particular setup:

1. Which version of Ubuntu are you on? I'm wondering whether the KVM 0.14-rc1 in natty still has this problem. If you're not on natty, I'll ask you to try to reproduce with the latest qemu and kernels (separately and together) so we can try to nail this down.

2. Can you give the full command being executed in both cases (i.e. ps -ef or /proc/$pid/cmdline output)? Since presumably this is being spawned through two layers (by eucalyptus through libvirt), there could be inadvertant bad interactions between some of the options.

3. I'll wait to see output of 2, but will probably want to see the results with the shortest possible hand-made kvm command lines.

[Walt Corey (waltc)]

I believe this is fixed in QEMU 0.14... referring to changelist (first paragraph) on
http://wiki.qemu.org/Changelog/0.14#virtio

The references I gave above specifically referenced the authors applying the 'bottom-half' fix, although they did not, as I recall, mention what the fixed was against. Where QEMU 0.14 is final now will Natty pick the final release of it?

To answer your questions, I am running 10.10 Maverick as the node controller and guest OS. Does that fix go against the host (node controller OS or the guest OS? In other words, would running a current Natty ec2 image resolve the issue?
Thanks!

[Walt Corey (waltc)]

I used iperf -s on one instance and iperf -c xxx.xx.xx.xx on the client image of two guest OS's running under UEC.

[Walt Corey (waltc)]

I tried running natty guests and the results were actually slightly slower. This makes sense as kvm is a host component and qemu is affiliated with kvm. Presumably there is no guest component?

[Serge Hallyn (serge-hallyn)]

I'll assume by your comments that you are in fact on 11.04. Yes, we will hopefully merge qemu-kvm-0.14.0 soon as a bug-fix for this.

[Walt Corey (waltc)]

My omission. I thought I had said I was on 10.10. I am at 10.10 for both UEC as well as guests. I merely tried 11.04 to see if that resolved the issue. However I really didn't expect to see a change, my nic is 1Gb. I was helping a guy on the Euca mb that didn't realize there was a virtio_net and then he came back with how it helped but only slightly and his 10Gb network was running at slightly better than 1Gb. I read a blog by Graziano@Euca who mentioned work on Qemu and I went to their site and saw the 0.14 was now GA, up from the rc1 in 11.04alpha. I looked into it and saw the reference to bottom-half which was in the other mb thread I tried to copy referring to the guy having rebuilt with the bottom-half fix but found no other reference to it. But the, aforementioned, guy did say that did resolve the issue, so yes, Qemu 0.14 resolves this problem.

[Serge Hallyn (serge-hallyn)]

Thanks, Waltc. I'll put together a package today based on the final qemu-kvm-0.14 for you to test, just to make sure.

Note that we won't be able to put 0.14 into maverick, but if we can choose a few commits to cherrypick, that would be do-able. If you're just as well off using natty, that'll be much simpler.

[Serge Hallyn (serge-hallyn)]

@Waltc,

actually the commits you were hoping for are already in the 0.14.0-rc1 which is currently in natty.

[Serge Hallyn (serge-hallyn)]

I'll still finish the 0.14.0 build for the archive, and then go ahead and create a new daily build with which to test. If that still exhibits the problem then we'll involve upstream.

[Serge Hallyn (serge-hallyn)]

Here are the amd64 debs of a package built on the qemu-kvm 0.14.0 final release. I hope to get this into natty in any case, but could you test to see if this fixes the performance degradation for you?

[Serge Hallyn (serge-hallyn)]

I've also put the very latest upstream qemu-kvm tree up at https://launchpad.net/~ubuntu-server-edgers/+archive/server-edgers-qemu-kvm. If you can reproduce the performance regression with that package, then we can engage upstream.

[Walt Corey (waltc)]

I don't know that I can successfully vet the fix. I can say from the hit I found on some forum the guy with the 10Gb network verified as fixed after applying the 'bottom-half fix. I will try that this w/e. I see you changed the status to incomplete. Is that due to waiting on me? Oh, I also appreciate you having gotten right on this.

Thank you!

[Serge Hallyn (serge-hallyn)]

Right, marking 'Incomplete' is so that when looking at long buglists I can tell which ones are waiting on input. Please feel free to move it back to 'confirmed' or 'in progress' when you have info. (Though I"ll get email so it's not necessary).

[Serge Hallyn (serge-hallyn)]

@waltc,

is this bug fixed for you in the latest natty qemu-kvm? Latest natty is based on qemu-kvm-0.14.0 release, which you say includes the fixes you were looking for.

[Walt Corey (waltc)]

From what I can see it is.

On 03/20/2011 08:57 PM, Serge Hallyn wrote:
> @waltc,
>
> is this bug fixed for you in the latest natty qemu-kvm? Latest natty is
> based on qemu-kvm-0.14.0 release, which you say includes the fixes you
> were looking for.
>

[Serge Hallyn (serge-hallyn)]

Thanks very much, will mark it 'fix released' then.