Network Performance Problem with GRE using Openvswitch

[Eren]

We are having GRE performance issues with Juno installation. From VM to network node, we can only get 3Gbit on 10Gbit interface. Finally, I tracked and solved the issue but that requires patches to nova and neutron-plugin-openvswitch. I am reporting this bug to find a clean solution instead of a hack.

The isssue is caused by MTU setting and lack of multiqueue net support in kvm. As official openstack documentation suggests, MTU settings are 1500 by default. This creates a bottleneck in VMs and it's only possible to process 3Gbit network traffic with 1500 MTU and without MQ support enabled in KVM.

What I did to solve the issue:
1- Set physical interface (em1) mtu to 9000
2- Set network_device_mtu = 8950 in nova and neutron.conf (both on compute/network nodes)
3- Set br-int mtu to 8950 manually
4- Set br-tun mtu to 8976 manually
5- Set VM MTU to be 8950 in dnsmasq-neutron.conf
6- Patch nova config code to add <device driver='vhost' queue='4'> element in libvirt.xml
7- Run "ethtool -L eth0 combined 4" in VMs

With network_device_mtu setting, tap/qvo/qvb in compute nodes and internal legs in the router/dhcp namespace in network node can be set automatically. However, it only solves half of the problem. I still need to set mtu to br-int and br-tun interfaces.

To enable MQ support in KVM, I needed to patch nova. Currently, there is no possible way to set queues in libvirt.xml. Without MQ support, even if jumbo frames are enabled, VMs are limited to 5Gbit. This is because of the fact that [vhost-xxxx] process is bound to one CPU and network load cannot be distributed to other CPUs. When MQ is enabled, [vhost-xxxx] can be distributed to other cores, which gives 9.3Gbit performance.

I am adding my ugly hacks just to give some idea on code change. I know that it is not a right way. Let's discuss how to properly address this issue.

Should I open another related bug to nova as this issue needs a change in nova code as well?

Note: this is a different bug than https://bugs.launchpad.net/bugs/1252900 affecting Juno release.

[Eren (erent)]

For the full history how how I came up with a solution, please take a look at this thread on openstack mailing list: http://lists.openstack.org/pipermail/openstack/2015-January/011207.html

This issue is solved by "ethtool -K em1 tx off". And the performance problem is solved as I explained in the bug report.

[rick jones (rick-jones2)]

Just as a matter of style/taste/whatnot I would probably separate the MTU issue from the virtio multiqueue issue. Separate bugs. Indeed, they both (along, I suspect with NIC HW offload support for GRE/VxLAN encapsulated traffic) contribute to performance, but they don't need to be bundled in the one defect. That's just my stripe of color on the bikeshed...

[Jay Pipes (jaypipes)]

This is excellent feedback and investigation, Eren. I've forwarded this and your mailing list thread to our Fuel engineers to digest (since part of the solution relates to the configuration of the bare-metal bridge interfaces). I think we can mark the Nova part of this as confirmed after verifying that you cannot set the multiple queues attribute in libvirt.xml.

[Eren (erent)]

@rick-jones: That may be doeble but the get full saturation of the link in VMs, MQ support should be enabled in nova. When we seperate the issues and fix only MTU part, VMs can only have (at most) 5Gbit due to lack of MQ support. CPU cannot handle the traffic and this gives you only half of the available bandwidth.

@jaypipes: Thank you! I'm pretty sure that we cannot set multiple queue attribute in nova. Whenever I grep, I couldn't find a reference to related part of the code. There is an open blueprint that wasn't accepted regarding to the issue but that blueprint suggests setting queues based on image attributes (the number of VCPUs in the instance). Marking the the queue based on image attribute can be done but I prefer the simplicity of writing the configuration in nova.conf. If the queues are enabled based on VCPU number, tiny instances will not get full link because there will be only 1 queue wheres other instances will be able to use it.

Nevertheless, we need to resurrect the MQ feature discussion.

https://blueprints.launchpad.net/nova/+spec/libvirt-virtio-net-multiqueue

[Ihar Hrachyshka (ihar-hrachyshka)]

A spec to solve MTU issues is approved for Kilo and in progress: https://github.com/openstack/neutron-specs/blob/master/specs/kilo/mtu-selection-and-advertisement.rst

[Eren (erent)]

On 09-02-2015 13:46, Ihar Hrachyshka wrote:
> A spec to solve MTU issues is approved for Kilo and in progress:
> https://github.com/openstack/neutron-specs/blob/master/specs/kilo/mtu-
> selection-and-advertisement.rst

Great news! What about multiqueue-net support for nova? If we cannot set MQ
support, VMs will be bound to 5Gbit for single CPU (depending on the power of
one CPU, of course).

With MQ support in Nova, I believe performance problems will be solved forever.

--
System Administrator
https://skyatlas.com/

[Stephen Gordon (sgordon)]

I believe Vladik is planning to re-submit the virtio multiqueue specification for Liberty.

[Stephen Gordon (sgordon)]

In fact looks like he already did, would be great to see more feedback on the relevant proposal: https://review.openstack.org/#/c/128825/

[Eren (erent)]

On 25-03-2015 22:29, Stephen Gordon wrote:
> In fact looks like he already did, would be great to see more feedback
> on the relevant proposal: https://review.openstack.org/#/c/128825/

Thanks for the link! I didn't know it was submitted again recently. I
hope it gets accepted this time.

--
Eren Türkay, System Administrator
https://skyatlas.com/ | [phone]

Yildiz Teknik Universitesi Davutpasa Kampusu
Teknopark Bolgesi, D2 Blok No:107
Esenler, Istanbul Pk.34220

[Eugene Nikanorov (enikanorov)]

I'm setting this to Confirmed for now. I think this might be addressed in the scope of some effort introducing changeble mtu.

[Ramu Ramamurthy (ramu-ramamurthy)]

GRO support for GRE with OVS has been added in 4.0 kernels.

That boosts TCP stream bandwidth significantly when you are using OVS with GRO - so you dont have to set MTUs all the way from the eth to the VM.

If you are running 3.x kernels, you can try this kernel module to boost the bandwidth
(I authored that kernel module but found it was fixed in 4.x already by the OVS folks)

http://marc.info/?t=144468885300003&r=1&w=2

[Eren (erent)]

On 04-11-2015 20:52, Ramu Ramamurthy wrote:
> That boosts TCP stream bandwidth significantly when you are using OVS
> with GRO - so you dont have to set MTUs all the way from the eth to the
> VM.

So that leaves us to the multiqueue support in the VMs, which is already merged
into Liberty (partial). With this change and newer kernels, that should fix the
problem without changing the MTU. However, for 3.x kernels, we need to compile
the kernel module.

Are there any documentation on how to build this module and use it? Or should
we just update to 4.x kernel and leave OVS version same to benefit from this
change?

--
Eren Türkay, System Administrator
https://skyatlas.com/ | [phone]

Yildiz Teknik Universitesi Davutpasa Kampusu
Teknopark Bolgesi, D2 Blok No:107
Esenler, Istanbul Pk.34220

[Ramu Ramamurthy (ramu-ramamurthy)]

Instructions to build that kernel module depend on the distribution (ubuntu/centos/etc) - but is quite straightforward, and is well documented.

You could try the 4.x kernel (and keep the same ovs) as well although I cannot confirm it as I have not tested with 4.x kernels and ovs.

[Ihar Hrachyshka (ihar-hrachyshka)]

From Neutron perspective, the following fixes should cover MTU setting as of Mitaka:

bug 1542475
bug 1542108

[Markus Zoeller (markus_z) (mzoeller)]

@Eren: If I understand your comment #14 correctly, the issue(s) described in this bug report are solved, I'm setting it to "Fix released". If this is not the case, it makes sense to open a new bug report which describes only one specific issue (not multiple at once).