Default MTU with openvswitch plugin not modifiable

Bug #1075336 reported by Michael H Wilson
102
This bug affects 17 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Jun Park

Bug Description

When using a provider network of type flat you will actually loose data due to vlan tagging and stripping if you don't increase the MTU for the patch ports between your integration bridge and the physical network bridge. I tried setting network_device_mtu which I found reference to in some searching around, but that doesn't seem to work.

Tags: ovs
Gary Kotton (garyk)
Changed in quantum:
status: New → Incomplete
status: Incomplete → New
Revision history for this message
dan wendlandt (danwent) wrote :

Adding Bob, who is probably the best person to look at this.

tags: added: plugin-ovs
Revision history for this message
Gary Kotton (garyk) wrote :

Hi,
My two cents is that we add the MTU as a provider network field.
Thanks
Gary

dan wendlandt (danwent)
Changed in quantum:
status: New → Confirmed
assignee: nobody → Robert Kukura (rkukura)
dan wendlandt (danwent)
Changed in quantum:
importance: Undecided → Medium
tags: added: ovs
removed: plugin-ovs
Revision history for this message
Michael H Wilson (geekinutah) wrote :

My colleague Jun Park has a patch for this... I'm going to get him to submit it here in a bit. Just have to get him set up as a contributor.

Revision history for this message
ben42 (ben42ml) wrote :

Hello,

I really like to have the ability to configure mtu for :
- Gre tunnel on hypervisor and network node
- Tap interface on the hypervisor node and network node

Any news about the path ?

Thank you in advance.

Regards,

Revision history for this message
Madko (madko) wrote :

Interested by the patch too, very slow perf with default mtu

Revision history for this message
Sergey Filimontsev (askort) wrote :

Hello!

Any news?

WBR

Revision history for this message
Jun Park (jun-park-earth) wrote :

I will provide a patch for this soon. Now simply trying to familiarize myself with the required process for contributing.

Revision history for this message
Jun Park (jun-park-earth) wrote :

I submitted an initial patch with lots of mistakes including "no bug reference in git log." Thanks for all the comments. I mostly reflected them on a new patch. However, one thing I would like to discuss is about the desired default MTU size of veth interfaces. I think since the current code uses a tagging mechanism for both 'flat' and 'vlan' network type, the default MTU size of veth interfaces (that are used for allowing traffic between two OVS bridges: br-int and br-ethxx) should be 1504 rather than 1500. So other than this different default MTU size, I believe I corrected all the problems that the initial patch had. I will resubmit the new patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (master)

Fix proposed to branch: master
Review: https://review.openstack.org/27925

Changed in quantum:
assignee: Robert Kukura (rkukura) → Jun Park (jun-park-earth)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/27937

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/31518

Revision history for this message
Ian Wells (ijw-ubuntu) wrote :

Seconded the comment above about MTUs on GRE-overlay networks. In fact, in general, it's pretty much impossible for anything but Quantum to work out what a network's MTU is - and it's rare that it's a straightforward 1500.

Revision history for this message
johnsterdotcom (y-john) wrote :

What about the case where some interfaces need one MTU size, and others need a different one?

IE: management network on 1G interfaces with MTU = 1500 versus a storage / data network on 10G interfaces with MTU = 9000

In that case, this patch seems inadequate, as there's only one MTU value that can be set.

Revision history for this message
Jun Park (jun-park-earth) wrote :

To Ian Wells:

If there is a specific bug report related GRE tunnels regarding MTU, wouldn't it be better to deal with it separately? This bug fix is only fixing the given bug for veth interfaces when VLAN tagging is used (just like in either flat or vlan mode). Basically, the current title of this thread is too broad, so I agree it's misleading.

To Johnsterdot.com:

Again this bug fix is to allow the MTU size adjustable only for veth interfaces, not for bridge, tap or anything else. This bug seems to occur in a certain environment only (e.g., in Linux 2.6.x, not Linux 3.x). So for those who use Linux 2.6.x, this patch can be used to fix the MTU size-related issue as described in this thread.

Revision history for this message
johnsterdotcom (y-john) wrote :

I have a patch that does what I'm suggesting (and what I need for my network topology).

Could someone help me understand the process for submitting it here?

Thanks,

- John

Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

Hi John,

The process is described here: https://wiki.openstack.org/wiki/GerritWorkflow

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (master)

Fix proposed to branch: master
Review: https://review.openstack.org/35225

Changed in neutron:
assignee: Jun Park (jun-park-earth) → johnsterdotcom (y-john)
Revision history for this message
johnsterdotcom (y-john) wrote :

I pushed and alternate implementation for consideration.

https://review.openstack.org/#/c/35225/

Looks like the system is a step ahead of me. :)

Changed in neutron:
assignee: johnsterdotcom (y-john) → Jun Park (jun-park-earth)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/27937
Committed: http://github.com/openstack/neutron/commit/5808a65fa12be096278f3793909a62bf1d79e82b
Submitter: Jenkins
Branch: master

commit 5808a65fa12be096278f3793909a62bf1d79e82b
Author: Jun Park <email address hidden>
Date: Fri Jul 19 11:58:00 2013 -0600

    Allow OVS default veth MTU to be configured.

    In some environments where a packet is dropped when a VLAN tag is
    added to the packet, you need to increase the MTU size of veth
    interfaces to 1504.

    Fixes: bug #1075336
    Change-Id: I4f03b4cdc571a462096d419d6dd8324cf096156b

Changed in neutron:
status: In Progress → Fix Committed
Changed in neutron:
milestone: none → havana-3
Revision history for this message
Uri Simchoni (uri-simchoni) wrote :

Are there plans to backport this to Grizzly? The bug exists on CentOS 6.4 kernel, so anyone wishing to deploy Grizzly on a CentOS 6.4 unmodified kernel (at least on compute nodes where the namespace issue does not exist) will have an issue here...

Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: havana-3 → 2013.2
Revision history for this message
Jesse Pretorius (jesse-pretorius) wrote :

I'd also like to see a backport to Grizzly.

Revision history for this message
Marcelo Amaral (marcelo-bsc) wrote :

This patch is for the openvswitch plugin, but what about the ml2 plugin.
It still has the same problem when using GRE.

Revision history for this message
Jun Park (jun-park-earth) wrote :

Marcelo,

Here is a sample conf in "/etc/neutron/plugins/ml2/ml2_conf.ini" to make use of the veth_mtu option for the ml2 plugin with the openvswitch mechanism driver.

[ml2]
...
mechanism_drivers = openvswitch
...
[agent]
...
tunnel_types = vxlan
veth_mtu = 1504
root_helper = sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
l2_population = True
...

For clarification, this patch was only for dealing with the mtu size on veth interfaces between br-int and br-ex. I'm not sure if there is another patch to deal with the mtu size for gre/vxlan tunnels. Can you provide a detailed problem description for the gre issue? Or simply provide another bug report regarding the gre issue that you have?

Revision history for this message
Ian Wells (ijw-ubuntu) wrote :

There are various settings that are relatively illogical and all need changing consistently. They also don't work universally.

network_device_mtu
veth_mtu (it's completely unclear why this needs a separate config option)
the MTU on any interface that transmits e.g. VLAN, GRE or VXLAN traffic (VXLAN traffic is UDP and may fragment, technically, although the proposed standard prohibits it)
Additionally the Path MTU must be sufficient to transmit a GRE or VXLAN packets.
Also, in the case of VLAN tagged packets over a tunnel such as VXLAN, the VLAN tag header costs an additional 4 (?) bytes that are not usually counted within the packet MTU, which means that the MTU is slightly reduced.

It's important to see here that because the network_device_mtu is, in many circumstances, dependent on the path MTU, there's no automatic way of deducing it. It's also important to understand that the network_device_mtu is not (because of the VLAN tag issue mentioned above) always the MTU you wish to set on your VM interface.

Revision history for this message
Jun Park (jun-park-earth) wrote :

Ian,

Ian>veth_mtu (it's completely unclear why this needs a separate config option)

Please refer to the original review: https://review.openstack.org/#/c/27937/. Basically what I remember about this patch (too long ago for me) is that the MTU size adjustment was needed on Linux Kernel 2.6.x, not in 3.x. I don't know what kind of kernel issue caused the problem of packet dropping in 2.6.x when no mtu size was increased. So at that time my patch was a simple workaround that allows to increase mtu size on veth interfaces for a certain env, e.g., CentOS6 that uses 2.6.x. It means that some other env (e.g., Linux Kernel 3.x) didn't need this patch at all (I don't know this statement is still valid).

Ian>There are various settings that are relatively illogical and all need changing consistently. They also don't work universally.

I am with you, Ian.

It's interesting for me to see continuos complaints or issues about mtu size on overlay network tunnels. After providing my patch back then in June 2013, I expected some other issue of mtu size on tunnels, if any, would have been dealt with thereafter by someone else. Seems not though. Unfortunately, I have not been involved in other mtu size issues.

Ian, I would suggest you to provide a blue print to cope with all these mtu size issues in a consistent way.

But, still I am really curios if there is any real bug on tunnels right now regarding mtu size? Anyone who can please point out to me what bug report out there?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.