Openvswitch VLAN stripping issue with tunneling

[Maël Kimmerlin]

Openvswitch VLAN tag stripping and MTU issue

When L2 population is not used in an environment using Openvswitch as ML2 or when the learned rules are matching, the vlan tag used internally by Neutron is not stripped. Hence, for VXLAN the overhead of the tunneling is higher than the MTU reduction on the virtual networks because the VLAN tag is not stripped, thus causing MTU issues.

In my setup, I have several OpenStack clouds (Newton) deployed using Fuel, with VXLAN segmentation and using Openvswitch. It runs on Ubuntu 16.04. Some machines in the tenants virtual networks act as bridges and thus L2 population is not sufficient, the learning feature of br-tun is required. The deployments are the most basic that can be performed with Fuel 10 (no additionnal services).

The overhead of VXLAN is 50 Bytes, if the original ethernet frame does not have a VLAN tag. However, if the ethernet frame has a vlan tag, the overhead is 54 Bytes. When setting up the virtual network MTU, Neutron assumes that there is no vlan tag. However, Neutron uses internally vlan tags to isolate the networks in br-int and br-tun. When using L2 populations, the rules set in br-tun strip the vlan tag before tunneling, hence everything work properly. But, when L2 population is not used or its rules not hit and the learning part takes place, the learned rules do not strip the vlan, they only zero it, hence the overhead is 54 Bytes and the communication is broken.

The following learning rule in br-tun installs flows that zero the vlan tag and do not remove it.

in table 10:
#table=10,priority=0,actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xa9e495b5c54cf90c,OXM_OF_VLAN_VID[],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->OXM_OF_VLAN_VID[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:1

resulting in flows in table 20 like this :
#table=20,priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:b6:53:e2 actions=load:0->OXM_OF_VLAN_VID[],load:0x18->NXM_NX_TUN_ID[],output:2
This flow does not remove the vlan tag. When using L2 population, some flows with higher priority are inserted, that do strip the vlan tag correctly. However, the learned flows are used if the L2 populations flow do not match.

Expected output : traffic without vlan tag tunneled in VXLAN with a 50 Bytes overhead
Actual output : traffic with a vlan tag (0) tunneled in VXLAN with a 54 Bytes overhead

The issue does not happen for GRE as the 4 additionnal bytes are still fitting in the 50 Bytes MTU reduction on the tenant network

[Maël Kimmerlin (mael-kimmerlin)]

A temporary workaround could be to decrease all MTUs to 1446 instead of 1450 by seting the path_mtu variable to 1496 instead of 1500 in /etc/neutron/plugins/ml2/ml2_conf.ini . However, to really fix the issue, the following could be done. The VLAN tag would be copied to a registry and stripped from the packet in table 2 of br-tun. The following matches (table 20 and 22) would be upon the registry instead of the vlan tags.

In terms of rules, the following changes are possible to fix the issue (output from ovs-ofctl dump-flows)

Categorizing the traffic into unicast and broadcast : Modify the rules in table 2 from
#table=2, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=goto_table:20
#table=2, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=goto_table:22
to
#table=2, priority=0,vlan_tci=0x1000/0x1000,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=move:OXM_OF_VLAN_VID[]->NXM_NX_REG0[0..11],pop_vlan,goto_table:20
#table=2, priority=1,vlan_tci=0x1000/0x1000,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=move:OXM_OF_VLAN_VID[]->NXM_NX_REG0[0..11],pop_vlan,goto_table:22
This will copy the vlan tag to the registry and remove the vlan tag

Mac learning rules : Modify the rules in table 10 from
#table=10, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=<UNSET>,OXM_OF_VLAN_VID[],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->OXM_OF_VLAN_VID[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:1
to:
#table=10, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=<UNSET>,NXM_NX_REG0[0..11]=OXM_OF_VLAN_VID[],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:1
This will match upon the registry instead of the vlan tag (that has been stripped)

Multicast output : finally modify the rules in table 22 following this patern
#table=22, priority=1,dl_vlan=<X> actions=pop_vlan,set_field:0x<Y>->tun_id,output:<Z>,output:<W>
to:
#table=22,priority=2,reg0=0x<X>00000/0xfff00000 actions=set_field:0x<Y>->tun_id,output:<Z>,output:<W>
This will match upon the registry instead of the vlan tag and will not strip it since it has been removed already in table 2.

[Maël Kimmerlin (mael-kimmerlin)]

This patch applies the proposed solution to the ovs-ofctl based implementation for br-tun : neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/br_tun.py

[Kevin Benton (kevinbenton)]

What version is this? It sounds like a dup of https://bugs.launchpad.net/neutron/+bug/1622017

[Ihar Hrachyshka (ihar-hrachyshka)]

Seems similar to https://bugs.launchpad.net/neutron/+bug/1622017 but for CLI interface that is no longer default since Mitaka+? I guess we can have a look. It would be cool to see Iwamoto chiming in here since he did the native fix.

[Ihar Hrachyshka (ihar-hrachyshka)]

Also, please attach neutron-server log, we may want to see which config option values you use.

[Ihar Hrachyshka (ihar-hrachyshka)]

And neutron net-show for the borked network.

[IWAMOTO Toshihiro (iwamoto)]

loading 0 to NXM_OF_VLAN_TCI (instead of vlan_vid) should remove the VLAN tag, and the ofctl of_interface is using NXM_OF_VLAN_TCI from a very long ago. I guess you are using the native of_interface.

The default value of of_interface has changed to the native in Newton. The fix for bug/1622017 is also done later, before cutting stable/newton. The time window that cause this failure is 8f294d94..82b180f3

[Maël Kimmerlin (mael-kimmerlin)]

This is indeed duplicate of https://bugs.launchpad.net/neutron/+bug/1622017. Using Neutron from Github stable/newton fixed the issue. I was using the Ubuntu package that has this bug. Thank you for your explanations