DPDK interface MTUs not set even when physical-network-mtus is set properly

[David Ames]

With neutron-api physical-network-mtus=9000

DPDK interfaces on neutron-openvswitch hosts only have 1500:

for x in `ovs-vsctl list-ifaces br-data | head -n2`; do ovs-vsctl get Interface $x mtu; done
1500
1500

This is proven out by running ping -s 8000 across compute nodes:

ping -Mdo -s 8000 172.20.0.4
PING 172.20.0.4 (172.20.0.4) 8000(8028) bytes of data.
^C
--- 172.20.0.4 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4094ms

[David Ames (thedac)]

Note it was believed that [0] would solve the problem. But this has proven not to be the case.

[0] https://github.com/juju/charm-helpers/commit/ac6c0f1819f17dae47d0d2928ab3f518db7ab11d

[David Ames (thedac)]

<cpaelzer> thedac: jhillman|PTO: I see you chatting about DPDK bonding + MTU - as you might have realized these areas are rather immature it probably isn't so much your setup, but the old DPDK version I know that brocade and (some other company I forgot) were hit often by that and the only real way is going forward to much newer versions e.g. 18.11.x that is in Disco and later (not sure what you are on) I heard that there is even a new bonding maintainer out of these upstream efforts and 19.11.x should be even better in that regard but that will be ubuntu 20.04 and due to that even less of an option

If you are running dpdk 18.11.x from a UCA let me know and I could ping you once I have an 18.11.2 version with plenty of stable fixes ready (some time in the next two weeks) maybe that helps a bit

TL;DR: don't doubt yourself, older DPDK is known to be crap for bond+MTU

[David Ames (thedac)]

Confirmed the Rocky UCA is deploying the same version of DPDK, 17.11.5-0~ubuntu18.04, and has the same behvior.

We can still test 18.11 per cpaelzer.

[David Ames (thedac)]

After the change ovs_use_veth=False we now have a tap device for the netns.

When we set charm MTU settings the tap device is set to jumbo frames (MTU 9000) we get truncated packets:

17:12:52.689179 IP truncated-ip - 464 bytes missing! (tos 0x0, ttl 64, id 38527, offset 0, flags [DF], proto TCP (6), length 1968)
    169.254.169.254.80 > 172.20.0.26.41838: Flags [P.], seq 0:1916, ack 1, win 219, options [nop,nop,TS val 2658401029 ecr 3658647933], length 1916: HTTP, length: 1916

Specifically with the HTTP request: http://169.254.169.254/openstack/2017-02-22/meta_data.json

It is not clear to me what is truncating the packets. In the netns the tap device states it has an MTU of 9000.

Manually setting the tap device MTU to 1500 seems to resolve this. Since we still have to manually setting the MTU of the dpdk-bond interfaces this is not that bad. But if the netns moves it will go back to 9000.