Fuel does not respect the physical network MTU when creating nova.conf in environments using VLAN segmentation

[Collin M.]

Detailed Bug Description:
Fuel always deploys nova.conf with a value of 65000 causing packet loss on Large UDP packets in environments with VLAN segmentation.

Instances fragment their packets to fit within the MTU of their internal interface, however they get reassembled by Conntrack and never get fragmented back down to a sane MTU because Conntrack's interfaces see an MTU of 65000.

This behavior is Optimal in situations where you're using a tunneled underlay since it avoids unnecessary fragmentation. However, when using VLAN encapsulation, it causes silent packet loss.

Steps to Reproduce:
  * Deploy a MOS environment using VLAN segmentation.
  * Create an instance
  * Attempt to send a large UDP or ICMP payload from this instance to any IP that will require traversing the underlay network. (I generally use another instance on a different compute node, or the instance's default gateway). Given an underlay not configured for Jumbo packets, this threashold will generally be anything greater than ~1450 or ~1460 Bytes.
  * Watch it fail

  My preferred method of generating these large packets is with ping: ping -c 2 -s 2000 192.168.111.1

Demonstrating a normal sized ping succeeding:
cirros$ ping -c 1 192.168.111.1
PING 192.168.111.1 (192.168.111.1): 56 data bytes
64 bytes from 192.168.111.1: seq=0 ttl=64 time=0.901 ms

--- 192.168.111.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.901/0.901/0.901 ms
$

root@node-4:~# tcpdump -nlei any icmp and host 192.168.111.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
21:18:52.166179 P fa:16:3e:ab:21:93 ethertype IPv4 (0x0800), length 100: 192.168.111.11 > 192.168.111.1: ICMP echo request, id 22347, seq 0, length 64
21:18:52.166243 Out fa:16:3e:ab:21:93 ethertype IPv4 (0x0800), length 100: 192.168.111.11 > 192.168.111.1: ICMP echo request, id 22347, seq 0, length 64
21:18:52.166245 P fa:16:3e:ab:21:93 ethertype IPv4 (0x0800), length 100: 192.168.111.11 > 192.168.111.1: ICMP echo request, id 22347, seq 0, length 64
21:18:52.166381 P fa:16:3e:ab:21:93 ethertype 802.1Q (0x8100), length 104: vlan 1030, p 0, ethertype IPv4, 192.168.111.11 > 192.168.111.1: ICMP echo request, id 22347, seq 0, length 64
21:18:52.166726 P fa:16:3e:a1:5c:5c ethertype 802.1Q (0x8100), length 104: vlan 1030, p 0, ethertype IPv4, 192.168.111.1 > 192.168.111.11: ICMP echo reply, id 22347, seq 0, length 64
21:18:52.166752 Out fa:16:3e:a1:5c:5c ethertype 802.1Q (0x8100), length 104: vlan 1030, p 0, ethertype IPv4, 192.168.111.1 > 192.168.111.11: ICMP echo reply, id 22347, seq 0, length 64
21:18:52.166758 Out fa:16:3e:a1:5c:5c ethertype IPv4 (0x0800), length 100: 192.168.111.1 > 192.168.111.11: ICMP echo reply, id 22347, seq 0, length 64
21:18:52.166766 P fa:16:3e:a1:5c:5c ethertype IPv4 (0x0800), length 100: 192.168.111.1 > 192.168.111.11: ICMP echo reply, id 22347, seq 0, length 64
21:18:52.166785 Out fa:16:3e:a1:5c:5c ethertype IPv4 (0x0800), length 100: 192.168.111.1 > 192.168.111.11: ICMP echo reply, id 22347, seq 0, length 64

Demonstrating a large ping getting fragmented, reassembled and never re-fragmented to leave the compute node at the correct MTU:
cirros$ ping -c 1 192.168.111.1 -s 2000
PING 192.168.111.1 (192.168.111.1): 2000 data bytes

--- 192.168.111.1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

root@node-4:~# tcpdump -nlei any icmp and host 192.168.111.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
21:19:07.397533 P fa:16:3e:ab:21:93 ethertype IPv4 (0x0800), length 1516: 192.168.111.11 > 192.168.111.1: ICMP echo request, id 22603, seq 0, length 1480
21:19:07.397576 P fa:16:3e:ab:21:93 ethertype IPv4 (0x0800), length 564: 192.168.111.11 > 192.168.111.1: ip-proto-1
21:19:07.397625 Out fa:16:3e:ab:21:93 ethertype IPv4 (0x0800), length 2044: 192.168.111.11 > 192.168.111.1: ICMP echo request, id 22603, seq 0, length 2008
21:19:07.397628 P fa:16:3e:ab:21:93 ethertype IPv4 (0x0800), length 2044: 192.168.111.11 > 192.168.111.1: ICMP echo request, id 22603, seq 0, length 2008
21:19:07.397628 P fa:16:3e:ab:21:93 ethertype 802.1Q (0x8100), length 2048: vlan 1030, p 0, ethertype IPv4, 192.168.111.11 > 192.168.111.1: ICMP echo request, id 22603, seq 0, length 2008

Workaround:
Update network_device_mtu in /etc/nova/nova.conf to reflect the MTU of the underlay network

Impact:
Large UDP Traffic gets dropped impacting services such as Video Streaming, VPN Endpoints, or VoIP traffic.

Environment:
I've confirmed this behavior in environments deployed by Fuel 6.1, Fuel 7.0, MOS 8.0 with VLAN segmentation.

I've also identified what I believe to be the code responsible for this issue:
https://github.com/openstack/fuel-library/blob/master/deployment/puppet/openstack_tasks/manifests/roles/compute.pp#L59