Can't ping router with packet size greater than 1476 when ovs datapath_type set to netdev

[Oleksiy Butenko]

Setup:
  Openstack origin/stable/wallaby deployed by devstack on single node
  [stack@localhost devstack]$ cat /etc/centos-release
  CentOS Linux release 8.4.2105
  [stack@localhost devstack]$ uname -a
  Linux localhost.localdomain 4.18.0-305.3.1.el8.x86_64 #1 SMP Tue Jun 1 16:14:33 UTC 2021 x86_64
  x86_64 x86_64 GNU/Linux
  [stack@localhost devstack]$

  [stack@localhost devstack]$ git status
  HEAD detached at origin/stable/wallaby

  ovs version (builded with DPDK)
  [stack@localhost devstack]$ ovs-vswitchd --version
  ovs-vswitchd (Open vSwitch) 2.13.4
  DPDK 19.11.7

  [stack@localhost devstack]$ cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep datapath_type
  datapath_type = netdev

  [stack@localhost devstack]$ cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep mtu
  path_mtu = 9000
  physical_network_mtus = 9000
  [stack@localhost devstack]$ cat /etc/neutron/neutron.conf |grep mtu
  global_physnet_mtu = 9000

Steps:
  1. Setup hugepages, update flavor (hw:mem_page_size='large', hw_rng:allowed='True')
  2. boot vm and try ping router and vice versa

br-int:

    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        datapath_type: netdev
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port vhua5beaaae-c8
            tag: 1
            Interface vhua5beaaae-c8
                type: dpdkvhostuserclient
                options: {vhost-server-path="/var/run/openvswitch/vhua5beaaae-c8"}
        Port qr-c48b27ee-0f
            tag: 1
            Interface qr-c48b27ee-0f
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port tap3b960ab3-6a
            tag: 1
            Interface tap3b960ab3-6a
                type: internal

From VM:
[root@test-mtu ~]# ip a |grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc fq_codel state UP group default qlen 1000
[root@test-mtu ~]#
[root@test-mtu ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.275 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms

--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1035ms
rtt min/avg/max/mdev = 0.167/0.221/0.275/0.054 ms
[root@test-mtu ~]# ping 192.168.1.1 -s 2000
PING 192.168.1.1 (192.168.1.1) 2000(2028) bytes of data.

--- 192.168.1.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3086ms

[root@test-mtu ~]# ping 192.168.1.1 -c 5
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.139 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.154 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.144 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.153 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=0.146 ms

--- 192.168.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 0.139/0.147/0.154/0.005 ms
[root@test-mtu ~]# ping 192.168.1.1 -c 5 -s 2000
PING 192.168.1.1 (192.168.1.1) 2000(2028) bytes of data.

--- 192.168.1.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4084ms

[root@test-mtu ~]#

In ns:
[root@localhost ~]# ip a |grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
29: qr-c48b27ee-0f: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 8950 qdisc fq_codel state UNKNOWN group default qlen 1000
[root@localhost ~]#
# default packet size:
10:44:41.276140 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 1, length 64
10:44:41.276176 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 1, length 64
10:44:42.323564 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 2, length 64
10:44:42.323594 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 2, length 64
10:44:43.325048 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 3, length 64
10:44:43.325075 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 3, length 64
10:44:44.326474 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 4, length 64
10:44:44.326501 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 4, length 64
10:44:45.327838 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 7, seq 5, length 64
10:44:45.327865 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 7, seq 5, length 64

# packet size 2000

[root@localhost ~]# tcpdump -i qr-c48b27ee-0f icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qr-c48b27ee-0f, link-type EN10MB (Ethernet), capture size 262144 bytes
10:46:00.934011 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 1, length 2008
10:46:00.934046 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 1, length 2008
10:46:01.941013 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 2, length 2008
10:46:01.941051 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 2, length 2008
10:46:02.965045 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 3, length 2008
10:46:02.965072 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 3, length 2008
10:46:03.989043 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 4, length 2008
10:46:03.989083 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 4, length 2008
10:46:05.013079 IP 192.168.1.176 > localhost.localdomain: ICMP echo request, id 9, seq 5, length 2008
10:46:05.013108 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id 9, seq 5, length 2008

From ns:

[root@localhost ~]# ping 192.168.1.176 -c 2
PING 192.168.1.176 (192.168.1.176) 56(84) bytes of data.
64 bytes from 192.168.1.176: icmp_seq=1 ttl=64 time=0.193 ms
64 bytes from 192.168.1.176: icmp_seq=2 ttl=64 time=0.163 ms
--- 192.168.1.176 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1038ms
rtt min/avg/max/mdev = 0.163/0.178/0.193/0.015 ms
[root@localhost ~]# ping 192.168.1.176 -c 2 -s 2000
PING 192.168.1.176 (192.168.1.176) 2000(2028) bytes of data.

--- 192.168.1.176 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1060ms
[root@localhost ~]#

in vm:
08:51:37.228218 IP truncated-ip - 524 bytes missing! host-192-168-1-1.openstacklocal > host-192-168-1-176.openstacklocal: ICMP echo request, id 27328, seq 1, length 2008
08:51:38.245796 IP truncated-ip - 524 bytes missing! host-192-168-1-1.openstacklocal > host-192-168-1-176.openstacklocal: ICMP echo request, id 27328, seq 2, length 2008

The same issue if I try to ping dhcp server

If I set ovs datapath_type=system and don't use dpdkvhostuserclient - all works fine

[Lajos Katona (lajos-katona)]

Hi, thanks for the detailed bug description.
I checked and it works on master with datapath_type = system.
Tomorrow I try to build dpdk in my env.
In the meantime could you please check if ping with jumboframes works for other directions than router?

[LIU Yulong (dragon889)]

"ovs datapath_type netdev" should only be used for VM, neutron router related virtual network devices are not compatible with it, [1] has those limitations. The only way you can run L3 routers with VMs (using DPDK) is to run l3-agents and ovs-agents in dedicated nodes with data patch type system, while VMs in hosts with netdev.

[1] https://docs.openstack.org/neutron/latest/admin/config-ovs-dpdk.html#known-limitations