DPDK fails to add network device

[Michał Ajduk]

# Problem description
Instance booted with DPDK interface is not reachable over physnet, OVS reports:
        Port vhu0f17b23d-50
            Interface vhu0f17b23d-50
                type: dpdkvhostuserclient
                options: {vhost-server-path="/run/libvirt-vhost-user/vhu0f17b23d-50"}
                error: "could not add network device vhu0f17b23d-50 to ofproto (Invalid argument)"

The machine should have 2 interfaces (within different physnets):
- one over regular br-data
- one over DPDK
- both should be attached to DPDK enabled OVS instance but both fail with "could not add network device vhu0f17b23d-50 to ofproto (Invalid argument)"

# Environment
Openstack USSURI + OVN
ovn-chassis version: cs:~openstack-charmers-next/ovn-chassis-40
CIS hardened system.
AppArmor profile enabled.

charm configuration:
settings:
  bridge-interface-mappings:
    value: br-data:bond1 br-dpdk:dpdk-bond0
  debug:
    value: false
  dpdk-bond-config:
    value: :balance-slb:off:fast
  dpdk-bond-mappings:
    value: dpdk-bond0:d4:f5:ef:06:59:e0 dpdk-bond0:d4:f5:ef:05:a9:20 dpdk-bond0:d4:f5:ef:05:e2:c0
  dpdk-driver:
    value: vfio-pci
  dpdk-socket-cores:
    value: 1
  dpdk-socket-memory:
    value: 1024
  enable-dpdk:
    value: true
  enable-hardware-offload:
    value: false
  enable-sriov:
    value: false
  openstack-metadata-workers:
    value: 2
  ovn-bridge-mappings:
    value: dcfabric:br-data dpdkfabric:br-dpdk

ovs-vsctl show output:
ddb4da73-4b57-4ebe-8586-da1a6da4c730
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        fail_mode: secure
        datapath_type: netdev
        Port br-int
            Interface br-int
                type: internal
        Port ovn-cmp4az-0
            Interface ovn-cmp4az-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.8"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
        Port ovn-cmp2az-0
            Interface ovn-cmp2az-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.1"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
        Port vhueeb21868-58
            Interface vhueeb21868-58
                type: dpdkvhostuserclient
                options: {vhost-server-path="/run/libvirt-vhost-user/vhueeb21868-58"}
                error: "could not add network device vhueeb21868-58 to ofproto (Invalid argument)"
        Port patch-br-int-to-provnet-f18f1922-2dad-4cd4-8526-5140e4d3d3a2
            Interface patch-br-int-to-provnet-f18f1922-2dad-4cd4-8526-5140e4d3d3a2
                type: patch
                options: {peer=patch-provnet-f18f1922-2dad-4cd4-8526-5140e4d3d3a2-to-br-int}
        Port ovn-cmp3az-1
            Interface ovn-cmp3az-1
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.7"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
        Port ovn-cmp4az-2
            Interface ovn-cmp4az-2
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.5"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
        Port ovn-cmp4az-1
            Interface ovn-cmp4az-1
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.4"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
        Port ovn-cmp2az-2
            Interface ovn-cmp2az-2
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.11"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
        Port ovn-juju-4-0
            Interface ovn-juju-4-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.245.244"}
        Port ovn-cmp1az-0
            Interface ovn-cmp1az-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.10"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
        Port ovn-cmp3az-0
            Interface ovn-cmp3az-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.3"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
        Port vhu0f17b23d-50
            Interface vhu0f17b23d-50
                type: dpdkvhostuserclient
                options: {vhost-server-path="/run/libvirt-vhost-user/vhu0f17b23d-50"}
                error: "could not add network device vhu0f17b23d-50 to ofproto (Invalid argument)"
        Port ovn-juju-4-2
            Interface ovn-juju-4-2
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.245.47"}
        Port ovn-cmp2az-1
            Interface ovn-cmp2az-1
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.12"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
        Port ovn-cmp1az-2 [0/110]
            Interface ovn-cmp1az-2
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.6"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
        Port ovn-juju-4-1
            Interface ovn-juju-4-1
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.245.41"}
        Port ovn-cmp1az-1
            Interface ovn-cmp1az-1
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.216.246.2"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
    Bridge br-data
        fail_mode: standalone
        datapath_type: netdev
        Port bond1
            Interface bond1
                type: system
        Port patch-provnet-f18f1922-2dad-4cd4-8526-5140e4d3d3a2-to-br-int
            Interface patch-provnet-f18f1922-2dad-4cd4-8526-5140e4d3d3a2-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-f18f1922-2dad-4cd4-8526-5140e4d3d3a2}
        Port br-data
            Interface br-data
                type: internal
    Bridge br-dpdk
        fail_mode: standalone
        datapath_type: netdev
        Port br-dpdk
            Interface br-dpdk
                type: internal
        Port dpdk-bond0
            Interface dpdk-abd308d
                type: dpdk
                options: {dpdk-devargs="0000:13:00.1"}
            Interface dpdk-f92ad9f
                type: dpdk
                options: {dpdk-devargs="0000:13:00.0"}
    ovs_version: "2.13.1"

Hugepages enabled:
root@cmp3az1cz20300kvd:~# grep -sri HugePages_ /proc/meminfo
HugePages_Total: 64
HugePages_Free: 60
HugePages_Rsvd: 0
HugePages_Surp: 0

Kernel:
root@cmp3az1cz20300kvd:~# uname -a
Linux cmp3az1cz20300kvd 5.4.0-53-generic #59~18.04.1-Ubuntu SMP Wed Oct 21 12:14:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

root@cmp3az1cz20300kvd:~# dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:13:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=i40e
0000:13:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=i40e

Network devices using kernel driver
===================================
0000:12:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=ens1f0 drv=i40e unused=vfio-pci
0000:12:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=ens1f1 drv=i40e unused=vfio-pci
0000:5d:00.0 'I350 Gigabit Network Connection 1521' if=eno5 drv=igb unused=vfio-pci
0000:5d:00.1 'I350 Gigabit Network Connection 1521' if=eno6 drv=igb unused=vfio-pci
0000:5d:00.2 'I350 Gigabit Network Connection 1521' if=eno7 drv=igb unused=vfio-pci
0000:5d:00.3 'I350 Gigabit Network Connection 1521' if=eno8 drv=igb unused=vfio-pci
0000:af:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=ens4f0 drv=i40e unused=vfio-pci
0000:af:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=ens4f1 drv=i40e unused=vfio-pci

[Michał Ajduk (majduk)]

Additional Openvswitch logs:
2020-11-18T12:57:36.684Z|00091|dpdk|INFO|VHOST_CONFIG: virtio is now ready for processing.
2020-11-18T12:57:36.729Z|00193|netdev_dpdk|ERR|Failed to create mempool "ovs31fcb6a500093260131072" with a request of 131072 mbufs
2020-11-18T12:57:36.729Z|00194|netdev_dpdk|ERR|Failed to create memory pool for netdev vhueeb21868-58, with MTU 8900 on socket 0: Invalid argument
2020-11-18T12:57:36.729Z|00195|dpif_netdev|ERR|Failed to set interface vhueeb21868-58 new configuration
2020-11-18T12:57:36.729Z|00196|dpif_netdev|INFO|Core 11 on numa node 0 assigned port 'dpdk-abd308d' rx queue 0 (measured processing cycles 0).
2020-11-18T12:57:36.729Z|00197|dpif_netdev|INFO|Core 11 on numa node 0 assigned port 'dpdk-f92ad9f' rx queue 0 (measured processing cycles 0).
2020-11-18T12:57:36.729Z|00198|dpif|WARN|netdev@ovs-netdev: failed to add vhueeb21868-58 as port: Invalid argument
2020-11-18T12:57:36.729Z|00199|bridge|WARN|could not add network device vhueeb21868-58 to ofproto (Invalid argument)
2020-11-18T12:57:36.730Z|00092|netdev_dpdk|INFO|vHost Device '/run/libvirt-vhost-user/vhueeb21868-58' has been added on numa node 0
2020-11-18T12:57:36.730Z|00200|dpdk|INFO|VHOST_CONFIG: free connfd = 83 for device '/run/libvirt-vhost-user/vhueeb21868-58'
2020-11-18T12:57:36.730Z|00201|netdev_dpdk|INFO|vHost Device '/run/libvirt-vhost-user/vhueeb21868-58' not found
2020-11-18T12:57:36.730Z|00202|dpdk|INFO|VHOST_CONFIG: Linear buffers requested without external buffers, disabling host segmentation offloading support
2020-11-18T12:57:36.730Z|00203|dpdk|INFO|VHOST_CONFIG: vhost-user client: socket created, fd: 83
2020-11-18T12:57:36.730Z|00204|netdev_dpdk|INFO|vHost User device 'vhu0f17b23d-50' created in 'client' mode, using client socket '/run/libvirt-vhost-user/vhu0f17b23d-5
0'

2020-11-18T12:57:36.814Z|00206|netdev_dpdk|ERR|Failed to create mempool "ovs6665799f00093260131072" with a request of 131072 mbufs
2020-11-18T12:57:36.814Z|00207|netdev_dpdk|ERR|Failed to create memory pool for netdev vhu0f17b23d-50, with MTU 8900 on socket 0: Invalid argument
2020-11-18T12:57:36.814Z|00208|dpif_netdev|ERR|Failed to set interface vhu0f17b23d-50 new configuration
2020-11-18T12:57:36.814Z|00209|dpif_netdev|INFO|Core 11 on numa node 0 assigned port 'dpdk-abd308d' rx queue 0 (measured processing cycles 0).
2020-11-18T12:57:36.814Z|00210|dpif_netdev|INFO|Core 11 on numa node 0 assigned port 'dpdk-f92ad9f' rx queue 0 (measured processing cycles 0).
2020-11-18T12:57:36.814Z|00211|dpif|WARN|netdev@ovs-netdev: failed to add vhu0f17b23d-50 as port: Invalid argument
2020-11-18T12:57:36.814Z|00212|bridge|WARN|could not add network device vhu0f17b23d-50 to ofproto (Invalid argument)
2020-11-18T12:57:36.814Z|00122|netdev_dpdk|INFO|vHost Device '/run/libvirt-vhost-user/vhu0f17b23d-50' has been added on numa node 0
2020-11-18T12:57:36.814Z|00213|dpdk|INFO|VHOST_CONFIG: free connfd = 83 for device '/run/libvirt-vhost-user/vhu0f17b23d-50'
2020-11-18T12:57:36.814Z|00214|netdev_dpdk|INFO|vHost Device '/run/libvirt-vhost-user/vhu0f17b23d-50' not found
2020-11-18T12:57:36.815Z|00215|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.13.1
2020-11-18T12:57:44.819Z|00216|memory|INFO|...

[Andrew McLeod (admcleod)]

I have seen what may be a related issue here:

https://<email address hidden>/msg27408.html

The relevant section being:

confirm whether all bridges are being set as system type datapaths (check in ovsdb) - could you do that and report back?

[Liam Young (gnuoy)]

The issue appears to be around the allocation of memory. Reading some discussion here *1 it looks like dpdk-socket-memory value of 1024 is too low. Can you try increasing it to 2048 ?

*1 https://<email address hidden>/msg06440.html

[Michał Ajduk (majduk)]

Root cause is:
Example 4¶
MTU = 9000 Bytes
Number of mbufs = 262144
Mbuf size = 10176 Bytes
Memory required = 262144 * 10176 = 2667 MB

For MTU 9000, 2x2 1G hugepages are required and the default example shows just one hugepage

[Michał Ajduk (majduk)]

The solution is to update the setting to: dpdk-socket-memory=2048

That applies 2x2 1G hugepages to meet the memory requirements calculated in comment #4

[Michał Ajduk (majduk)]

Changed bug to Opinion.

I believe that:
1) charm description of the setting could point to OVS memory cacluclator at https://docs.openvswitch.org/en/latest/topics/dpdk/memory/#example-4
2) Openstack deployment guide at https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-ovn.html could also say word or two on this topic as it is rather terse now:
----
By default, the charm will configure Open vSwitch/DPDK to consume one processor core + 1G of RAM from each NUMA node on the unit being deployed. This can be tuned using the dpdk-socket-memory and dpdk-socket-cores configuration options. The userspace kernel driver can be configured using the dpdk-driver configuration option. See config.yaml for more details.
----
That could point to https://docs.openvswitch.org/en/latest/topics/dpdk/memory/ for dpdk-socket-memory size tuning.

[Liam Young (gnuoy)]

I believe this bug can be closed when the following land:

https://review.opendev.org/c/openstack/charm-deployment-guide/+/763951
https://github.com/openstack-charmers/charm-layer-ovn/pull/29

[Michael Skalka (mskalka)]

Dropping the crit subscription as we have a workaround.