dpdk-socket-cores misleading description

[Michał Ajduk]

As per the description of the config option:
dpdk-socket-cores:
(int) Number of cores to allocate to DPDK per NUMA socket in deployed systems. . Only used when DPDK is enabled.

System has 2 NUMA nodes:
NUMA node0 CPU(s): 0-19,40-59
NUMA node1 CPU(s): 20-39,60-79

Only one thread is allocated to PMD driver instead of expected 2 - consult attached htop screenshot.

[Michał Ajduk (majduk)]

ubuntu@inf1az1cz202904rz:~/deploy$ juju config ovn-chassis-dpdk dpdk-socket-cores
1

This renders following config on the machine:
root@cmp3az1cz20300kvd:~# ovs-vsctl get Open_vSwitch . other_config:dpdk-lcore-mask
"0x100001"

dpdk-lcore-mask
Specifies the CPU cores on which dpdk lcore threads should be spawned and expects hex string (eg ‘0x123’).

root@cmp3az1cz20300kvd:~# ovs-vsctl get Open_vSwitch . other_config:pmd-cpu-mask
ovs-vsctl: no key "pmd-cpu-mask" in Open_vSwitch record "." column other_config

That contains 2 CPUs:
1
1048576 mod 80 = 16

I'm not usre how this maps to PMD core 12 that is actually in use

[James Page (james-page)]

Currently I can't see code that sets pmd-cpu-mask in the ovn or ovs charm codebases - only dpdk-lcore-mask is set.

[James Page (james-page)]

So we get whatever the default DPDK behaviour is if pmd-cpu-mask is not set.

[Michał Ajduk (majduk)]

Attaching output of ls -1 /sys/devices/system/node/node*

Please note that with current setting, the CPU is use, core 11 (counting from 0) overlaps with the nova pinned CPU set:

  dpdk-dedicated-set: &dpdk-dedicated-set "1-19,41-59"
  dpdk-shared-set: &dpdk-shared-set "23-39,63-79"
  dpdk-vcpu-pin-set: &dpdk-vcpu-pin-set "1-19,23-39,41-59,63-79"

[Michał Ajduk (majduk)]

juju config ovn-chassis-dpdk dpdk-socket-cores=2

That did not change anything. PMD is still one thread, core 12 (if 11 counting from 0)

[Michał Ajduk (majduk)]

WORKAROUND:

sudo -i ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=<hex-mask>

This moves PMD thread to selected cores

[Billy Olsen (billy-olsen)]

I agree this is probably a bit misleading. The dpdk-socket-cores option sets the dpdk-lcore-mask option on Openvswitch, which determines where the non-datapath ovs-dpdk threads (i.e. handler, revalidator) run.

The option does not exist today to set the pmd-cpu-mask, which are the cores used for datapath packet processing.

A new option will need to be introduced in order to be able to set the pmd-cpu-mask and the dpdk-socket-cores option will need to be clarified in the documentation.

[Frode Nordahl (fnordahl)]

Adding the neutron-openvswitch charm to this bug as they share the code and behave exactly the same way.

My guess is that making this configurable was missed when upstream Open vSwitch moved from a command line options based configuration workflow back in the v2.6 time frame.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/804787

[Chris MacNaughton (chris.macnaughton)]

This description was updated in the OVN charm early enough that it's currently fixed in the latest stable charms, and I've proposed an update to the OVS charm with a matching update. The addition of the new feature (pmd-cpu-mask) is being tracked in bug https://bugs.launchpad.net/charm-ovn-chassis/+bug/1905416.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/804787
Committed: https://opendev.org/openstack/charm-neutron-openvswitch/commit/b9c66d81b9791692e4e89eff02eff9cb1f7b7e09
Submitter: "Zuul (22348)"
Branch: master

commit b9c66d81b9791692e4e89eff02eff9cb1f7b7e09
Author: Chris MacNaughton <email address hidden>
Date: Mon Aug 16 11:54:36 2021 -0500

    Update description of dpdk-socket-cores

    Closes-bug: #1905284
    Change-Id: I8eb03cda1bff845b39b3fbab3a2e344889453993