ovn-chassis charm does not expose pmd-cpu-mask

[Michał Ajduk]

ovn-chassis charm does not allow specifying pmd-cpu-mask, which leads to a situation where PMD thread is scheduled on randomply selected core.

In my case it is core 11 (counting from 0), which overlaps with the nova pinned CPU set:

  dpdk-dedicated-set: &dpdk-dedicated-set "1-19,41-59"
  dpdk-shared-set: &dpdk-shared-set "23-39,63-79"
  dpdk-vcpu-pin-set: &dpdk-vcpu-pin-set "1-19,23-39,41-59,63-79"

The charm could expose this setting.

WORKAROUND:
sudo -i ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x4000010000400001

[Liam Young (gnuoy)]

The charm should also ensure there is no intersection between the cores used for PMD and those assigned via dpdk-lcore-mask

[Billy Olsen (billy-olsen)]

This is a good change to have, however it is ultimately a feature (one which should be implemented) but is not applicable for Field High.

[Nikolay Vinogradov (nikolay.vinogradov)]

Sometimes we're asked to deploy OpenStack Queens with Neutron ML2/OVS, so adding neutron-openvswitch charm to the list of the affected projects.

[Nobuto Murata (nobuto)]

https://github.com/openstack-charmers/charm-layer-ovn/pull/41
https://review.opendev.org/c/x/charm-ovn-chassis/+/784557

[Nobuto Murata (nobuto)]

Attaching ~field-high again.

I initially thought it was good to have, but in the end it's a must-have actually. When PMD cores/threads are randomly chosen then those may collide with cores/threads used by a VM. If that happens VM traffic is actually affected and there is no traffic going through until the VM is rebooted to reply on the luckiness to have another set of cores allocated other than the PMD cores. It's not gonna be ~field-critical since there is a clear and manual workaround though. There were some ongoing reviews 6 months ago. It needs to be revived and taken over as necessary.

[Nobuto Murata (nobuto)]

A part of the networking traffic issue described in this bug was actually caused by random pmd cores/threads chosen just to give you some background.
https://bugs.launchpad.net/charm-nova-compute/+bug/1943863

[Bayani Carbone (bcarbone)]

As Nobuto mentioned in comment #4 [1], there was some work done to set pmd-cpu-mask via a config option in the ovn-chassis charm. In addition to the changes in layer-ovn, changes in charmhelpers were proposed to fully support this new config option [2].

However both changes have not been merged but in the meantime the following support code did get merged into charmhelpers [3]. The logic to set pmd-cpu-mask is similar in both charmhelpers changes i.e. assign the cores that follow the dpdk-lcore-mask however the implementation is different which makes the layer-ovn proposed changes [4] incompatible with the already merged charmhelpers support code [3].

The difference is that [4] allows the operator to provide a list of cores to set pmd-cpu-mask, while [3] supports giving the number of cores to be dedicated to PMD. This implementation is similar to the already existing dpdk-lcore-mask which is materialized as the dpdk-socket-cores config option in layer-ovn.

I'm not sure how to proceed from here, on one hand [3] is already merged but lacks the layer-ovn/openswitch charm code to take advantage of it. On the other, [2] and [4] fully implement the feature for ovn-chassis but are not merged. And finally, [4] is incompatible with [3].

[1] https://bugs.launchpad.net/charm-ovn-chassis/+bug/1905416/comments/4
[2] https://github.com/juju/charm-helpers/pull/594
[3] https://github.com/juju/charm-helpers/commit/3b7a86420629cc45ced7f78fa3706d99f514448e
[4] https://github.com/openstack-charmers/charm-layer-ovn/pull/41

[Dmitrii Shcherbakov (dmitriis)]

With the below code and low-level tests (kudos to Frode) I think this can be marked as fix-committed for charm-ovn-chassis.

https://github.com/openstack-charmers/charm-layer-ovn/pull/41

https://github.com/openstack-charmers/zaza-openstack-tests/pull/746
https://review.opendev.org/c/x/charm-ovn-chassis/+/839072