charm doesn't disable mlnx-switchdev-mode when enable-hardware-offload is false

[Giuseppe Petralia]

While working on a conversion of a node to support SRIOV, I removed ovn-chassis from the machine and added a new app ovn-chassis-sriov with this configuration:

  ovn-chassis-sriov:
    charm: cs:ovn-chassis-10
    options:
      bridge-interface-mappings: br-data:bond0
      enable-hardware-offload: false
      enable-sriov: true
      ovn-bridge-mappings: physnet1:br-data sriovfabric1:br-data sriovfabric2:br-data
      sriov-device-mappings: sriovfabric1:enp129s0f0 sriovfabric2:enp129s0f1
      sriov-numvfs: enp129s0f0:32 enp129s0f1:32

But sriov port creation was failing with error:

2021-04-13 12:54:02.468 8249 ERROR nova.compute.manager [instance: 3c029b35-8e28-46de-88fd-7b5b8ef0dedd] libvirt.libvirtError: Cannot set interface MAC/vlanid to fa:16:3e:67:8b:0d/3060 for ifname enp129s0f0 vf 31: Operation not supported

We found that the issue was with having mlnx-switchdev-mode service enabled.
Once disabled it and rebooted the node, sriov worked.

Charm should disable this service if enable-hardware-offload is false.