libvirt on the host does not work when *not* using iptables firewall driver

[Will Szumski]

I'm trying to use the libvirt on the host feature (https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#host-vs-containerised-libvirt). I did not have any issue when using OVS, but when doing a deployment with OVN, instance creation is failing with:

  2022-11-01 15:30:34.997 7 ERROR nova.compute.manager [instance: 2253d5f3-110a-40fe-8eca-7b3b8a21a5ab] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1385, in createWithFlags
  2022-11-01 15:30:34.997 7 ERROR nova.compute.manager [instance: 2253d5f3-110a-40fe-8eca-7b3b8a21a5ab] raise libvirtError('virDomainCreateWithFlags() failed')
  2022-11-01 15:30:34.997 7 ERROR nova.compute.manager [instance: 2253d5f3-110a-40fe-8eca-7b3b8a21a5ab] libvirt.libvirtError: internal error: Unable to add port tap3a55248e-ef to OVS bridge br-int
  2022-11-01 15:30:34.997 7 ERROR nova.compute.manager [instance: 2253d5f3-110a-40fe-8eca-7b3b8a21a5ab]

It seems libvirt is looking for ovs-vsctl on the host. The following solution seemed to workaround the issue:

  (venv-openstack) [cloud-user@kayobe-aio-will ~]$ cat /usr/bin/ovs-vsctl
  #!/bin/bash

  docker exec openvswitch_vswitchd ovs-vsctl "$@"

I noticed that the libvirt xml interface definition looks like:

    <interface type='bridge'>
      <mac address='fa:16:3e:b2:17:71'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='52b5be5b-54ea-4917-aa50-814b61061cbf'/>
      </virtualport>
      <target dev='tap52b5be5b-54'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <mtu size='1500'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

compared to an OVS installation:

    <interface type='bridge'>
      <mac address='fa:16:3e:0b:cf:48'/>
      <source bridge='qbr20cdb401-e5'/>
      <target dev='tap20cdb401-e5'/>
      <model type='virtio'/>
      <driver name='vhost' queues='4'/>
      <mtu size='1450'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

I believe the difference relates to the following option that we configure on openvswitch agent:

  ml2_conf.ini:firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

From https://docs.openstack.org/neutron/latest/admin/config-ovsfwdriver.html:

   Historically, Open vSwitch (OVS) could not interact directly with iptables to implement security groups. Thus, the OVS agent and Compute service use a Linux bridge between each instance (VM) and the OVS integration bridge br-int to implement security groups.

I think this means that this would also affect OVS deployments that changed the firewall driver to openvswitch.

[Will Szumski (willjs)]

Just realised this is more of a kayobe bug as kolla-ansible doesn't actually do the libvirt installation.

[Will Szumski (willjs)]

Re-opened as kayobe bug: https://storyboard.openstack.org/#!/story/2010391

[Will Szumski (willjs)]

Well I suppose its not that clear cut as we could just use the workaround:

```
  (venv-openstack) [cloud-user@kayobe-aio-will ~]$ cat /usr/bin/ovs-vsctl
  #!/bin/bash

  docker exec openvswitch_vswitchd ovs-vsctl "$@"
```

if we are deploying openvswitch in a container.

[Mark Goddard (mgoddard)]

I think it's fair to have a kolla-ansible bug for this. Probably there will be some changes necessary, or at least documentation changes to add more prerequisites/conditions for libvirt on the host.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/863543

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/863543
Committed: https://opendev.org/openstack/kolla-ansible/commit/a82443481e17edecda728f8698f8b401ec910546
Submitter: "Zuul (22348)"
Branch: master

commit a82443481e17edecda728f8698f8b401ec910546
Author: Will Szumski <email address hidden>
Date: Thu Nov 3 15:53:32 2022 +0000

    Adds a wrapper script to run ovs-vsctl commands in the container

    Libvirt needs to be able to plug ports into openvswitch bridges.
    It does this using the ovs-vsctl command, which it searches for
    in $PATH[1, 2]. This change will optionally install a wrapper
    script that executes the ovs-vsctl commands in the context of the
    openvswitchd container. This is useful when running libvirt on the
    host whilst still running openvswitch in a container. The advantage
    of this method over install the packages on the host is that it
    ensures client compatability with the daemon. The default is set
    to false as the wrapper could overwrite ovs-vsctl installed on the
    host.

    [1] https://github.com/libvirt/libvirt/blob/ee51ab86c2e52b6ff1b17a4c7ad11439fd610c9e/src/util/virnetdevopenvswitch.c#L59
    [2] https://github.com/libvirt/libvirt/blob/a89b17c2a75cfbaeb9e430f88e0f8a7475eb4f54/docs/kbase/internals/command.rst#id3

    Closes-Bug: #1995409
    Change-Id: Iaa6bfb012ae847f5f6aa0a1fc1c27970ac265f93

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/865169

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 15.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/865169
Committed: https://opendev.org/openstack/kolla-ansible/commit/37640c6ba9b695975853031a8c6de880b39112eb
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 37640c6ba9b695975853031a8c6de880b39112eb
Author: Will Szumski <email address hidden>
Date: Thu Nov 3 15:53:32 2022 +0000

    Adds a wrapper script to run ovs-vsctl commands in the container

    Libvirt needs to be able to plug ports into openvswitch bridges.
    It does this using the ovs-vsctl command, which it searches for
    in $PATH[1, 2]. This change will optionally install a wrapper
    script that executes the ovs-vsctl commands in the context of the
    openvswitchd container. This is useful when running libvirt on the
    host whilst still running openvswitch in a container. The advantage
    of this method over install the packages on the host is that it
    ensures client compatability with the daemon. The default is set
    to false as the wrapper could overwrite ovs-vsctl installed on the
    host.

    [1] https://github.com/libvirt/libvirt/blob/ee51ab86c2e52b6ff1b17a4c7ad11439fd610c9e/src/util/virnetdevopenvswitch.c#L59
    [2] https://github.com/libvirt/libvirt/blob/a89b17c2a75cfbaeb9e430f88e0f8a7475eb4f54/docs/kbase/internals/command.rst#id3

    Closes-Bug: #1995409
    Change-Id: Iaa6bfb012ae847f5f6aa0a1fc1c27970ac265f93
    (cherry picked from commit a82443481e17edecda728f8698f8b401ec910546)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 14.8.0 release.