octavia next fails to bind o-hm0 to octavia unit host

[Edward Hope-Morley]

I have deployed the same Stein cloud using stable (19.07) and next (19.10) charms and on the former I have no problems with having the o-hm0 bind to the octavia host but it fails on next and actually breaks neutron-openvswitch-agent which spins continuously. I will attach relevant output to this LP.

[Edward Hope-Morley (hopem)]

I have a peek at some potentially relevant changes that have landed and I suspect this one may be relevant - https://review.opendev.org/#/c/682608/. The code that's getting stuck is at [1]. I checked the good and bad envs and I can confirm that in the working one the o-hm0 port is assigned a vlan tag:

        Port "o-hm0"
            tag: 2
            Interface "o-hm0"
                type: internal

but in the broken env it has not.

[1] https://github.com/openstack/neutron/blob/stable/stein/neutron/agent/linux/openvswitch_firewall/firewall.py#L522

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/685942

[Frode Nordahl (fnordahl)]

Fix proposed to branch: master
Review: https://review.opendev.org/685941/

[Frode Nordahl (fnordahl)]

Fix proposed to branch: master
Review: https://review.opendev.org/685940/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685940
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-openvswitch/commit/?id=32d7c17b7f3de919276bdc0dd1a104149d81af2b
Submitter: Zuul
Branch: master

commit 32d7c17b7f3de919276bdc0dd1a104149d81af2b
Author: Frode Nordahl <email address hidden>
Date: Tue Oct 1 07:46:55 2019 +0200

    Provide configured hostname over subordinate relation

    Change-Id: I75cbc5eb97cf3603ffa5a9a49670411288d90520
    Closes-Bug: #1845303

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685941
Committed: https://git.openstack.org/cgit/openstack/charm-interface-neutron-plugin/commit/?id=6dca7338b2fa8181342a0a6c49bc582a141c3972
Submitter: Zuul
Branch: master

commit 6dca7338b2fa8181342a0a6c49bc582a141c3972
Author: Frode Nordahl <email address hidden>
Date: Tue Oct 1 07:46:04 2019 +0200

    Add auto accessor for ``host`` attribute

    Depends-On: I75cbc5eb97cf3603ffa5a9a49670411288d90520
    Change-Id: I3b11eec3f1c4b8a673ccb6f9e6320d79dbde8f7a
    Closes-Bug: #1845303

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/686082

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686082
Committed: https://git.openstack.org/cgit/openstack/charm-interface-neutron-plugin/commit/?id=adbfcb4ddd5c83d5fa369d3fcce4381a3ac6711e
Submitter: Zuul
Branch: master

commit adbfcb4ddd5c83d5fa369d3fcce4381a3ac6711e
Author: Frode Nordahl <email address hidden>
Date: Wed Oct 2 06:25:23 2019 +0200

    Add ``host`` attribute on requires side

    Commit 6dca7338b2fa8181342a0a6c49bc582a141c3972 added the attribute
    on the wrong side of the interface.

    Change-Id: I6d561617805280caebfe6e408d74ceaccefa1f08
    Closes-Bug: #1845303

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/686333

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686333
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-openvswitch/commit/?id=d0028a7dcd4de6a3bf5176a4267fd78f400076a6
Submitter: Zuul
Branch: master

commit d0028a7dcd4de6a3bf5176a4267fd78f400076a6
Author: Frode Nordahl <email address hidden>
Date: Thu Oct 3 06:48:48 2019 +0200

    Gate HostIPContext on OpenStack release

    The original implementation in [0] relied on the template content
    appearing at the ``Stein`` release.

    With the addition of sharing of this value on the subordinate
    relation in [1] we need this gating in the context too.

    0: Iee73164358745628a4b8658614608bc872771fd1
    1: I75cbc5eb97cf3603ffa5a9a49670411288d90520

    Change-Id: I2b252fd3512c07bb7bccd388809b0514ebcd5bde
    Closes-Bug: #1845303

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685942
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=901df0e1f17bd552f388898787958cbc15fc9e1e
Submitter: Zuul
Branch: master

commit 901df0e1f17bd552f388898787958cbc15fc9e1e
Author: Frode Nordahl <email address hidden>
Date: Tue Oct 1 07:54:37 2019 +0200

    Use hostname from ovs relation when available

    When creating a Neutron port the value of ``binding:host_id`` needs
    to match the host identity of the Neutron OpenvSwitch subordinate.

    Depends-On: I3b11eec3f1c4b8a673ccb6f9e6320d79dbde8f7a
    Change-Id: I6bec4c5a5dade1295414ff8eefc070e2e8127f37
    Closes-Bug: #1845303

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/701929

[Edward Hope-Morley (hopem)]

We have identified an unexpected consequence of the patches landed in this bug which is that /etc/resolve.conf will likely contain more than one search domain, one from the host network dhcp (e.g. maas) and one from the o-hm0 port dhcp (ml2-dns). The issue is that if the openstack dns search domain is written first, dns requests timeout before trying the maas one which is needed to resolve the hostname on o-hm0 and this caused neutron to fail to configure networking (ovs) on that unit. There is a really simple fix for this which is to add UseDomains=route to the networkd config written by the charm. We have tested this in production and are confident it fixes the problem so I will submit as a patch.

[Edward Hope-Morley (hopem)]

Patch for ^^ https://review.opendev.org/#/c/722835/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/722835

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/722835
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=9f050979841d0af546a3d1d13d5116cd86d5295b
Submitter: Zuul
Branch: master

commit 9f050979841d0af546a3d1d13d5116cd86d5295b
Author: Edward Hope-Morley <email address hidden>
Date: Fri Apr 24 16:26:33 2020 +0100

    Don't use neutron dns search domain

    If there is a neutron search domain available don't use
    it since we use the Juju provider dns for the o-hm0 port.

    Change-Id: I369d7708f7149e7b491c0f4a1f86126cb0707373
    Related-Bug: #1845303