[OVN] Manual sync misidentifies Octavia OVN Load Balancer health monitor port and deletes metadata port

[Bartosz Bezak]

Versions:
Neutron: Stable/Yoga (from commit cbe514ecf6)
Octavia: Stable/Yoga (from commit 5bf40c89)
OVN: 22.09 - ovn22.09-22.09.0-31.el9s.x86_64

Looks related to https://bugs.launchpad.net/neutron/+bug/2004238, however the fix from it is already applied.

Doing manual ovn sync:
neutron-ovn-db-sync-util --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair

Misidentifies Octavia OVN Load Balancer Health Monitor Port and Deletes Metadata Port:

47:2023-09-28 13:22:29.565 2053 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-b037dd97-5936-4981-b0be-d4c88c6e806d - - - - -] Unnecessary DHCP port 77cc790a-1712-4fc2-b184-918842588a06 for network dd8dbcd7-0f6f-424f-aa2e-256630af44fd found in Neutron
48:2023-09-28 13:22:29.565 2053 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-b037dd97-5936-4981-b0be-d4c88c6e806d - - - - -] Deleting unnecessary DHCP port 77cc790a-1712-4fc2-b184-918842588a06 for network dd8dbcd7-0f6f-424f-aa2e-256630af44fd

OVN LB health monitor port got .3 IP, and metadata uses it now as its port:

ip netns exec ovnmeta-dd8dbcd7-0f6f-424f-aa2e-256630af44fd ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tapdd8dbcd7-01@if2868: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fa:16:3e:c9:12:45 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.169.254/32 brd 169.254.169.254 scope global tapdd8dbcd7-01
       valid_lft forever preferred_lft forever
    inet 192.168.100.3/24 brd 192.168.100.255 scope global tapdd8dbcd7-01
       valid_lft forever preferred_lft forever

openstack port show ovn-lb-hm-b7383d04-9220-48cd-bac9-3e2c53ad2d5f
+-------------------------+------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | |
| binding_profile | |
| binding_vif_details | |
| binding_vif_type | unbound |
| binding_vnic_type | normal |
| created_at | 2023-09-14T12:30:30Z |
| data_plane_status | None |
| description | |
| device_id | |
| device_owner | network:distributed |
| device_profile | None |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='192.168.100.3', subnet_id='b7383d04-9220-48cd-bac9-3e2c53ad2d5f' |
| id | 2831bc8f-60e4-4a44-88e1-f3010cce3cbd |
| ip_allocation | None |
| mac_address | fa:16:3e:c9:12:45 |
| name | ovn-lb-hm-b7383d04-9220-48cd-bac9-3e2c53ad2d5f |
| network_id | dd8dbcd7-0f6f-424f-aa2e-256630af44fd |
| numa_affinity_policy | None |
| port_security_enabled | False |
| project_id | e4529e5a687e4128af2935fb33cfa7c2 |
| propagate_uplink_status | None |
| qos_network_policy_id | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 1 |
| security_group_ids | |
| status | DOWN |
| tags | |
| trunk_details | None |
| updated_at | 2023-09-14T12:30:30Z |
+-------------------------+------------------------------------------------------------------------------+

Workaround - removed health monitor, readded manually ovnmeta port, restart metadata agents.

[Michal Nasiadka (mnasiadka)]

The sync script is checking for all metadata ports (network:distributed owner) - https://github.com/openstack/neutron/blob/ea0bc376a2ec93a78ee5c9d4772a970832dc367b/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L971

And ovn-octavia-provider is creating port per HM with network:distributed owner (https://review.opendev.org/c/openstack/ovn-octavia-provider/+/874271/1/ovn_octavia_provider/helper.py#144)

[Bartosz Bezak (bbezak)]

it is also a weird case when metadata agents is trying to use health monitor port for serving metadata

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/897345

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/897348

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/897858

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/898535

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/897345
Committed: https://opendev.org/openstack/neutron/commit/b2e14b23f3aa3fa7ff86cd47f346bebe48256615
Submitter: "Zuul (22348)"
Branch: master

commit b2e14b23f3aa3fa7ff86cd47f346bebe48256615
Author: Fernando Royo <email address hidden>
Date: Wed Oct 4 17:58:43 2023 +0200

    Add constant to identify OVN LB HM ports

    OVN metadata and OVN LB HM port (used for health checks) are both
    using network:distributed in the device_owner field. Some tasks
    in Neutron search by that value, considering just one port will
    exist per network with that value.

    This patch adds a new constant to differentiate OVN metadata ports
    from the OVN LB HM ones. The latter will use a device_owner
    of ovn-lb-hm:distributed to be different but will still receive the
    treatment of a localport in the OVN NB DB.

    Related-Bug: 2038091
    Change-Id: I70de163fad34371de10fec28c51959384900ebc8

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Fernando Royo <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/897858
Reason: In favor of https://review.opendev.org/c/openstack/ovn-octavia-provider/+/898535 in chain with the parent one

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/898535
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/1661f3815c1518aa2e3f1761e33ec494be31d584
Submitter: "Zuul (22348)"
Branch: master

commit 1661f3815c1518aa2e3f1761e33ec494be31d584
Author: Fernando Royo <email address hidden>
Date: Tue Oct 17 10:49:23 2023 +0200

    Add maintenance task to update entities on component start

    Some changes require updating the existing entities in a
    clear and transparent way for the user.

    This patch adds a mechanism to create separate tasks that
    can run periodically or just once in order to update or
    modify existing entities that require changes after a new
    patch or RFE.

    As an example, a first task has been included for updating
    existing OVN LB HM ports, changing their device_owner, and
    adding their device_id.

    Closes-Bug: 2038091
    Change-Id: I0d4feb1e5c128d5a768d1b87deb2dcb3ab6d1ea1

[Michal Nasiadka (mnasiadka)]

Any chance to get those patches back ported all the way to Yoga?

[Fernando Royo (froyoredhat)]

I don't think we have much chance to bring this to stable/yoga, basically due to complexity of these changes that introduce dependencies between component versions (ovn-octavia-provider and neutron)

[Bartosz Bezak (bbezak)]

Is it possible to backport to Antelope?
Unfortunately this bug prevents from running safely manual sync - which is sometimes needed. Any possible workarounds for it?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/901164

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/901165

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/901179

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/901181

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/901164
Committed: https://opendev.org/openstack/neutron/commit/d5a049ef071c490d254fb383ec63bd4a74a89d95
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit d5a049ef071c490d254fb383ec63bd4a74a89d95
Author: Fernando Royo <email address hidden>
Date: Wed Oct 4 17:58:43 2023 +0200

    Add constant to identify OVN LB HM ports

    OVN metadata and OVN LB HM port (used for health checks) are both
    using network:distributed in the device_owner field. Some tasks
    in Neutron search by that value, considering just one port will
    exist per network with that value.

    This patch adds a new constant to differentiate OVN metadata ports
    from the OVN LB HM ones. The latter will use a device_owner
    of ovn-lb-hm:distributed to be different but will still receive the
    treatment of a localport in the OVN NB DB.

    Related-Bug: 2038091
    Change-Id: I70de163fad34371de10fec28c51959384900ebc8
    (cherry picked from commit b2e14b23f3aa3fa7ff86cd47f346bebe48256615)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/901165
Committed: https://opendev.org/openstack/neutron/commit/de0755232fe497dfdc82c793b9a21472405bcb45
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit de0755232fe497dfdc82c793b9a21472405bcb45
Author: Fernando Royo <email address hidden>
Date: Wed Oct 4 17:58:43 2023 +0200

    Add constant to identify OVN LB HM ports

    OVN metadata and OVN LB HM port (used for health checks) are both
    using network:distributed in the device_owner field. Some tasks
    in Neutron search by that value, considering just one port will
    exist per network with that value.

    This patch adds a new constant to differentiate OVN metadata ports
    from the OVN LB HM ones. The latter will use a device_owner
    of ovn-lb-hm:distributed to be different but will still receive the
    treatment of a localport in the OVN NB DB.

    Conflicts:
          neutron/common/ovn/utils.py
          neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py
          neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_impl_idl.py
          neutron/tests/unit/common/ovn/test_utils.py

    Related-Bug: 2038091
    Change-Id: I70de163fad34371de10fec28c51959384900ebc8
    (cherry picked from commit b2e14b23f3aa3fa7ff86cd47f346bebe48256615)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/901179
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/9a410d1b4206b5fecff81130b10fdb5429e04ba3
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 9a410d1b4206b5fecff81130b10fdb5429e04ba3
Author: Fernando Royo <email address hidden>
Date: Tue Oct 17 10:49:23 2023 +0200

    Add maintenance task to update entities on component start

    Some changes require updating the existing entities in a
    clear and transparent way for the user.

    This patch adds a mechanism to create separate tasks that
    can run periodically or just once in order to update or
    modify existing entities that require changes after a new
    patch or RFE.

    As an example, a first task has been included for updating
    existing OVN LB HM ports, changing their device_owner, and
    adding their device_id.

    Closes-Bug: 2038091
    Change-Id: I0d4feb1e5c128d5a768d1b87deb2dcb3ab6d1ea1
    (cherry picked from commit 1661f3815c1518aa2e3f1761e33ec494be31d584)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/901181
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/2f72bc870b01bca6e16246074881145a5223cba5
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 2f72bc870b01bca6e16246074881145a5223cba5
Author: Fernando Royo <email address hidden>
Date: Tue Oct 17 10:49:23 2023 +0200

    Add maintenance task to update entities on component start

    Some changes require updating the existing entities in a
    clear and transparent way for the user.

    This patch adds a mechanism to create separate tasks that
    can run periodically or just once in order to update or
    modify existing entities that require changes after a new
    patch or RFE.

    As an example, a first task has been included for updating
    existing OVN LB HM ports, changing their device_owner, and
    adding their device_id.

    Closes-Bug: 2038091
    Change-Id: I0d4feb1e5c128d5a768d1b87deb2dcb3ab6d1ea1
    (cherry picked from commit 1661f3815c1518aa2e3f1761e33ec494be31d584)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovn-octavia-provider 6.0.0.0rc1 release candidate.