[OVN] host id in NB database not updated correctly for virtual ports

[Michel Nederlof]

We're using the ovn-bgp-agent's nb-bgp driver and noticed that while trying to expose octavia load balancers there were two issues.

1. The neutron:host_id information on the lsp is one step behind the actual location, while the information from the neutron API _is_ correct.

2. OVN stores the resolved mac address of this virtual port in the Mac_Binding table, but does not remove it upon move. -> addressed in issue https://bugs.launchpad.net/neutron/+bug/2038422

Steps to reproduce (on our end at least):
Create 3 ports:
- virtual port (used for VIP)
- internal port 1 - attached to vm1
- internal port 2 - attached to vm2

Then create keepalived config (or just manually assign the vip ip to one of the internal ports), and send out gratuitous arp replies or ping from the other vm so there is a normal arp reply so OVN binds the port to the virtual port.

Then move the VIP from vm1 to vm2 and make sure OVN picks it up (the virtual parent and/or chassis changes on the vip lsp).

Now neutron will update the information based on events PortBindingChassisUpdateEvent and PortBindingUpdateVirtualPortsEvent.

Because these are running simultaniously, the chassis update event updates the information based on neutron database, which might not have been updated yet by the virtualport event, causing inconsistency in the OVN db's

We're running OVN 23.09 and Neutron 2023.1, deployed with kolla.

[Michel Nederlof (mnederlof)]

https://review.opendev.org/c/openstack/neutron/+/896883

Potential fix

[Michel Nederlof (mnederlof)]

The event that happens when a failover occurs (on OVN DB):

PortBindingUpdateVirtualPortsEvent
    event: update
    row: Port_Binding(
        additional_chassis=[],
        additional_encap=[],
        chassis=[<ovs.db.idl.Row object at 0x7f34df610280>],
        datapath=Datapath_Binding(
            external_ids={
                'logical-switch': '0b76fac4-37da-4e8c-b3a9-6e56eb629eda',
                'name': 'neutron-46146c1c-629f-4339-90b7-d5aab4728280',
                'name2': 'internal-int-ipv4'},
            load_balancers=[],
            tunnel_key=3),
        encap=[],
        external_ids={'name': 'vip-10-90-0-100',
            'neutron:cidrs': '10.90.0.100/24',
            'neutron:device_id': '',
            'neutron:device_owner': 'ha-ip',
            'neutron:host_id': 'compute-host-3',
            'neutron:network_name': 'neutron-46146c1c-629f-4339-90b7-d5aab4728280',
            'neutron:port_capabilities': '',
            'neutron:port_fip': '1.1.1.47',
            'neutron:port_name': 'vip-10-90-0-100',
            'neutron:project_id': 'b8ab3ca699314f9895e6e758a8c33640',
            'neutron:revision_number': '789',
            'neutron:security_group_ids': 'cdd1a155-37bc-45b7-9eb6-d99c023c5792 fa00a21f-a671-4ef1-af74-da1bbdffcea8',
            'neutron:subnet_pool_addr_scope4': '',
            'neutron:subnet_pool_addr_scope6': '',
            'neutron:vnic_type': 'normal'
        },
        gateway_chassis=[],
        ha_chassis_group=[],
        logical_port=1f243ee1-81ba-404e-ba1a-6c9d8dac139b,
        mac=['fa:16:3e:38:30:05 10.90.0.100'],
        mirror_rules=[],
        nat_addresses=[],
        options={
            'virtual-ip': '10.90.0.100',
            'virtual-parents': '43596871-e65f-4812-9932-223c7f690432,aa0b1377-f0ef-463c-9f9e-79b8c62b2712,98e5aa4f-adbf-4dde-8ea3-26dc3693ee89'
        },
        parent_port=[],
        port_security=['fa:16:3e:38:30:05 10.90.0.100'],
        requested_additional_chassis=[],
        requested_chassis=[],
        tag=[],
        tunnel_key=15,
        type=virtual,
        up=[True],
        virtual_parent=['98e5aa4f-adbf-4dde-8ea3-26dc3693ee89']
    )

    old: Port_Binding(
        chassis=[<ovs.db.idl.Row object at 0x7f34df608160>],
        virtual_parent=['aa0b1377-f0ef-463c-9f9e-79b8c62b2712']
    )

[Michel Nederlof (mnederlof)]

information about the ports:

mnederlof@dev:~$ openstack port show 43596871-e65f-4812-9932-223c7f690432
+-------------------------+--------------------------------------------------------------------------+
| Field | Value |
+-------------------------+--------------------------------------------------------------------------+
| allowed_address_pairs | ip_address='10.90.0.100', mac_address='fa:16:3e:41:b5:41' |
| binding_host_id | compute-host-11 |
| fixed_ips | ip_address='10.90.0.4', subnet_id='33cbdec1-a0b0-4b1e-8d99-1d22b764e1c5' |
| id | 43596871-e65f-4812-9932-223c7f690432 |
+-------------------------+--------------------------------------------------------------------------+

mnederlof@dev:~$ openstack port show aa0b1377-f0ef-463c-9f9e-79b8c62b2712
+-------------------------+----------------------------------------------------------------------------+
| Field | Value |
+-------------------------+----------------------------------------------------------------------------+
| allowed_address_pairs | ip_address='10.90.0.100', mac_address='fa:16:3e:3d:29:bb' |
| binding_host_id | compute-host-3 |
| fixed_ips | ip_address='10.90.0.109', subnet_id='33cbdec1-a0b0-4b1e-8d99-1d22b764e1c5' |
| id | aa0b1377-f0ef-463c-9f9e-79b8c62b2712 |
+-------------------------+----------------------------------------------------------------------------+

mnederlof@dev:~$ openstack port show 98e5aa4f-adbf-4dde-8ea3-26dc3693ee89
+-------------------------+----------------------------------------------------------------------------+
| Field | Value |
+-------------------------+----------------------------------------------------------------------------+
| allowed_address_pairs | ip_address='10.90.0.100', mac_address='fa:16:3e:64:fc:d0' |
| binding_host_id | compute-host-1 |
| fixed_ips | ip_address='10.90.0.239', subnet_id='33cbdec1-a0b0-4b1e-8d99-1d22b764e1c5' |
| id | 98e5aa4f-adbf-4dde-8ea3-26dc3693ee89 |
+-------------------------+----------------------------------------------------------------------------+

mnederlof@dev:~$ openstack port show 1f243ee1-81ba-404e-ba1a-6c9d8dac139b
+-------------------------+----------------------------------------------------------------------------+
| Field | Value |
+-------------------------+----------------------------------------------------------------------------+
| admin_state_up | UP ...

[Michel Nederlof (mnederlof)]

When running the latest neutron version (master), i see that all updates to NBDB have stopped working for this port upon failover.

I do see, that only one event (PortBindingUpdateVirtualPortsEvent [2]) is triggered now; the PortBindingChassisUpdateEvent and PortBindingChassisEvent no longer trigger upon this VIP failover.

So the neutron sql database is updated properly, according to the self.driver.update_virtual_port_host call. but the NorthBound database has not been updated.

So the review should only reflect updating the NB DB in collaboration with the db update.

[1] https://github.com/openstack/neutron/blob/dad8c3fecc4e97764a92f558d9df510fa2516877/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#L1064

[2] Log line from ovsdbapp:
2023-10-24 13:09:52.718 22 INFO ovsdbapp.backend.ovs_idl.event [None req-cdf5ab0d-5614-4ea0-88f7-fe3fe8f802c1 - - - - - -] Matched UPDATE:
    PortBindingUpdateVirtualPortsEvent(events=('update', 'delete'), table='Port_Binding', conditions=None, old_conditions=None), priority=20 to
        row=Port_Binding(
            additional_chassis=[],
            additional_encap=[],
            chassis=[<ovs.db.idl.Row object at 0x7fdfafbcfbb0>],
            datapath=bb74b997-d33d-41cc-914c-6af30eff17f7,
            encap=[],
            external_ids={
                'name': 'vip-10-90-0-100',
                'neutron:cidrs': '10.90.0.100/24',
                'neutron:device_id': '',
                'neutron:device_owner': '',
                'neutron:host_id': 'compute-host-11',
                'neutron:network_name': 'neutron-28926ec4-c653-48f5-a62e-3bcd86b25114',
                'neutron:port_capabilities': '',
                'neutron:port_fip': '198.51.100.220',
                'neutron:port_name': 'vip-10-90-0-100',
                'neutron:project_id': 'b8ab3ca699314f9895e6e758a8c33640',
                'neutron:revision_number': '223',
                'neutron:security_group_ids': '',
                'neutron:subnet_pool_addr_scope4': '',
                'neutron:subnet_pool_addr_scope6': '',
                'neutron:vnic_type': 'normal'},
            gateway_chassis=[],
            ha_chassis_group=[],
            logical_port=1f243ee1-81ba-404e-ba1a-6c9d8dac139b,
            mac=['fa:16:3e:38:30:05 10.90.0.100'],
            mirror_rules=[])
            nat_addresses=[],
            options={
                'virtual-ip': '10.90.0.100',
                'virtual-parents': '98e5aa4f-adbf-4dde-8ea3-26dc3693ee89,aa0b1377-f0ef-463c-9f9e-79b8c62b2712,43596871-e65f-4812-9932-223c7f690432'},
            parent_port=[],
            port_security=[],
            requested_additional_chassis=[],
            requested_chassis=[],
            tag=[],
            tunnel_key=8,
            type=virtual,
            up=[True],
            virtual_parent=['98e5aa4f-adbf-4dde-8ea3-26dc3693ee89'],
        old=Port_Binding(chassis=[<ovs.db.idl.Row object at 0x7fdfafbcea10>], virtual_parent=['aa0b1377-f0ef-463c-9f9e-79b8c62b2712'])

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/899109

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Michel Nederlof <email address hidden>" on branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/899109
Reason: duplicate, wrong review.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896883
Committed: https://opendev.org/openstack/neutron/commit/e68a920c114010a36a667d90866bfb6243148b6d
Submitter: "Zuul (22348)"
Branch: master

commit e68a920c114010a36a667d90866bfb6243148b6d
Author: Michel Nederlof <email address hidden>
Date: Wed Oct 4 09:31:50 2023 +0200

    [OVN] Update lsp host id when virtual parent moves

    When a virtual port is moved from one port to another port the
    PortBindingUpdateVirtualPortsEvent event would only update the binding
    host id in the neutron database, while it is also usefull to keep the
    information in the OVN database up to date with the host information

    Other plugins that connect to the OVN database can then also rely on the
    information stored in the OVN DB's

    Closes-Bug: #2038413

    Change-Id: I59c6c4b2c8b023b9c9c3bab1741d957fa1f738fc

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 24.0.0.0rc1 release candidate.