[OVN][Trunk] subport doesn't reach status ACTIVE

[Eduardo Olivares]

Test test_live_migration_with_trunk has been failing for the last two days.
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3f3/831018/30/check/nova-live-migration/3f3b065/testr_results.html

It's a test about live-migration, but it is important to notice that it fails before any live migration happens.

The test creates a VM with a port and a subport.
The test waits until the VM status is ACTIVE -> this passes
The test waits until the subport status is ACTIVE -> this started failing two days ago because the port status is DOWN

There was only one neutron patch merged that day[1], but I checked the test failed during some jobs even before that patch was merged.

I compared some logs.
Neutron logs when the test passes: [2]
Neutron logs when the test fails: [3]

When it fails, I see this during the creation of the subport (and I don't see this event when it passes):
Jun 14 18:13:43.052982 np0034303809 neutron-server[77531]: DEBUG ovsdbapp.backend.ovs_idl.event [None req-929dd199-4247-46f5-9466-622c7d538547 None None] Matched DELETE: PortBindingUpdateVirtualPortsEvent(events=('update', 'delete'), table='Port_Binding', conditions=None, old_conditions=None), priority=20 to row=Port_Binding(parent_port=[], mac=['fa:16:3e:93:9d:5a 19.80.0.42'], chassis=[], ha_chassis_group=[], options={'mcast_flood_reports': 'true', 'requested-chassis': ''}, type=, tag=[], requested_chassis=[], tunnel_key=2, up=[False], logical_port=f8c707ec-ecd8-4f1e-99ba-6f8303b598b2, gateway_chassis=[], encap=[], external_ids={'name': 'tempest-subport-2029248863', 'neutron:cidrs': '19.80.0.42/24', 'neutron:device_id': '', 'neutron:device_owner': '', 'neutron:network_name': 'neutron-5fd9faa7-ec1c-4f42-ab87-6ce19edda245', 'neutron:port_capabilities': '', 'neutron:port_name': 'tempest-subport-2029248863', 'neutron:project_id': '6f92a9f8e16144148026725b25711d3a', 'neutron:revision_number': '1', 'neutron:security_group_ids': '5eab41ef-c5c1-425c-a931-f5b6b4b330ad', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': '', 'neutron:vnic_type': 'normal'}, virtual_parent=[], nat_addresses=[], datapath=3c472399-d6ee-4b7c-aa97-6777f2bc2772) old= {{(pid=77531) matches /usr/local/lib/python3.10/dist-packages/ovsdbapp/backend/ovs_idl/event.py:43}}
...
Jun 14 18:13:49.597911 np0034303809 neutron-server[77531]: DEBUG neutron.plugins.ml2.plugin [None req-3588521e-7878-408d-b1f8-15db562c69f8 None None] Port f8c707ec-ecd8-4f1e-99ba-6f8303b598b2 cannot update to ACTIVE because it is not bound. {{(pid=77531) _port_provisioned /opt/stack/neutron/neutron/plugins/ml2/plugin.py:361}}

It seems the ovn version has changed between these jobs:
Passes [4]:
2023-06-14 10:01:46.358875 | controller | Preparing to unpack .../ovn-common_22.03.0-0ubuntu1_amd64.deb ...

Fails [5]:
2023-06-14 17:55:07.077377 | controller | Preparing to unpack .../ovn-common_22.03.2-0ubuntu0.22.04.1_amd64.deb ...

[1] https://review.opendev.org/c/openstack/neutron/+/883687
[2] https://96b562ba0d2478fe5bc1-d58fbc463536b3122b4367e996d5e5b0.ssl.cf1.rackcdn.com/831018/30/check/nova-live-migration/312c2ab/controller/logs/screen-q-svc.txt
[3] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3f3/831018/30/check/nova-live-migration/3f3b065/controller/logs/screen-q-svc.txt
[4] https://96b562ba0d2478fe5bc1-d58fbc463536b3122b4367e996d5e5b0.ssl.cf1.rackcdn.com/831018/30/check/nova-live-migration/312c2ab/job-output.txt
[5] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3f3/831018/30/check/nova-live-migration/3f3b065/job-output.txt

[Lajos Katona (lajos-katona)]

thanks, opensearch link:
https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22line%20273,%20in%20test_live_migration_with_trunk%22'),sort:!())

[Lajos Katona (lajos-katona)]

OVN version bump was introduced with this patch:
https://review.opendev.org/c/openstack/neutron/+/883681

And it is related to port binding so suspicious: "Improve the ``PortBindingUpdateVirtualPortsEvent`` match filter"

[Lajos Katona (lajos-katona)]

Marking the failing tests as unstable:
https://review.opendev.org/c/openstack/tempest/+/886496

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

This issue is related to [0]. I'm investigating the patch proposed for this bug [1] that is solving this issue by updating the subport VIF type and the host ID, missing in the Trunk service, in the OVN driver.

[0]https://bugs.launchpad.net/neutron/+bug/2018289
[1]https://review.opendev.org/c/openstack/neutron/+/882581

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Patch proposed: https://review.opendev.org/c/openstack/neutron/+/882581

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.1.2 release.

[Ghanshyam Mann (ghanshyammann)]

I am seeing this test failing, may be on same things?

https://zuul.opendev.org/t/openstack/build/8ace3a8e9f3b4815950b40db262d621e/log/job-output.txt#33049

[Eduardo Olivares (eolivare)]

@ghanshyammann,
This bug was reproduced with a "live-migration" test, but notice the issue happened before any migrarion was performed:
"It's a test about live-migration, but it is important to notice that it fails before any live migration happens."
And it failed 100% times.

What you are reporting now looks similar, but it happens after live-migration (a few lines above, the test checked both subport and parent port are ACTIVE before live-migration):
https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_live_migration.py#L292
And apparently it doesn't fail always ("887220,4" was rechecked and the test passed):
https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&project=openstack/tempest

So, IMO, this is a different bug. Some things are common: OVN and trunk ports. Some things are different: happens after live-migration, doesn't fail 100% times.

[Eduardo Olivares (eolivare)]

After talking with Rodolfo, I have opened a new bug for the issue I described in the previous comment:
https://bugs.launchpad.net/neutron/+bug/2027605

[Ghanshyam Mann (ghanshyammann)]

Thanks eolivare.

It is failing 48 times after it was unskipped (https://review.opendev.org/c/openstack/tempest/+/887682).

may be first fux unhide this bug. thanks for reporting new bug, I agree with that.

https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2023-06-10T17:30:00.000Z',to:now))&_a=(columns:!(build_branch,build_change,build_name,project,build_status,build_queue),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:d,query:(language:lucene,query:'message:%22test_live_migration.py,%20line%20292,%20in%20test_live_migration_with_trunk%22%20'),sort:!())

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 23.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/895152

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/895152
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/370f71e9877fad13883413225bf25051c0c9c391
Submitter: "Zuul (22348)"
Branch: master

commit 370f71e9877fad13883413225bf25051c0c9c391
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sat Sep 9 21:33:55 2023 +0000

    Mark "test_subport_connectivity_soft_reboot" as unstable

    This issue is related to:
    * https://bugs.launchpad.net/neutron/+bug/2033887: the patches
      previously merged in the Neutron repository [1][2] should be
      reverted.
    * https://bugs.launchpad.net/neutron/+bug/2024160: this issue
      is still present.

    [1]https://review.opendev.org/c/openstack/neutron/+/882581
    [2]https://review.opendev.org/c/openstack/neutron/+/888776

    Change-Id: I9d8d76810d15913c29610464ab1b8fa34863aaf6
    Related-Bug: #2024160
    Related-Bug: #2033887

[Mohammed Naser (mnaser)]

I'm running into this and putting some notes for my troubleshooting, we're running OVN 23.03.0-1~cloud0 and running into this issue.

I'm dredging through this to see what I have, but I have a failing case that I'm looking here in front of me right now.

https://github.com/ovn-org/ovn/compare/v22.03.0...v22.03.3

[Mohammed Naser (mnaser)]

So the thing that I noticed is that I assume there are no addresses assigned for the sub-ports, so what happens seems to be this in the `ovn-controller` logs:

```
2023-09-22T20:15:38.402Z|380055|binding|INFO|Claiming lport c6858836-2ef8-4259-bf30-e7d769d8c165 for this chassis.
2023-09-22T20:15:38.402Z|380056|binding|INFO|c6858836-2ef8-4259-bf30-e7d769d8c165: Claiming unknown
```

But I'm not seeing the same output as a usual virtual port...

```
2023-09-22T20:15:47.959Z|380057|pinctrl|INFO|Claiming virtual lport fa57424b-3228-4b29-9bb4-c6edb7927a5f for this chassis with the virtual parent cb15f805-943d-4efa-9646-7a700e6cd81b
2023-09-22T20:15:52.853Z|380058|pinctrl|INFO|Claiming virtual lport ced5a54f-bfab-40b2-8c40-82c97f3c2586 for this chassis with the virtual parent cb15f805-943d-4efa-9646-7a700e6cd81b
```

So I wonder what's up with that. It seems that OVN is happy if there is an IP assigned to parent and the sub-ports via Neutron, but it doesn't like it when there is no IP assigned (and port security enabled?)

[Mohammed Naser (mnaser)]

So, running an `ovn-trace` for both the sub-port and the trunk, when port security is disabled, shows that it kills it early when it sees VLAN traffic. I suspect this is another bug:

```
❯ kubectl -n openstack exec pods/ovn-ovsdb-sb-2 -- ovn-trace --friendly-names --ovs neutron-ea05f3c2-b370-49e6-ab71-635376211fd5 '
  inport=="bffe6ba0-6bdc-4f99-b43e-a5dc9e5b6aea" &&
  vlan.vid==20 &&
  eth.src==52:54:00:3f:1d:a9 &&
  ip4.src==172.17.0.11 &&
  ip4.dst==172.17.0.100 &&
  ip.ttl == 64 &&
  icmp4.type == 8'
Defaulted container "ovsdb" out of: ovsdb, init (init)
# icmp,reg14=0xf,dl_vlan=20,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=52:54:00:3f:1d:a9,dl_dst=00:00:00:00:00:00,nw_src=172.17.0.11,nw_dst=172.17.0.100,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0

ingress(dp="zuul-ci-net-17dc44ba", inport="controller-ffd424a4-3b96-489a-84dd-e6d742e9cf8e-20")
-----------------------------------------------------------------------------------------------
 0. ls_in_check_port_sec (northd.c:8377): vlan.present, priority 100, uuid d24b30bb
    drop;
```

[Mohammed Naser (mnaser)]

So in this case, it seems that for some reason, the maintenance task seems to 'unset' the state of the port:

2023-09-26 05:59:37.812 27 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-30eb5d90-11a4-422f-b017-a4c4099c8722 - - - - - -] Maintenance task: Fixing resource 34c341e5-3eb6-47d7-bc96-54a9a1aca61f (type: ports) at create/update check_for_inconsistencies /var/lib/openstack/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:384
2023-09-26 05:59:37.855 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(name=34c341e5-3eb6-47d7-bc96-54a9a1aca61f, resource={'id': '34c341e5-3eb6-47d7-bc96-54a9a1aca61f', 'name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'network_id': '16b4d11e-6734-4058-814e-7d22c4c9a8da', 'tenant_id': '4b633c451ac74233be3721a3635275e5', 'mac_address': 'fa:16:3e:30:c1:41', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '', 'device_owner': 'trunk:subport', 'standard_attr_id': 64968243, 'fixed_ips': [], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': '', 'binding:vif_type': 'unbound', 'binding:vif_details': {}, 'dns_name': '', 'dns_assignment': [], 'dns_domain': '', 'port_security_enabled': False, 'qos_policy_id': None, 'qos_network_policy_id': None, 'resource_request': None, 'ip_allocation': 'none', 'tags': [], 'created_at': '2023-09-26T05:54:30Z', 'updated_at': '2023-09-26T05:54:36Z', 'revision_number': 4, 'project_id': '4b633c451ac74233be3721a3635275e5'}, resource_type=ports, if_exists=True) do_commit /var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2023-09-26 05:59:37.856 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLSwitchPortCommand(lport=34c341e5-3eb6-47d7-bc96-54a9a1aca61f, external_ids_update=None, columns={'external_ids': {'neutron:port_name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'neutron:device_id': '', 'neutron:project_id': '4b633c451ac74233be3721a3635275e5', 'neutron:cidrs': '', 'neutron:device_owner': 'trunk:subport', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': '', 'neutron:network_name': 'neutron-16b4d11e-6734-4058-814e-7d22c4c9a8da', 'neutron:security_group_ids': '', 'neutron:revision_number': '4'}, 'parent_name': [], 'tag': [], 'options': {'requested-chassis': ''}, 'enabled': True, 'port_security': [], 'dhcpv4_options': [], 'dhcpv6_options': [], 'type': '', 'addresses': ['fa:16:3e:30:c1:41', 'unknown'], 'ha_chassis_group': []}, if_exists=False) do_commit /var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2023-09-26 05:59:37.857 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): PgDelPortCommand(port_group=neutron_pg_drop, lsp=['34c341e5-3eb6-47d7-bc96-54a9a1aca61f'], if_exists=False) do_commit /var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2023-09-26 05:59:38.117 11 DEBUG ovsdbapp.backend.ovs_idl.event [None req...

[Mohammed Naser (mnaser)]

So I think the root cause is these events are causing the data to be cleared in the Neutron database, which then result in the port being removed after 5 minutes on the next maintenance cycle:

```
2023-09-26 05:54:36.201 11 DEBUG ovsdbapp.backend.ovs_idl.event [None req-c92127e4-2312-41be-9705-97f45cfcd374 - - - - - -] Matched UPDATE: PortBindingChassisEvent(events=('update',), table='Port_Binding', conditions=(('type', '=', 'chassisredirect'),), old_conditions=None), priority=20 to row=Port_Binding(parent_port=['7703365a-b55a-4856-a223-8ee5115e8400'], mac=['unknown'], chassis=[<ovs.db.idl.Row object at 0x7f3fe8ae89d0>], options={'requested-chassis': ''}, ha_chassis_group=[], type=, additional_encap=[], tag=[20], requested_additional_chassis=[], up=[True], tunnel_key=4, requested_chassis=[], additional_chassis=[], logical_port=34c341e5-3eb6-47d7-bc96-54a9a1aca61f, encap=[], gateway_chassis=[], external_ids={'name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'neutron:cidrs': '', 'neutron:device_id': '', 'neutron:device_owner': 'trunk:subport', 'neutron:network_name': 'neutron-16b4d11e-6734-4058-814e-7d22c4c9a8da', 'neutron:port_name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'neutron:project_id': '4b633c451ac74233be3721a3635275e5', 'neutron:revision_number': '2', 'neutron:security_group_ids': '', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': ''}, nat_addresses=[], virtual_parent=[], port_security=[], datapath=dc18c51f-d011-41ff-8c0e-1cd6a778bd28, mirror_rules=[]) old=Port_Binding(chassis=[], up=[False]) matches /var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/event.py:43
2023-09-26 05:54:36.259 14 DEBUG ovsdbapp.backend.ovs_idl.event [None req-dae54ec1-7359-47ff-bf53-34cdfcb55fdf - - - - - -] Matched DELETE: PortBindingUpdateVirtualPortsEvent(events=('update', 'delete'), table='Port_Binding', conditions=None, old_conditions=None), priority=20 to row=Port_Binding(parent_port=[], mac=['unknown'], chassis=[], options={'requested-chassis': ''}, ha_chassis_group=[], type=, additional_encap=[], tag=[], requested_additional_chassis=[], up=[False], tunnel_key=4, requested_chassis=[], additional_chassis=[], logical_port=34c341e5-3eb6-47d7-bc96-54a9a1aca61f, encap=[], gateway_chassis=[], external_ids={'name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'neutron:cidrs': '', 'neutron:device_id': '', 'neutron:device_owner': '', 'neutron:network_name': 'neutron-16b4d11e-6734-4058-814e-7d22c4c9a8da', 'neutron:port_name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'neutron:project_id': '4b633c451ac74233be3721a3635275e5', 'neutron:revision_number': '1', 'neutron:security_group_ids': '', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': ''}, nat_addresses=[], virtual_parent=[], port_security=[], datapath=dc18c51f-d011-41ff-8c0e-1cd6a778bd28, mirror_rules=[]) old= matches /var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/event.py:43
```

More specifically, the DELETE on `PortBindingUpdateVirtualPortsEvent` is clearing out the binding details, so on the next maintenance run, Neutron kils it off.

[Mohammed Naser (mnaser)]

OK, I am pretty sure the broken change is:

https://review.opendev.org/c/openstack/neutron/+/883681

I believe this is now firing this code for ALL ports instead of virtual ports..

[Mohammed Naser (mnaser)]

There we go, I tracked it down to this commit:

https://github.com/ovn-org/ovn/commit/28d40c0a024227c9c845117087b7594723ff0ec7

The reality is that we 'fixed' Neutron by accommodating for this:

https://github.com/ovn-org/ovn/commit/ec933537f9b04b35ef4b79fb2b4743f9095da209

But we didn't notice we might be getting deletes for trunk ports (parent_port aka container). We need to accomodate for this.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Patch fixing a filtering issue in [1]: https://review.opendev.org/c/openstack/neutron/+/896590

[1]https://review.opendev.org/c/openstack/neutron/+/883681

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/896803

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/896804

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/896805

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896590
Committed: https://opendev.org/openstack/neutron/commit/1160aaa4716904fdba06a8dcfbc1d5a2ce419e72
Submitter: "Zuul (22348)"
Branch: master

commit 1160aaa4716904fdba06a8dcfbc1d5a2ce419e72
Author: Mohammed Naser <email address hidden>
Date: Tue Sep 26 20:02:55 2023 +0000

    [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent

    With newer versions of OVN, port bindings are removed and added
    for both virtual ports and container ports (aka trunk ports in
    Neutron world)[1][2].

    With this, Neutron recently landed a change in order to handle
    virtual ports for the update/delete steps[3], however, this has
    resulted in unexpected behaviour where the binding information
    is wiped for trunk ports, then subsequently removed on the
    next maintenance run.

    We should make sure that we only have row deletes for
    virtual ports specifically rather than all ports.

    [1]: https://github.com/ovn-org/ovn/commit/ec933537f9b04b35ef4b79fb2b4743f9095da209
    [2]: https://github.com/ovn-org/ovn/commit/28d40c0a024227c9c845117087b7594723ff0ec7
    [3]: https://review.opendev.org/c/openstack/neutron/+/883681

    Related-Bug: #2024160
    Change-Id: Ide3204748274cbab9731310c2e2bd8bd8c9a53ff

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896804
Committed: https://opendev.org/openstack/neutron/commit/2319e7c35f79798cba199f52a5a7c4a4ef7bc088
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 2319e7c35f79798cba199f52a5a7c4a4ef7bc088
Author: Mohammed Naser <email address hidden>
Date: Tue Sep 26 20:02:55 2023 +0000

    [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent

    With newer versions of OVN, port bindings are removed and added
    for both virtual ports and container ports (aka trunk ports in
    Neutron world)[1][2].

    With this, Neutron recently landed a change in order to handle
    virtual ports for the update/delete steps[3], however, this has
    resulted in unexpected behaviour where the binding information
    is wiped for trunk ports, then subsequently removed on the
    next maintenance run.

    We should make sure that we only have row deletes for
    virtual ports specifically rather than all ports.

    [1]: https://github.com/ovn-org/ovn/commit/ec933537f9b04b35ef4b79fb2b4743f9095da209
    [2]: https://github.com/ovn-org/ovn/commit/28d40c0a024227c9c845117087b7594723ff0ec7
    [3]: https://review.opendev.org/c/openstack/neutron/+/883681

    Related-Bug: #2024160
    Change-Id: Ide3204748274cbab9731310c2e2bd8bd8c9a53ff
    (cherry picked from commit 1160aaa4716904fdba06a8dcfbc1d5a2ce419e72)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896805
Committed: https://opendev.org/openstack/neutron/commit/0364f18848971446bb98619c94aa0c4024b33531
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 0364f18848971446bb98619c94aa0c4024b33531
Author: Mohammed Naser <email address hidden>
Date: Tue Sep 26 20:02:55 2023 +0000

    [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent

    With newer versions of OVN, port bindings are removed and added
    for both virtual ports and container ports (aka trunk ports in
    Neutron world)[1][2].

    With this, Neutron recently landed a change in order to handle
    virtual ports for the update/delete steps[3], however, this has
    resulted in unexpected behaviour where the binding information
    is wiped for trunk ports, then subsequently removed on the
    next maintenance run.

    We should make sure that we only have row deletes for
    virtual ports specifically rather than all ports.

    [1]: https://github.com/ovn-org/ovn/commit/ec933537f9b04b35ef4b79fb2b4743f9095da209
    [2]: https://github.com/ovn-org/ovn/commit/28d40c0a024227c9c845117087b7594723ff0ec7
    [3]: https://review.opendev.org/c/openstack/neutron/+/883681

    Related-Bug: #2024160
    Change-Id: Ide3204748274cbab9731310c2e2bd8bd8c9a53ff
    (cherry picked from commit 1160aaa4716904fdba06a8dcfbc1d5a2ce419e72)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896803
Committed: https://opendev.org/openstack/neutron/commit/04e9b063ad526778ba80438fc851e03fa16e5085
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 04e9b063ad526778ba80438fc851e03fa16e5085
Author: Mohammed Naser <email address hidden>
Date: Tue Sep 26 20:02:55 2023 +0000

    [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent

    With newer versions of OVN, port bindings are removed and added
    for both virtual ports and container ports (aka trunk ports in
    Neutron world)[1][2].

    With this, Neutron recently landed a change in order to handle
    virtual ports for the update/delete steps[3], however, this has
    resulted in unexpected behaviour where the binding information
    is wiped for trunk ports, then subsequently removed on the
    next maintenance run.

    We should make sure that we only have row deletes for
    virtual ports specifically rather than all ports.

    [1]: https://github.com/ovn-org/ovn/commit/ec933537f9b04b35ef4b79fb2b4743f9095da209
    [2]: https://github.com/ovn-org/ovn/commit/28d40c0a024227c9c845117087b7594723ff0ec7
    [3]: https://review.opendev.org/c/openstack/neutron/+/883681

    Related-Bug: #2024160
    Change-Id: Ide3204748274cbab9731310c2e2bd8bd8c9a53ff
    (cherry picked from commit 1160aaa4716904fdba06a8dcfbc1d5a2ce419e72)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron xena-eom release.