set_gateway_mtu fails if one of the lrp is deleted while it's running

[yatin]

Seen in some downstream jobs running with antelope content and ovn_emit_need_to_frag = true, tests fails randomly while running add_router_interface or remove_router_interface calls like:-

ft9.1: setUpClass (tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/test.py", line 206, in setUpClass
    raise value.with_traceback(trace)
  File "/usr/lib/python3.9/site-packages/tempest/test.py", line 191, in setUpClass
    cls.setup_credentials()
  File "/usr/lib/python3.9/site-packages/tempest/api/compute/servers/test_server_actions.py", line 827, in setup_credentials
    super(ServerActionsV293TestJSON, cls).setup_credentials()
  File "/usr/lib/python3.9/site-packages/tempest/api/compute/base.py", line 75, in setup_credentials
    super(BaseV2ComputeTest, cls).setup_credentials()
  File "/usr/lib/python3.9/site-packages/tempest/test.py", line 419, in setup_credentials
    manager = cls.get_client_manager(
  File "/usr/lib/python3.9/site-packages/tempest/test.py", line 764, in get_client_manager
    creds = getattr(cred_provider, credentials_method)()
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 473, in get_primary_creds
    return self.get_project_member_creds()
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 508, in get_project_member_creds
    return self.get_credentials(['member'], scope='project')
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 459, in get_credentials
    network, subnet, router = self._create_network_resources(
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 319, in _create_network_resources
    self._add_router_interface(router['id'], subnet['id'])
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 383, in _add_router_interface
    self.routers_admin_client.add_router_interface(router_id,
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/network/routers_client.py", line 72, in add_router_interface
    return self.update_resource(uri, kwargs)
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/network/base.py", line 77, in update_resource
    resp, body = self.put(req_uri, req_post_data)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 372, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 922, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Request Failed: internal server error while processing your request.

Fails with Internal server error:-
192.168.16.2 - - [13/May/2024:15:09:44 +0000] "PUT /v2.0/routers/a76e7487-6fcb-4134-8de4-f08a3eca33cb/add_router_interface HTTP/1.1" 500 150 "-" "python-urllib3/1.26.5"

2024-05-13T15:10:15.685665853+00:00 stdout F 2024-05-13 15:10:15.684 16 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-8b7c93b4-c0d8-4963-b711-cab362fcada4 - - - - - -] Running txn n=1 command(idx=20): LrpSetOptionsCommand(_result=None, entity=lrp-9309bd6d-2e46-404e-8c47-fda1adc15cc5, options={}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89^[[00m
2024-05-13T15:10:15.686489887+00:00 stdout F 2024-05-13 15:10:15.685 16 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d211c9ca-9694-4d69-9e57-b330a231c30f 9486b7f231584e7ab8f143c840957bf3 6bcd5aaa226d4e69b9e20287ea2d3b11 - - default default] Traceback (most recent call last):
2024-05-13T15:10:15.686489887+00:00 stdout F File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2024-05-13T15:10:15.686489887+00:00 stdout F txn.results.put(txn.do_commit())
2024-05-13T15:10:15.686489887+00:00 stdout F File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
2024-05-13T15:10:15.686489887+00:00 stdout F command.run_idl(txn)
2024-05-13T15:10:15.686489887+00:00 stdout F File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 349, in run_idl
2024-05-13T15:10:15.686489887+00:00 stdout F entity = self.api.lookup(self.table, self.entity)
2024-05-13T15:10:15.686489887+00:00 stdout F File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 183, in lookup
2024-05-13T15:10:15.686489887+00:00 stdout F return self._lookup(table, record)
2024-05-13T15:10:15.686489887+00:00 stdout F File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 234, in _lookup
2024-05-13T15:10:15.686489887+00:00 stdout F row = idlutils.row_by_value(self, rl.table, rl.column, record)
2024-05-13T15:10:15.686489887+00:00 stdout F File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2024-05-13T15:10:15.686489887+00:00 stdout F raise RowNotFound(table=table, col=column, match=match)
2024-05-13T15:10:15.686489887+00:00 stdout F ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-9309bd6d-2e46-404e-8c47-fda1adc15cc5
2024-05-13T15:10:15.686489887+00:00 stdout F ^[[00m

This happens as router port is removed while the LrpSetOptionsCommand runs for that router port, the transaction assumes the port exists[1]. Would need to add if-exists=true in LrpSetOptionsCommand/lrp_set_options to handle such cases.

[1] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2084-L2088

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/919699

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/919806

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/919699
Committed: https://opendev.org/openstack/neutron/commit/79676270354112fd4062442d25a5eaefbc830d24
Submitter: "Zuul (22348)"
Branch: master

commit 79676270354112fd4062442d25a5eaefbc830d24
Author: yatinkarel <email address hidden>
Date: Wed May 15 14:16:53 2024 +0530

    Do not fail on missing logical router ports

    set_gateway_mtu runs for all the gateway ports for a network
    and if one of the ports get's deleted in meanwhile
    whole transaction fails. Added if_exists=True so the
    transaction do not fail for such deleted ports.

    Also update minimal version of "ovsdbapp" required for
    this.

    Depends-On: https://review.opendev.org/c/openstack/ovsdbapp/+/919696
    Depends-On: https://review.opendev.org/c/openstack/requirements/+/920062
    Closes-Bug: #2065701
    Related-Bug: #2060163
    Change-Id: Ie0ce420414f1e6056d9df2733b7446d02c89fcfc

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/neutron/+/920608

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/920609

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/920610

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/neutron/+/920621

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/920608
Committed: https://opendev.org/openstack/neutron/commit/5bdd0efb3970a52c60043f166bc728778ac3f395
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 5bdd0efb3970a52c60043f166bc728778ac3f395
Author: yatinkarel <email address hidden>
Date: Tue May 28 13:11:58 2024 +0530

    [stable only] Do not fail on missing logical router ports

    set_gateway_mtu runs for all the gateway ports for a network
    and if one of the ports get's deleted in meanwhile
    whole transaction fails.

    To handle this we need to add if_exists=True to the transaction
    but for that it needs to be supported in ovsdbapp. It's fixed
    in ovsdbapp with [1] but would require to bump ovsdbapp
    minimal version in requirements.txt which we normally don't
    do for stable branches.

    So using "update_lrouter_port" instead as that have the
    required option available. Before [2] that was only used
    but during the switch if_exists part was missed.

    [1] https://review.opendev.org/q/I56685478214aae7b6d3a2a3187297ad4eb1869a3
    [2] https://review.opendev.org/c/openstack/neutron/+/762695

    Closes-Bug: #2065701
    Related-Bug: #2060163
    Change-Id: I447990509cdea9830228d3bc92a97062cc57a472

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/920609
Committed: https://opendev.org/openstack/neutron/commit/da8f6ef4dd2172a203f261fff535bd3f372e1276
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit da8f6ef4dd2172a203f261fff535bd3f372e1276
Author: yatinkarel <email address hidden>
Date: Tue May 28 13:11:58 2024 +0530

    [stable only] Do not fail on missing logical router ports

    set_gateway_mtu runs for all the gateway ports for a network
    and if one of the ports get's deleted in meanwhile
    whole transaction fails.

    To handle this we need to add if_exists=True to the transaction
    but for that it needs to be supported in ovsdbapp. It's fixed
    in ovsdbapp with [1] but would require to bump ovsdbapp
    minimal version in requirements.txt which we normally don't
    do for stable branches.

    So using "update_lrouter_port" instead as that have the
    required option available. Before [2] that was only used
    but during the switch if_exists part was missed.

    [1] https://review.opendev.org/q/I56685478214aae7b6d3a2a3187297ad4eb1869a3
    [2] https://review.opendev.org/c/openstack/neutron/+/762695

    Closes-Bug: #2065701
    Related-Bug: #2060163
    Change-Id: I447990509cdea9830228d3bc92a97062cc57a472
    (cherry picked from commit 5bdd0efb3970a52c60043f166bc728778ac3f395)
    Conflicts:
            neutron/tests/unit/fake_resources.py

[OpenStack Infra (hudson-openstack)]

Change abandoned by "yatin <email address hidden>" on branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/919806
Reason: obsoleted by https://review.opendev.org/c/openstack/neutron/+/920609

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/920610
Committed: https://opendev.org/openstack/neutron/commit/cbfb4349a17b597290801358182450e63106b20f
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit cbfb4349a17b597290801358182450e63106b20f
Author: yatinkarel <email address hidden>
Date: Tue May 28 13:11:58 2024 +0530

    [stable only] Do not fail on missing logical router ports

    set_gateway_mtu runs for all the gateway ports for a network
    and if one of the ports get's deleted in meanwhile
    whole transaction fails.

    To handle this we need to add if_exists=True to the transaction
    but for that it needs to be supported in ovsdbapp. It's fixed
    in ovsdbapp with [1] but would require to bump ovsdbapp
    minimal version in requirements.txt which we normally don't
    do for stable branches.

    So using "update_lrouter_port" instead as that have the
    required option available. Before [2] that was only used
    but during the switch if_exists part was missed.

    [1] https://review.opendev.org/q/I56685478214aae7b6d3a2a3187297ad4eb1869a3
    [2] https://review.opendev.org/c/openstack/neutron/+/762695

    Closes-Bug: #2065701
    Related-Bug: #2060163
    Change-Id: I447990509cdea9830228d3bc92a97062cc57a472
    (cherry picked from commit 5bdd0efb3970a52c60043f166bc728778ac3f395)
    Conflicts:
            neutron/tests/unit/fake_resources.py

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/920621
Committed: https://opendev.org/openstack/neutron/commit/ece6a9a7acab20d5a39f54784427258d54b72cfd
Submitter: "Zuul (22348)"
Branch: unmaintained/zed

commit ece6a9a7acab20d5a39f54784427258d54b72cfd
Author: yatinkarel <email address hidden>
Date: Tue May 28 13:11:58 2024 +0530

    [stable only] Do not fail on missing logical router ports

    set_gateway_mtu runs for all the gateway ports for a network
    and if one of the ports get's deleted in meanwhile
    whole transaction fails.

    To handle this we need to add if_exists=True to the transaction
    but for that it needs to be supported in ovsdbapp. It's fixed
    in ovsdbapp with [1] but would require to bump ovsdbapp
    minimal version in requirements.txt which we normally don't
    do for stable branches.

    So using "update_lrouter_port" instead as that have the
    required option available. Before [2] that was only used
    but during the switch if_exists part was missed.

    [1] https://review.opendev.org/q/I56685478214aae7b6d3a2a3187297ad4eb1869a3
    [2] https://review.opendev.org/c/openstack/neutron/+/762695

    Closes-Bug: #2065701
    Related-Bug: #2060163
    Change-Id: I447990509cdea9830228d3bc92a97062cc57a472
    (cherry picked from commit 5bdd0efb3970a52c60043f166bc728778ac3f395)
    Conflicts:
            neutron/tests/unit/fake_resources.py

[yatin (yatinkarel)]

[Stable only] Missed some real test and hitting below error:-
2024-06-04 23:41:14.788 15 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-60337838-51d7-4247-a3dc-ebc9e674f9f3 - - - - - -] Running txn n=1 command(idx=3): UpdateLRouterPortCommand(_result=None, name=lrp-75477949-3d23-4d2d-8896-49ab58cb814f, columns={'gateway_mtu': '1312'}, if_exists=True) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89ESC[00m
2024-06-04 23:41:14.789 15 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-4a0c39f7-1960-4542-9ab8-9a1c78a0e6c9 a2eb22eff0f64608a74a59dd6998ec50 e86eed9be113477aae4860d2e6b1ff1f - - default default] Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/commands.py", line 416, in run_idl
    setattr(lrouter_port, col, val)

Trying to reproduce in CI https://review.opendev.org/c/openstack/devstack/+/919788, and then can test the upcoming a simple fix.
  File "/usr/lib64/python3.9/site-packages/ovs/db/idl.py", line 1419, in __setattr__
    column = self._table.columns[column_name]
KeyError: 'gateway_mtu'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/neutron/+/921437

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/921438

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/921440

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/neutron/+/921441

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/921437
Committed: https://opendev.org/openstack/neutron/commit/0d8cc09c4a5f652503c4ba9d968959b522aca980
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 0d8cc09c4a5f652503c4ba9d968959b522aca980
Author: yatinkarel <email address hidden>
Date: Thu Jun 6 16:17:07 2024 +0530

    [stable only] Fix KeyError in set_gateway_mtu

    Got missed in initial fixes[1], this patch fixes
    it.

    [1] https://review.opendev.org/q/I447990509cdea9830228d3bc92a97062cc57a472
    Closes-Bug: #2065701
    Related-Bug: #2060163

    Change-Id: Icdab45ab0873c003977e3f02277d267116002973

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/921438
Committed: https://opendev.org/openstack/neutron/commit/4c56dfded153b9c6412bafd65bd76d8e5d0bfe2c
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 4c56dfded153b9c6412bafd65bd76d8e5d0bfe2c
Author: yatinkarel <email address hidden>
Date: Thu Jun 6 16:17:07 2024 +0530

    [stable only] Fix KeyError in set_gateway_mtu

    Got missed in initial fixes[1], this patch fixes
    it.

    [1] https://review.opendev.org/q/I447990509cdea9830228d3bc92a97062cc57a472
    Closes-Bug: #2065701
    Related-Bug: #2060163

    Change-Id: Icdab45ab0873c003977e3f02277d267116002973
    (cherry picked from commit 0d8cc09c4a5f652503c4ba9d968959b522aca980)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/921440
Committed: https://opendev.org/openstack/neutron/commit/cd92d42cfa993d542d07ac7a05943b99deb8c208
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit cd92d42cfa993d542d07ac7a05943b99deb8c208
Author: yatinkarel <email address hidden>
Date: Thu Jun 6 16:17:07 2024 +0530

    [stable only] Fix KeyError in set_gateway_mtu

    Got missed in initial fixes[1], this patch fixes
    it.

    [1] https://review.opendev.org/q/I447990509cdea9830228d3bc92a97062cc57a472
    Closes-Bug: #2065701
    Related-Bug: #2060163

    Change-Id: Icdab45ab0873c003977e3f02277d267116002973
    (cherry picked from commit 0d8cc09c4a5f652503c4ba9d968959b522aca980)
    (cherry picked from commit 4c56dfded153b9c6412bafd65bd76d8e5d0bfe2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/921441
Committed: https://opendev.org/openstack/neutron/commit/763000d72b0644dc680a569d2307b30c795451eb
Submitter: "Zuul (22348)"
Branch: unmaintained/zed

commit 763000d72b0644dc680a569d2307b30c795451eb
Author: yatinkarel <email address hidden>
Date: Thu Jun 6 16:17:07 2024 +0530

    [stable only] Fix KeyError in set_gateway_mtu

    Got missed in initial fixes[1], this patch fixes
    it.

    [1] https://review.opendev.org/q/I447990509cdea9830228d3bc92a97062cc57a472
    Closes-Bug: #2065701
    Related-Bug: #2060163

    Change-Id: Icdab45ab0873c003977e3f02277d267116002973
    (cherry picked from commit 0d8cc09c4a5f652503c4ba9d968959b522aca980)
    (cherry picked from commit 4c56dfded153b9c6412bafd65bd76d8e5d0bfe2c)
    (cherry picked from commit cd92d42cfa993d542d07ac7a05943b99deb8c208)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 25.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 24.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 23.2.0 release.