[ovn] OVSDB Error ' references nonexistent' occurs when updating one router.

[Liu Xie]

Hi folks, I encountered an issue when updating a router, but I'm not sure how to reproduce it. Does anyone have any opinions? The phenomenon is as follows:

When i updating one router:
neutron router-update --name test 22d98f75-063e-4413-b40c-69d1db3d9849

The error logs are:

2024-07-16 16:11:37.651 146 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}^[[00m
2024-07-16 16:11:37.651 146 ERROR ovsdbapp.backend.ovs_idl.transaction [req-c855bc22-2d9b-48c4-964b-eb0579a75f2c f743596e02534618b855e9a1f8dd1640 a4912dcbde8e4533b9f7128cabf7ab72 - default default] Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}
^[[00m
2024-07-16 16:11:37.651 146 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [req-c855bc22-2d9b-48c4-964b-eb0579a75f2c f743596e02534618b855e9a1f8dd1640 a4912dcbde8e4533b9f7128cabf7ab72 - default default] Unable to update router 22d98f75-063e-4413-b40c-69d1db3d9849. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}^[[00m
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin [req-c855bc22-2d9b-48c4-964b-eb0579a75f2c f743596e02534618b855e9a1f8dd1640 a4912dcbde8e4533b9f7128cabf7ab72 - default default] Unable to update lrouter for 22d98f75-063e-4413-b40c-69d1db3d9849: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin Traceback (most recent call last):
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/neutron/services/ovn_l3/plugin.py", line 219, in update_router
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin original_router)
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1612, in update_router
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin 'error': e})
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin self.force_reraise()
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin raise self.value
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1594, in update_router
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin self._qos_driver.update_router(txn, new_router)
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin next(self.gen)
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 269, in transaction
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin yield t
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin next(self.gen)
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 110, in transaction
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin del self._nested_txns_map[cur_thread_id]
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 61, in __exit__
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin self.result = self.commit()
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin raise result.ex
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin txn.results.put(txn.do_commit())
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin raise RuntimeError(msg)
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin ^[[00m

neutron version: wallaby

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Liu:

Please provide:
* The Neutron version you are using.
* The router description (openstack router show <router_id>)
* The OVN router description (ovn-nbctl list logical_router <router_id>)
* The OVN router static routes referred in the logical router (ovn-nbctl list Logical_Router_Static_Route <IDs of the LRSR>)

Regards.

[Liu Xie (liushy)]

neutron version: wallaby

router info:
openstack router show 22d98f75-063e-4413-b40c-69d1db3d9849
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | default-az |
| availability_zones | default-az |
| created_at | 2024-07-11T07:09:40Z |
| description | |
| external_gateway_info | {"network_id": "6270cd06-db0d-4949-baf8-619b13e261b7", "enable_snat": false, "external_fixed_ips": [{"subnet_id": "ea764118-cb35-4f3d-81b3-ccd6383d333f", "ip_address": |
| | "172.27.252.19"}]} |
| gw_port_id | 72084cd2-8de3-4046-959e-56a5f4558e80 |
| id | 22d98f75-063e-4413-b40c-69d1db3d9849 |
| name | router2 |
| project_id | a4912dcbde8e4533b9f7128cabf7ab72 |
| project_id | a4912dcbde8e4533b9f7128cabf7ab72 ...

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Liu:

First of all, Wallaby is an unmaintained branch so I would recommend you to upgrade to a newer version.

Second, this code path is present only in Zed. The gateway IP QoS was implemented in [1] and the first version having this code is Zed. Please, check what version are you really using.

What is "Logical_Router_Static_Route" row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d? It is not present in the Logical_Router register thus I don't understand why this is happening. In the LR we have only two LRSR ([44a69276-e70d-4bf2-868e-6426d7087099, 865cca52-e270-4f0c-a16e-3cd04962e0b0]) that are actually present in the DB.

Questions:
* Every time you perform an action on this router, do you have the same exception?
* Is this happening for other routers?

Regards.

[1]https://review.opendev.org/c/openstack/neutron/+/833455

[Liu Xie (liushy)]

Hi Rodolfo,

We have cherry-picked this patch [1] to our Wallaby version of Neutron so that you can see QoS information.

The issue occurred accidentally and has also happened on another router. I found the same error logs in issue [2].

It might be an issue with OVSDB, but this is just my guess.

[1]https://review.opendev.org/c/openstack/neutron/+/833455
[2]https://bugs.launchpad.net/neutron/+bug/1879950

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hi Liu:

Sorry, we can't help you on this issue. You are using an old branch and manually modified. We can't provide support on this scenario.

Regards.