[ovn] OVSDB Error ' references nonexistent' occurs when updating one router.

Bug #2073326 reported by Liu Xie
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Won't Fix
Undecided
Unassigned

Bug Description

Hi folks, I encountered an issue when updating a router, but I'm not sure how to reproduce it. Does anyone have any opinions? The phenomenon is as follows:

When i updating one router:
neutron router-update --name test 22d98f75-063e-4413-b40c-69d1db3d9849

The error logs are:

2024-07-16 16:11:37.651 146 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}^[[00m
2024-07-16 16:11:37.651 146 ERROR ovsdbapp.backend.ovs_idl.transaction [req-c855bc22-2d9b-48c4-964b-eb0579a75f2c f743596e02534618b855e9a1f8dd1640 a4912dcbde8e4533b9f7128cabf7ab72 - default default] Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}
^[[00m
2024-07-16 16:11:37.651 146 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [req-c855bc22-2d9b-48c4-964b-eb0579a75f2c f743596e02534618b855e9a1f8dd1640 a4912dcbde8e4533b9f7128cabf7ab72 - default default] Unable to update router 22d98f75-063e-4413-b40c-69d1db3d9849. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}^[[00m
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin [req-c855bc22-2d9b-48c4-964b-eb0579a75f2c f743596e02534618b855e9a1f8dd1640 a4912dcbde8e4533b9f7128cabf7ab72 - default default] Unable to update lrouter for 22d98f75-063e-4413-b40c-69d1db3d9849: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin Traceback (most recent call last):
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/neutron/services/ovn_l3/plugin.py", line 219, in update_router
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin original_router)
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1612, in update_router
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin 'error': e})
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin self.force_reraise()
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin raise self.value
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1594, in update_router
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin self._qos_driver.update_router(txn, new_router)
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin next(self.gen)
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 269, in transaction
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin yield t
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin next(self.gen)
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 110, in transaction
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin del self._nested_txns_map[cur_thread_id]
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 61, in __exit__
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin self.result = self.commit()
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin raise result.ex
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin txn.results.put(txn.do_commit())
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin raise RuntimeError(msg)
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row f4e61e8d-2225-4aa3-a42a-1dd9b0b219ea references nonexistent row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d in table Logical_Router_Static_Route.","error":"referential integrity violation"}
2024-07-16 16:11:37.652 146 ERROR neutron.services.ovn_l3.plugin ^[[00m

neutron version: wallaby

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello Liu:

Please provide:
* The Neutron version you are using.
* The router description (openstack router show <router_id>)
* The OVN router description (ovn-nbctl list logical_router <router_id>)
* The OVN router static routes referred in the logical router (ovn-nbctl list Logical_Router_Static_Route <IDs of the LRSR>)

Regards.

Changed in neutron:
status: New → Incomplete
Revision history for this message
Liu Xie (liushy) wrote :
Download full text (5.9 KiB)

neutron version: wallaby

router info:
openstack router show 22d98f75-063e-4413-b40c-69d1db3d9849
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | default-az |
| availability_zones | default-az |
| created_at | 2024-07-11T07:09:40Z |
| description | |
| external_gateway_info | {"network_id": "6270cd06-db0d-4949-baf8-619b13e261b7", "enable_snat": false, "external_fixed_ips": [{"subnet_id": "ea764118-cb35-4f3d-81b3-ccd6383d333f", "ip_address": |
| | "172.27.252.19"}]} |
| gw_port_id | 72084cd2-8de3-4046-959e-56a5f4558e80 |
| id | 22d98f75-063e-4413-b40c-69d1db3d9849 |
| name | router2 |
| project_id | a4912dcbde8e4533b9f7128cabf7ab72 |
| project_id | a4912dcbde8e4533b9f7128cabf7ab72 ...

Read more...

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello Liu:

First of all, Wallaby is an unmaintained branch so I would recommend you to upgrade to a newer version.

Second, this code path is present only in Zed. The gateway IP QoS was implemented in [1] and the first version having this code is Zed. Please, check what version are you really using.

What is "Logical_Router_Static_Route" row 3652a4cb-7fac-4811-a4cc-d812a6d81c3d? It is not present in the Logical_Router register thus I don't understand why this is happening. In the LR we have only two LRSR ([44a69276-e70d-4bf2-868e-6426d7087099, 865cca52-e270-4f0c-a16e-3cd04962e0b0]) that are actually present in the DB.

Questions:
* Every time you perform an action on this router, do you have the same exception?
* Is this happening for other routers?

Regards.

[1]https://review.opendev.org/c/openstack/neutron/+/833455

Revision history for this message
Liu Xie (liushy) wrote :

Hi Rodolfo,

We have cherry-picked this patch [1] to our Wallaby version of Neutron so that you can see QoS information.

The issue occurred accidentally and has also happened on another router. I found the same error logs in issue [2].

It might be an issue with OVSDB, but this is just my guess.

[1]https://review.opendev.org/c/openstack/neutron/+/833455
[2]https://bugs.launchpad.net/neutron/+bug/1879950

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hi Liu:

Sorry, we can't help you on this issue. You are using an old branch and manually modified. We can't provide support on this scenario.

Regards.

Changed in neutron:
status: Incomplete → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.