neutron

[ovn] The 'ovn_client' of the L3 plugin is occasionally NoneType

Bug #2061350 reported by Liu Xie

This bug report will be marked for expiration in 32 days if no further activity occurs. (find out why)

8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Incomplete
Undecided
Unassigned

Bug Description

I don't know how to reproduce it, and we have encountered it twice. The errors are as follows:

2024-04-01 10:53:46.185 252 INFO neutron.wsgi [req-3ed42a43-8895-4ad3-b675-af89c6d47631 9862b8ef431c43489ada1bb12f97be59 c2c9335342874b758aa6af2166fca837 - default default] 192.168.10.5 "GET /v2.0/ports?device_id=ae622d1a-416f-4676-b340-be0af8e0626a HTTP/1.1" status: 200 len: 7306 time: 0.0576138
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource [req-e0dcd168-ad4e-43d8-b88c-47393077aa64 b255e17db7c5499c8c8e45e363ae3b47 12edc48812c94bc3a861bc8bafcf8fa6 - default default] index failed: No details.: AttributeError: 'NoneType' object has no attribute 'get_lrouter'
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource Traceback (most recent call last):
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource result = method(request=request, **args)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/base.py", line 369, in index
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource return self._items(request, True, parent_id)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/base.py", line 304, in _items
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource obj_list = obj_getter(request.context, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 217, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource return method(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron/db/l3_db.py", line 601, in get_routers
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource page_reverse=page_reverse)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 319, in get_collection
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource for c in query
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 319, in <listcomp>
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource for c in query
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron/db/l3_db.py", line 231, in _make_router_dict
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource resource_extend.apply_funcs(l3_apidef.ROUTERS, res, router)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/timeutils.py", line 383, in wrapper
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource result = func(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/resource_extend.py", line 91, in apply_funcs
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource resolved_func(response, db_object)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron/db/availability_zone/router.py", line 40, in _add_az_to_response
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource l3_plugin.get_router_availability_zones(router_db))
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron/services/ovn_l3/plugin.py", line 516, in get_router_availability_zones
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource lr = self._ovn.get_lrouter(router['id'])
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource AttributeError: 'NoneType' object has no attribute 'get_lrouter'
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource

I suspect that at the completion of ovn_l3 plugin instantiation, the ovn client has not yet been fully instantiated.

neutron version: wallaby.

See original description
Tags: ovn
Liu Xie (liushy)
description: updated
tags: added: ovn
Revision history for this message
Brian Haley (brian-haley) wrote :

Wallaby is very old, can you reproduce on a newer version?

At first look it looks related to this:

https://bugs.launchpad.net/neutron/+bug/1997970

Which was fixed by https://review.opendev.org/c/openstack/neutron/+/867206 in 2023.1

Changed in neutron:
status: New → Incomplete
Revision history for this message
Liu Xie (liushy) wrote :

@Brian
Thanks, I will try it!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.