neutron

[L3] router HA port concurrently deleting

Bug #1906375 reported by LIU Yulong
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Low
LIU Yulong

Bug Description

Router HA port may be deleted concurrently while the plugin is trying to update. Then a PortNotFound exception raised.
ERROR was found at rocky deployment, but the master branch has the same code.

2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server [req-86012433-ab6b-41f5-bbba-411ec3e1d973 - - - - -] Exception during message handling: PortNotFound: Port 3f838c59-e84a-49de-a381-f3328d47a69f could not be found.
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 93, in update_all_ha_network_port_statuses
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server context, p, {'port': {'status': constants.PORT_STATUS_DOWN}})
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 632, in inner
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1420, in update_port
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server original_port = self.get_port(context, id)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 993, in wrapper
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return fn(*args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1462, in get_port
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server port = self._get_port(context, id)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 264, in _get_port
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server raise n_exc.PortNotFound(port_id=id)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server PortNotFound: Port 3f838c59-e84a-49de-a381-f3328d47a69f could not be found.
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server
2020-12-01 10:53:03.921 62076 ERROR oslo_messaging.rpc.server [req-3822c366-0cb7-46c3-ad6b-b50081da3dc8 - - - - -] Exception during message handling: PortNotFound: Port 80911dcd-06be-456a-9054-4bf9f40680aa could not be found.

Tags: l3-ha
Revision history for this message
LIU Yulong (dragon889) wrote :

Patch works:
https://review.opendev.org/c/openstack/neutron/+/764913

Bernard Cafarelli (bcafarel)
tags: added: l3-ha
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.0.0.0rc1

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

Revision history for this message
Edward Hope-Morley (hopem) wrote :

This fix for this is merged [1] and released to ill update the status accordingly.

[1] https://github.com/openstack/neutron/commit/91eb3d8346a8964aa046d1e016d571056de868de

Changed in neutron:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.