Mechanism driver 'ovn' failed in create_port_precommit: AttributeError: 'NoneType' object has no attribute 'chassis_exists'

Bug #1929633 reported by Sam Wan
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Lucas Alvares Gomes

Bug Description

Error occurs in devstack@q-svc.
Seems this error arises after devstack 'Change default network backend driver to ML2/OVN', https://review.opendev.org/c/openstack/devstack/+/791436

========================================
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server [None req-80ce99c0-4be0-44c1-a62d-bea8619c6471 None None] Exception during message handling: neutron.plugins.ml2.common.exceptions.MechanismDriverError
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 241, in inner
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server self.force_reraise()
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server raise self.value
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154,in wrapper
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server self.force_reraise()
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server raise self.value
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142,in wrapper
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server self.force_reraise()
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server raise self.value
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/neutron/neutron/quota/resource_registry.py", line 9
5, in wrapper
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server ret_val = f(_self, context, *args, **kwargs)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/neutron/neutron/api/rpc/handlers/dhcp_rpc.py", line
 295, in create_dhcp_port
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return self._port_action(plugin, context, port, 'create_port')
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/neutron/neutron/api/rpc/handlers/dhcp_rpc.py", line
 115, in _port_action
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return p_utils.create_port(plugin, context, port)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/neutron_lib/plugins/utils.py", line 337, in create_port
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return core_plugin.create_port(
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/neutron/neutron/common/utils.py", line 685, in inner
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 217, in wrapped
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server self.force_reraise()
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server raise self.value
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154,
in wrapper
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server self.force_reraise()
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server raise self.value
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142,in wrapper
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server self.force_reraise()
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server raise self.value
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1457,in create_port
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server result, mech_context = self._create_port_db(context, port)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1447,in _create_port_db
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server self.mechanism_manager.create_port_precommit(mech_context)
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/neutron/neutron/plugins/ml2/managers.py", line 700, in create_port_precommit
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server self._call_on_drivers("create_port_precommit", context,
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/neutron/neutron/plugins/ml2/managers.py", line 493, in _call_on_drivers
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server raise ml2_exc.MechanismDriverError(May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server neutron.plugins.ml2.common.exceptions.MechanismDriverError
May 25 22:44:50 e2e-os-pflex101 neutron-server[2801827]: ERROR oslo_messaging.rpc.server
========================================

Revision history for this message
Sam Wan (sam-wan) wrote :
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

It is happening in Your local devstack env or in some gate job?
Can You also share with us Your devstack's local.conf file?

Changed in neutron:
importance: Undecided → Critical
Revision history for this message
Sam Wan (sam-wan) wrote :

It's happening in our CI. We're still using zuulv2 + jenkins + devstack-gate for CI jobs although we're working on migrating to latest zuul but not done yet.
Attached is our local.conf, please take a look if there's any misconfig.

Though I'm not sure why this issue happens, I think we'd better add a check before calling 'self._sb_ovn.chassis_exists' in https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#L656
some patch like this:
===============================
diff --git a/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py b/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py
index 229ea0f85b..2c129179f9 100644
--- a/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py
+++ b/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py
@@ -653,7 +653,8 @@ class OVNMechanismDriver(api.MechanismDriver):
                       port['id'])
             return False

- if not self._sb_ovn.chassis_exists(host):
+ if (hasattr(self._sb_ovn, 'chassis_exists') and
+ not self._sb_ovn.chassis_exists(host)):
             LOG.debug('No provisioning block for port %(port_id)s since no '
                       'OVN chassis for host: %(host)s',
                       {'port_id': port['id'], 'host': host})
===============================

thanks and regards
Sam

Revision history for this message
Sam Wan (sam-wan) wrote :
Revision history for this message
Jakub Libosvar (libosvar) wrote :

I think what is happening is that there is the DHCP agent calls core plugin to create DHCP port too early when ovn mechanism driver is still being initialized. The IDLs are instantiated in post fork and there is a wait() call that opens the mech driver API after IDLs are there.

The traceback means sb_idl is set to none and hence not instantiated.

Revision history for this message
Jakub Libosvar (libosvar) wrote :

Sam, can you please share the whole svc log? There could be a problem during the initialization and the error you pasted is just a consequence of IDLs not being instantiated well.

Changed in neutron:
status: New → In Progress
Revision history for this message
Sam Wan (sam-wan) wrote :
Revision history for this message
Sam Wan (sam-wan) wrote :

Hi Jakub,

New q-svc log with initialization part uploaded.
please take a look.
thanks and regards

Sam

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :
Revision history for this message
Sam Wan (sam-wan) wrote :
Download full text (18.3 KiB)

Hi Rodolfo,

Seems the patch does not work as expected.
Error occurs during q-svc initialization:
==========================================
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647271]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-f16b321f-4b90-4aef-abd2-9e0344f0f336 None None] Logical port df604210-4631-4a77-af1b-df6bba01de2d is not bound to a chassis
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647271]: DEBUG neutron.db.provisioning_blocks [None req-f16b321f-4b90-4aef-abd2-9e0344f0f336 None None] Could not find standard attr ID for object df604210-4631-4a77-af1b-df6bba01de2d. {{(pid=1647271) _get_standard_attr_id /opt/stack/new/neutron/neutron/db/provisioning_blocks.py:175}}
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647271]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-f16b321f-4b90-4aef-abd2-9e0344f0f336 None None] OVN reports status down for port: 4ac114a9-1071-46b8-994e-5ed7317eb5ca
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event [None req-29eb3377-b954-43a5-9901-8e33dec7804e None None] Unexpected exception in notify_loop: neutron_lib.exceptions.PortNotFound: Port 161bbf58-db6f-4e81-8d1f-3a2123456770 could not be found.
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event Traceback (most recent call last):
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event File "/opt/stack/new/neutron/neutron/db/db_base_plugin_common.py", line 289, in _get_port
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event port = model_query.get_by_id(context, models_v2.Port, id)
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/model_query.py", line 161, in get_by_id
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event return query.filter(model.id == object_id).one()
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3500, in one
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event raise orm_exc.NoResultFound("No row was found for one()")
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event sqlalchemy.orm.exc.NoResultFound: No row was found for one()
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event During handling of the above exception, another exception occurred:
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event Traceback (most recent call last):
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/event.py", line 159, in notify_loop
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event match.run(event, row, updates)
May 31 02:36:53 e2e-os-pstorefc103 neutron-server[1647270]: ERROR ovsdbapp.event File "/opt/stac...

Revision history for this message
Sam Wan (sam-wan) wrote :
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello Sam:

In the logs you provided in c#11, the error is not related to the bug one. What you are hitting there is an inconsistency between the network backend DB and Neutron DB. OVN has a port, "9b140978-cb71-445b-832c-34e858dd2cf1", that is not registered in Neutron. This port seems to be the DHCP port for "dhcp7941d1a9-befe-5667-84ed-3c0878d55cec-997d75b0-30df-4cb4-a11f-485cb66a9940". This is the DHCP for network "997d75b0-30df-4cb4-a11f-485cb66a9940".

As reported before, this network is in OVN but not in Neutron:
May 31 02:58:47 e2e-os-pstorefc103 neutron-server[1747202]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-97734830-2782-4c0c-85f7-e019f73998ae None None] Network found in OVN but not in Neutron, network_id=neutron-997d75b0-30df-4cb4-a11f-485cb66a9940

You need to fix the inconsistencies between your network backend and Neutron. If this is a testing environment, delete the OVN registers (LS and LSP related). Then create, from Neutron, those networks and ports again.

Regards.

Revision history for this message
Sam Wan (sam-wan) wrote :

Hi Rodolfo,

Thanks for the update.
I'm not a neutron nor an OVN expert, so could you please be more specific on how to delete the OVN registers and create the networks and ports again?
A guide or example would be helpful.

best regards
Sam

Revision history for this message
Bartosz Bezak (bbezak) wrote :

this also affects recent stable/victoria deployments - is there any traction on this bug? seems pretty critical - https://review.opendev.org/c/openstack/neutron/+/781555

Changed in neutron:
assignee: nobody → Lucas Alvares Gomes (lucasagomes)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/800780

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/800781

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/800782

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/ussuri)

Change abandoned by "Lucas Alvares Gomes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/800782
Reason: Will cherry-pick from wallaby

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/victoria)

Change abandoned by "Lucas Alvares Gomes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/800781
Reason: Will cherry-pick from wallaby

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/800783

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/800784

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/wallaby)

Change abandoned by "Lucas Alvares Gomes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/800780
Reason: We've changed the master branch approach and now the patch is backportable

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/victoria)

Change abandoned by "Lucas Alvares Gomes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/800784
Reason: We've changed the master branch approach and now the patch is backportable

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/ussuri)

Change abandoned by "Lucas Alvares Gomes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/800783
Reason: We've changed the master branch approach and now the patch is backportable

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/800892

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/800893

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/800894

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/800679
Committed: https://opendev.org/openstack/neutron/commit/e5f19a29dcfb70ea08fa39267bc4aecd44bec2eb
Submitter: "Zuul (22348)"
Branch: master

commit e5f19a29dcfb70ea08fa39267bc4aecd44bec2eb
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Jul 14 16:46:47 2021 +0100

    [OVN] Fix ML2/OVN + Neutron DHCP agent use case

    In an attempt to minimize the number of connections to the OVSDBs the
    ML2/OVN driver stopped connecting to the OVN dbs if the Neutron worker
    was a RpcWorker but, that introduced a regression for those using
    ML2/OVN + Neutron DHCP agent (for baremetal provisioning).

    Because the RpcWorker didn't have a connection to the OVN database the
    Neutron DHCP agent wasn't able to create the DHCP port and failed with:

    2021-07-13 13:36:18.116 17 ERROR neutron.plugins.ml2.managers if not
    self._sb_ovn.chassis_exists(host):
    2021-07-13 13:36:18.116 17 ERROR neutron.plugins.ml2.managers
    AttributeError: 'NoneType' object has no attribute 'chassis_exists'

    This patch adds the RpcWorker back to the list of workers that should
    connect to the OVN databases.

    Change-Id: I454f3b52376a02f6cc9ca9124083295631bd03d5
    Closes-Bug: #1929633
    Signed-off-by: Lucas Alvares Gomes <email address hidden>

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/800892
Committed: https://opendev.org/openstack/neutron/commit/335614be5fdc4d9aa562cb66806a5429357e338f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 335614be5fdc4d9aa562cb66806a5429357e338f
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Jul 14 16:46:47 2021 +0100

    [OVN] Fix ML2/OVN + Neutron DHCP agent use case

    In an attempt to minimize the number of connections to the OVSDBs the
    ML2/OVN driver stopped connecting to the OVN dbs if the Neutron worker
    was a RpcWorker but, that introduced a regression for those using
    ML2/OVN + Neutron DHCP agent (for baremetal provisioning).

    Because the RpcWorker didn't have a connection to the OVN database the
    Neutron DHCP agent wasn't able to create the DHCP port and failed with:

    2021-07-13 13:36:18.116 17 ERROR neutron.plugins.ml2.managers if not
    self._sb_ovn.chassis_exists(host):
    2021-07-13 13:36:18.116 17 ERROR neutron.plugins.ml2.managers
    AttributeError: 'NoneType' object has no attribute 'chassis_exists'

    This patch adds the RpcWorker back to the list of workers that should
    connect to the OVN databases.

    Change-Id: I454f3b52376a02f6cc9ca9124083295631bd03d5
    Closes-Bug: #1929633
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit e5f19a29dcfb70ea08fa39267bc4aecd44bec2eb)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/800894
Committed: https://opendev.org/openstack/neutron/commit/172a628c5a59a8db98aed62418ebf64b8b10b392
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 172a628c5a59a8db98aed62418ebf64b8b10b392
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Jul 14 16:46:47 2021 +0100

    [OVN] Fix ML2/OVN + Neutron DHCP agent use case

    In an attempt to minimize the number of connections to the OVSDBs the
    ML2/OVN driver stopped connecting to the OVN dbs if the Neutron worker
    was a RpcWorker but, that introduced a regression for those using
    ML2/OVN + Neutron DHCP agent (for baremetal provisioning).

    Because the RpcWorker didn't have a connection to the OVN database the
    Neutron DHCP agent wasn't able to create the DHCP port and failed with:

    2021-07-13 13:36:18.116 17 ERROR neutron.plugins.ml2.managers if not
    self._sb_ovn.chassis_exists(host):
    2021-07-13 13:36:18.116 17 ERROR neutron.plugins.ml2.managers
    AttributeError: 'NoneType' object has no attribute 'chassis_exists'

    This patch adds the RpcWorker back to the list of workers that should
    connect to the OVN databases.

    Change-Id: I454f3b52376a02f6cc9ca9124083295631bd03d5
    Closes-Bug: #1929633
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit e5f19a29dcfb70ea08fa39267bc4aecd44bec2eb)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/800893
Committed: https://opendev.org/openstack/neutron/commit/d326ec6b9dec9c1f5cfbba7f7183840d3eaee226
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit d326ec6b9dec9c1f5cfbba7f7183840d3eaee226
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Jul 14 16:46:47 2021 +0100

    [OVN] Fix ML2/OVN + Neutron DHCP agent use case

    In an attempt to minimize the number of connections to the OVSDBs the
    ML2/OVN driver stopped connecting to the OVN dbs if the Neutron worker
    was a RpcWorker but, that introduced a regression for those using
    ML2/OVN + Neutron DHCP agent (for baremetal provisioning).

    Because the RpcWorker didn't have a connection to the OVN database the
    Neutron DHCP agent wasn't able to create the DHCP port and failed with:

    2021-07-13 13:36:18.116 17 ERROR neutron.plugins.ml2.managers if not
    self._sb_ovn.chassis_exists(host):
    2021-07-13 13:36:18.116 17 ERROR neutron.plugins.ml2.managers
    AttributeError: 'NoneType' object has no attribute 'chassis_exists'

    This patch adds the RpcWorker back to the list of workers that should
    connect to the OVN databases.

    Change-Id: I454f3b52376a02f6cc9ca9124083295631bd03d5
    Closes-Bug: #1929633
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit e5f19a29dcfb70ea08fa39267bc4aecd44bec2eb)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/781555
Committed: https://opendev.org/openstack/neutron/commit/90980f496cfa3cc5df1c93cf834a44f33d3f1f6f
Submitter: "Zuul (22348)"
Branch: master

commit 90980f496cfa3cc5df1c93cf834a44f33d3f1f6f
Author: Terry Wilson <email address hidden>
Date: Thu Mar 18 15:01:07 2021 -0500

    Add wait for the post-fork event to nb/sb objects

    It is possible for events from the nb/sb to fire before the opposite
    db connection is made. These events can call back into driver code
    which tries to access the other db before it is connected.

    Closes-Bug: #1929197
    Closes-Bug: #1928794
    Closes-Bug: #1929633
    Change-Id: If947581b90ced42981c4611c32de8f428a052c69

tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/805768

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/805769

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/victoria)

Change abandoned by "Mark Goddard <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/805769
Reason: Seems this fix was sufficient, but not yet in RDO victoria: https://github.com/openstack/neutron/commit/d326ec6b9dec9c1f5cfbba7f7183840d3eaee226

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/wallaby)

Change abandoned by "Mark Goddard <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/805768
Reason: Seems this fix was sufficient, but not yet in RDO victoria: https://github.com/openstack/neutron/commit/d326ec6b9dec9c1f5cfbba7f7183840d3eaee226

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.4.1

This issue was fixed in the openstack/neutron 16.4.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.2.1

This issue was fixed in the openstack/neutron 17.2.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.1.1

This issue was fixed in the openstack/neutron 18.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.0.0.0rc1

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/805768
Committed: https://opendev.org/openstack/neutron/commit/3139bbcbe6b3747ce432901a5a316d93f8f168fb
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3139bbcbe6b3747ce432901a5a316d93f8f168fb
Author: Terry Wilson <email address hidden>
Date: Thu Mar 18 15:01:07 2021 -0500

    Add wait for the post-fork event to nb/sb objects

    It is possible for events from the nb/sb to fire before the opposite
    db connection is made. These events can call back into driver code
    which tries to access the other db before it is connected.

    Conflicts:
            neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py
            neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py
            neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py
            neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py
            neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py
            neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py

    Closes-Bug: #1929197
    Closes-Bug: #1928794
    Closes-Bug: #1929633
    Change-Id: If947581b90ced42981c4611c32de8f428a052c69
    (cherry picked from commit 90980f496cfa3cc5df1c93cf834a44f33d3f1f6f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.4.0

This issue was fixed in the openstack/neutron 18.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn train-eol

This issue was fixed in the openstack/networking-ovn train-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.