[OVN] DHCP agent fails to update network

[Piotr Parczewski]

In latest Victoria build, DHCP agent is unable to update networks with OVN driver:

2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent [req-e8c3cd96-2a53-4ea8-b2eb-4de64af3600c - - - - -] Unable to enable dhcp for c9caf247-edb4-4282-a253-b948de5f9da7.: oslo_messaging.rpc.client.RemoteError: Remote error: MechanismDriverError
['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 311, in update_dhcp_port\n return self._port_action(plugin, context, port, \'update_port\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 101, in _port_action\n return plugin.update_port(context, port[\'id\'], port)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 681, in inner\n return f(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped\n return method(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1728, in update_port\n self.mechanism_manager.update_port_precommit(mech_context)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 731, in update_port_precommit\n raise_db_retriable=True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 495, in _call_on_drivers\n errors=errors\n', 'neutron.plugins.ml2.common.exceptions.MechanismDriverError\n'].
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 227, in call_driver
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 266, in enable
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 697, in wait_until_true
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent while not predicate():
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 278, in _enable
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1676, in setup
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent self._cleanup_stale_devices(network, dhcp_port=None)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent self.force_reraise()
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent six.reraise(self.type_, self.value, self.tb)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent raise value
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1671, in setup
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent port = self.setup_dhcp_port(network)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1610, in setup_dhcp_port
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent dhcp_port = setup_method(network, device_id, dhcp_subnets)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1534, in _setup_reserved_dhcp_port
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent 'device_id': device_id}})
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 894, in update_dhcp_port
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent port_id=port_id, port=port, host=self.host)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/rpc.py", line 157, in call
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent return self._original_context.call(ctxt, method, **kwargs)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent transport_options=self.transport_options)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent transport_options=transport_options)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 682, in send
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent transport_options=transport_options)
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 672, in _send
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent raise result
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent oslo_messaging.rpc.client.RemoteError: Remote error: MechanismDriverError
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 311, in update_dhcp_port\n return self._port_action(plugin, context, port, \'update_port\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 101, in _port_action\n return plugin.update_port(context, port[\'id\'], port)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 681, in inner\n return f(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped\n return method(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1728, in update_port\n self.mechanism_manager.update_port_precommit(mech_context)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 731, in update_port_precommit\n raise_db_retriable=True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 495, in _call_on_drivers\n errors=errors\n', 'neutron.plugins.ml2.common.exceptions.MechanismDriverError\n'].
2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent

Neutron:

2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers [req-e8c3cd96-2a53-4ea8-b2eb-4de64af3600c - - - - -] Mechanism driver 'ovn' failed in update_port_precommit: AttributeError: 'NoneType' object has no attribute 'chassis_exists'
2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers
2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 695, in update_port_precommit
2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers context.original_host):
2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 616, in _is_port_provisioning_required
2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers if not self._sb_ovn.chassis_exists(host):
2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers AttributeError: 'NoneType' object has no attribute 'chassis_exists'
2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server [req-e8c3cd96-2a53-4ea8-b2eb-4de64af3600c - - - - -] Exception during message handling: neutron.plugins.ml2.common.exceptions.MechanismDriverError
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 311, in update_dhcp_port
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return self._port_action(plugin, context, port, 'update_port')
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 101, in _port_action
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return plugin.update_port(context, port['id'], port)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 681, in inner
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1728, in update_port
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.mechanism_manager.update_port_precommit(mech_context)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 731, in update_port_precommit
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise_db_retriable=True)
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 495, in _call_on_drivers
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server errors=errors
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server neutron.plugins.ml2.common.exceptions.MechanismDriverError
2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server

This problem does not exist in earlier builds (>1 month) of Neutron server.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Piotr:

With OVN driver, DHCP and L3 agents are not necessary. OVN backend provides network provisioning (IP assignation, DNS configuration) and routing services. You should disable DHCP and L3 agents.

Regards.

[Bernard Cafarelli (bcafarel)]

Indeed, can you check and confirm after removing these agents? If it still happens and networks are visibly not updated correctly (VMs not getting IPs, etc, ...), can you give details on the setup: simple test setup, or large scale deployment, etc...

[Piotr Parczewski (parczewski)]

Thank you both for replying; this is mixed bare metal and hypervisors environment. AFAIK, Neutron would not be able to create proper baremetal type port in OVN with DHCP options, hence the agents in place.

Putting that aside, I think the core problem is Neutron's handling of SB DB after recent changes in stable/victoria - simply "downgrading" to a previous build solves the issue immediately.

Once again, for better readability: http://paste.openstack.org/show/805541/

[Piotr Parczewski (parczewski)]

Re-opened as a new bug: https://bugs.launchpad.net/neutron/+bug/1929197

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

This patch is related to this bug: https://review.opendev.org/c/openstack/neutron/+/781555

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/781555
Committed: https://opendev.org/openstack/neutron/commit/90980f496cfa3cc5df1c93cf834a44f33d3f1f6f
Submitter: "Zuul (22348)"
Branch: master

commit 90980f496cfa3cc5df1c93cf834a44f33d3f1f6f
Author: Terry Wilson <email address hidden>
Date: Thu Mar 18 15:01:07 2021 -0500

    Add wait for the post-fork event to nb/sb objects

    It is possible for events from the nb/sb to fire before the opposite
    db connection is made. These events can call back into driver code
    which tries to access the other db before it is connected.

    Closes-Bug: #1929197
    Closes-Bug: #1928794
    Closes-Bug: #1929633
    Change-Id: If947581b90ced42981c4611c32de8f428a052c69

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/805768

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/805769

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Mark Goddard <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/805769
Reason: Seems this fix was sufficient, but not yet in RDO victoria: https://github.com/openstack/neutron/commit/d326ec6b9dec9c1f5cfbba7f7183840d3eaee226

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Mark Goddard <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/805768
Reason: Seems this fix was sufficient, but not yet in RDO victoria: https://github.com/openstack/neutron/commit/d326ec6b9dec9c1f5cfbba7f7183840d3eaee226

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/805768
Committed: https://opendev.org/openstack/neutron/commit/3139bbcbe6b3747ce432901a5a316d93f8f168fb
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3139bbcbe6b3747ce432901a5a316d93f8f168fb
Author: Terry Wilson <email address hidden>
Date: Thu Mar 18 15:01:07 2021 -0500

    Add wait for the post-fork event to nb/sb objects

    It is possible for events from the nb/sb to fire before the opposite
    db connection is made. These events can call back into driver code
    which tries to access the other db before it is connected.

    Conflicts:
            neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py
            neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py
            neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py
            neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py
            neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py
            neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py

    Closes-Bug: #1929197
    Closes-Bug: #1928794
    Closes-Bug: #1929633
    Change-Id: If947581b90ced42981c4611c32de8f428a052c69
    (cherry picked from commit 90980f496cfa3cc5df1c93cf834a44f33d3f1f6f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.4.0 release.