Deploying neutron with httpd with ovn driver fails with the 500 with the below traceback[1] when creating networks.
'openstack network agent list' does not list any controller agents or metadata agents.
Looks like ovn driver pre_fork_initialize/post_fork_initialize are not triggered that initializes ovn db connections.
neutron.conf
-------------
....
auth_strategy=keystone
core_plugin=ml2
host=oc0-controller-0.mydomain.tld
dns_domain=openstacklocal
dhcp_agent_notification=True
allow_overlapping_ips=True
global_physnet_mtu=1500
vlan_transparent=False
service_plugins=qos,ovn-router,trunk,segments,port_forwarding,log
l3_ha=False
max_l3_agents_per_router=3
api_workers=2
rpc_workers=1
router_scheduler_driver=neutron.scheduler.l3_agent_scheduler.ChanceScheduler
router_distributed=False
enable_dvr=False
allow_automatic_l3agent_failover=True
ml2_conf.ini
------------
[ml2]
type_drivers=geneve,vxlan,vlan,flat
tenant_network_types=geneve,vlan
mechanism_drivers=ovn
path_mtu=0
extension_drivers=qos,port_security,dns
overlay_ip_version=4
[ml2_type_geneve]
max_header_size=38
vni_ranges=1:65536
[ml2_type_vxlan]
vxlan_group=224.0.0.1
vni_ranges=1:65536
[ml2_type_vlan]
network_vlan_ranges=datacentre:1:1000
[ml2_type_flat]
flat_networks=datacentre
[ovn]
ovn_nb_connection=tcp:172.16.13.9:6641
ovn_sb_connection=tcp:172.16.13.9:6642
ovsdb_connection_timeout=180
neutron_sync_mode=log
ovn_metadata_enabled=True
additional_worker_classes_with_ovn_idl=[MaintenanceWorker, RpcWorker]
enable_distributed_floating_ip=True
dns_servers=
ovn_emit_need_to_frag=False
[1]
2021-07-26 10:52:26.530 19 DEBUG neutron.api.v2.base [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] Request body: {'network': {'name': 'test_net', 'admin_state_up': True}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:729
2021-07-26 10:52:26.532 19 INFO neutron.quota [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] Loaded quota_driver: <neutron.db.quota.driver_nolock.DbQuotaNoLockDriver object at 0x7f8daf3696d8>.
2021-07-26 10:52:26.560 19 DEBUG neutron.pecan_wsgi.hooks.quota_enforcement [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] Made reservation on behalf of 20ad786fb60d459a9ac43bea8623d8b3 for: {'network': 1} before /usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/quota_enforcement.py:55
2021-07-26 10:52:26.615 19 DEBUG neutron.policy [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] Loaded default policies from ['neutron'] under neutron.policies entry points register_rules /usr/lib/python3.6/site-packages/neutron/policy.py:75
2021-07-26 10:52:26.671 19 DEBUG neutron_lib.callbacks.manager [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] Notify callbacks ['neutron.plugins.ml2.plugin.SecurityGroupDbMixin._ensure_default_security_group_handler--9223372036846591125'] for network, before_create _notify_loop /usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193
2021-07-26 10:52:26.732 19 DEBUG neutron.plugins.ml2.drivers.helpers [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] geneve segment allocate from pool success with {'geneve_vni': 32248} allocate_partially_specified_segment /usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/helpers.py:155
2021-07-26 10:52:26.747 19 DEBUG neutron_lib.callbacks.manager [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] Notify callbacks ['neutron.services.segments.db._add_segment_host_mapping_for_segment--9223363271446231241', 'neutron.plugins.ml2.plugin.Ml2Plugin._handle_segment_change--9223372036848473802'] for segment, precommit_create _notify_loop /usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193
2021-07-26 10:52:26.747 19 INFO neutron.db.segments_db [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] Added segment 71185329-c337-40d3-b4b6-42797d1d76d7 of type geneve for network 8313f893-f5ca-4dfd-8a3a-b87a305b10ef
2021-07-26 10:52:26.813 19 DEBUG neutron_lib.callbacks.manager [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] Notify callbacks ['neutron.services.qos.qos_plugin.QoSPlugin._validate_create_network_callback--9223372036854650062', 'neutron.services.auto_allocate.db._ensure_external_network_default_value_callback-8765406609216'] for network, precommit_create _notify_loop /usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] Mechanism driver 'ovn' failed in create_network_precommit: AttributeError: 'NoneType' object has no attribute 'nb_global'
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 507, in create_network_precommit
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers self._validate_network_segments(context.network_segments)
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 459, in _validate_network_segments
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers max_tunid = self._get_max_tunid()
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 453, in _get_max_tunid
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers return int(self._nb_ovn.nb_global.options.get('max_tunid'))
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers AttributeError: 'NoneType' object has no attribute 'nb_global'
2021-07-26 10:52:26.904 19 ERROR neutron.plugins.ml2.managers
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] POST failed.: neutron.plugins.ml2.common.exceptions.MechanismDriverError
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pecan/core.py", line 683, in __call__
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pecan/core.py", line 574, in invoke_controller
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 163, in post
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return self.create(resources)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 181, in create
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return {key: creator(*creator_args, **creator_kwargs)}
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 685, in inner
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 218, in wrapped
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1102, in create_network
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation result, mech_context = self._create_network_db(context, network)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1095, in _create_network_db
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation self.mechanism_manager.create_network_precommit(mech_context)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 513, in create_network_precommit
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation raise_db_retriable=True)
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 495, in _call_on_drivers
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation errors=errors
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError
2021-07-26 10:52:26.910 19 ERROR neutron.pecan_wsgi.hooks.translation
2021-07-26 10:52:26.913 19 DEBUG neutron.pecan_wsgi.hooks.notifier [req-67d890d1-889e-40c7-a380-99396f88cd29 1f65b02b06844d59aa18031903301a79 20ad786fb60d459a9ac43bea8623d8b3 - default default] No notification will be sent due to unsuccessful status code: 500 after /usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/notifier.py:79
