rocky router creation fails with CallbackFailure (no such option router_distributed)

[Daniel 'f0o' Preussker]

System is running Rocky using RHEL and RHEL's OpenStack repositories
python-networking-vmware-nsx-2018.0.0-0.20180803224551.322d37b.el7ost.noarch
python2-vmware-nsxlib-13.0.1-0.20181004050247.2d9f856.el7ost.noarch

`router_distributed` was not defined anywhere so I've tried defining `router_distributed = false` in `[DEFAULT]` but error remains.

I'm attaching the initial log where Neutron tries to create the Metadata Proxy.

When trying to create a router manually the same error will occur.

I'm a bit lost where to start looking, any pointers would be great.
I've already grep'ed through the entire distribution to see where `router_distributed` is being used but it's hardly used anywhere at all and defaults safely to `false`...

log:

2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy [req-6bc7b9bb-214c-4031-a8a2-f16db2bf6876 - - - - -] Exception Callback vmware_nsx.plugins.nsx_v.plugin.NsxVPluginV2._process_az_req
uest--9223372036854116211 failed with "no such option router_distributed in group [DEFAULT]" while creating internal edge for metadata service: CallbackFailure: Callback vmware_nsx.plugins.nsx_v.plug
in.NsxVPluginV2._process_az_request--9223372036854116211 failed with "no such option router_distributed in group [DEFAULT]"
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy Traceback (most recent call last):
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/vmware_nsx/plugins/nsx_v/md_proxy.py", line 457, in _setup_new_proxy_edge
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy allow_metadata=False)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/vmware_nsx/plugins/nsx_v/plugin.py", line 3194, in create_router
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy lrouter = super(NsxVPluginV2, self).create_router(context, router)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1935, in create_router
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy router)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy return method(*args, **kwargs)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy setattr(e, '_RETRY_EXCEEDED', True)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy self.force_reraise()
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy six.reraise(self.type_, self.value, self.tb)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy return f(*args, **kwargs)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy ectxt.value = e.inner_exc
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy self.force_reraise()
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy six.reraise(self.type_, self.value, self.tb)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy return f(*args, **kwargs)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy LOG.debug("Retry wrapper got retriable exception: %s", e)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy self.force_reraise()
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy six.reraise(self.type_, self.value, self.tb)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy return f(*dup_args, **dup_kwargs)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 253, in create_router
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy transaction=False)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron/db/_utils.py", line 60, in safe_creation
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy obj = create_fn()
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 233, in _create_router_db
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy router_id=router['id'], router_db=router_db)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron_lib/callbacks/registry.py", line 56, in notify
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy _get_callback_manager().notify(resource, event, trigger, **kwargs)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron_lib/db/utils.py", line 107, in _wrapped
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy raise db_exc.RetryRequest(e)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy self.force_reraise()
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy six.reraise(self.type_, self.value, self.tb)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron_lib/db/utils.py", line 102, in _wrapped
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy return function(*args, **kwargs)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy File "/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py", line 177, in notify
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy raise exceptions.CallbackFailure(errors=errors)
2019-03-04 13:32:04.915 21 ERROR vmware_nsx.plugins.nsx_v.md_proxy CallbackFailure: Callback vmware_nsx.plugins.nsx_v.plugin.NsxVPluginV2._process_az_request--9223372036854116211 failed with "no such
 option router_distributed in group [DEFAULT]"

[Daniel 'f0o' Preussker (dpreussker)]

Additional Info:
VCSA 6.7.0.20000
NSX 6.4.4.11197766
ESXi 6.7.0 11675023

neutron.conf:
[DEFAULT]
fatal_deprecations = false
debug = true
use_stderr = true
use_json = false
notification_format = cadf
transport_url = rabbit://user:pass@rabbit:5672/
host = neutron
core_plugin = vmware_nsx.plugins.nsx_v.plugin.NsxVPluginV2
service_plugins = neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2,neutron_fwaas.services.firewall.fwaas_plugin.FirewallPlugin
api_workers = 1
[database]
connection = mysql+pymysql://user:pass@mysql/neutron
[keystone_authtoken]
www_authenticate_uri = http://keystone
auth_url = http://keystone
memcached_servers = memcached:11211
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = user
password = pass
[nova]
auth_type = password
auth_url = http://keystone
project_domain_name = default
user_domain_name = default
region_name = Dev
project_name = service
username = user
password = pass
[oslo_messaging_notifications]
driver = messagingv2
[service_providers]
service_provider = LOADBALANCERV2:VMWareEdge:neutron_lbaas.drivers.vmware.edge_driver_v2.EdgeLoadBalancerDriverV2:default
service_provider = FIREWALL:fwaasDriver:vmware_nsx.services.fwaas.nsx_v.edge_fwaas_driver.EdgeFwaasDriver:default
[fwaas]
driver = vmware_nsx.services.fwaas.nsx_v.edge_fwaas_driver.EdgeFwaasDriver
enabled = true
[lbaas]
driver = neutron_lbaas.drivers.vmware.edge_driver_v2.EdgeLoadBalancerDriverV2
enabled = true

plugin.ini:
[DEFAULT]
nsx_l2gw_driver = NSX-V
[nsx_lsn]
sync_on_missing_data = true
[nsxv]
user = "admin"
password = "pass"
manager_uri = https://1.2.3.6
insecure = true
cluster_moid = domain-c7
datacenter_moid = datacenter-2
resource_pool_id = resgroup-16
datastore_id = datastore-11
external_network = dvportgroup-19
vdn_scope_id = vdnscope-1
dvs_id = dvs-17
maximum_tunnels_per_vnic = 110
backup_edge_pool = service:compact:0:1,vdr:compact:0:1
mgt_net_moid = dvportgroup-20
mgt_net_proxy_ips = 1.2.3.5
mgt_net_proxy_netmask = 255.255.255.0
nova_metadata_ips = 1.2.3.4
metadata_shared_secret = pass
metadata_insecure = true
tenant_router_types = shared,distributed,exclusive
dhcp_lease_time = 86400
metadata_initializer = true
edge_ha = false
exclusive_router_appliance_size = compact
shared_router_appliance_size = large
nameservers = 8.8.8.8,8.8.4.4,1.1.1.1
service_insertion_redirect_all = true
bind_floatingip_to_all_interfaces = true
exclusive_dhcp_edge = false
share_edges_between_tenants = true
use_default_block_all = false

[Boden R (boden)]

The router_distributed option is not defined in vmware-nsx; it comes from neutron.

Have you tried setting it in the [DEFAULT] section of neutron.conf??
I don't see it in the above snippet.

[Daniel 'f0o' Preussker (dpreussker)]

@Boden Thanks again for the input yesterday on IRC.

I can confirm now that adding `from neutron.db import l3_dvr_db` into the top of `vmware_nsx/plugins/nsx_v/plugin.py` solves the issue.

I've run several tests and they all resulted in router creation and also functioning router edges.

[Boden R (boden)]

dpreussker thanks for the update.
I need to follow up with the neutron team on this one.
Based on what I know, it seems that the router_distributed option should be registered with the conf as part of the framework and should not be something a plugin has to do.
That said it seems to me the fix should go into neutron and not vmware-nsx, but I will follow up with the neutron team on this.

[Boden R (boden)]

As we discussed on IRC [1] the specific problem in this bug is that the router_distributed config option is not being registered in the global CONF before its accessed. While vmware-nsx does not use or even support that option, it use of the RouterAvailabilityZoneMixin requires the option to be registered (under the covers).

While the above describes the root cause, as per the IRC chat [1] this only occurs on mumboyumbo and not on RDO, devstack, etc..
In addition I don't believe we even support the nsx-v plugin with RedHat.
Given the above, I'm not sure we can move forward with any kind of code change/fix for this unless we can prove its a supported configuration/env.

Marking this bug as such.

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2019-03-21.log.html#t2019-03-21T17:04:41

[Launchpad Janitor (janitor)]

[Expired for vmware-nsx because there has been no activity for 60 days.]