Activity log for bug #1811070

Date Who What changed Old value New value Message
2019-01-09 09:32:12 Daniel Marks bug added bug
2019-01-09 09:47:39 Daniel Marks description ## OUR SETUP OSA 18.1.0 Ubuntu 16.04 Neutron: OVS & DVR, neutron-server in lxc, neutron-agents on baremetal ## BUG Deploying FWaaS v2 should be fairly simple (according to the docs): https://docs.openstack.org/openstack-ansible-os_neutron/latest/configure-network-services.html#deploying-fwaas-v2 Unfortunately doing so sends neutron-server into a crash loop. The config changes (applied by os-neutron-install.yml) in neutron.conf and l3_agent.ini do not look like the examples in the FWaaS v2 scenario docs at https://docs.openstack.org/neutron/rocky/admin/fwaas-v2-scenario.html ## NEUTRON SERVER The neutron-server log file shows the following error message and restarts: 2019-01-04 19:36:47.052 130210 ERROR neutron.services.service_base [req-f679e2b1-85b5-45c0-b21b-95ca22256 8f7 - - - - -] No providers specified for 'FIREWALL_V2' service, exiting Adding the service provider as stated in the scenario doc also did not help. I had to add the "_V2" to the line as stated in the error message above. The config line that works is: [service_providers] # ... service_provider = FIREWALL_V2:Iptables:neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver:default That at least resolved the error above, but neutron-server is still crash looping. Now with a new error: 2019-01-09 07:44:22.956 5509 ERROR neutron.agent.linux.utils [req-a792d15d-46ac-462b-adee-6ea9a3a8549a - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "<neutron_fwaas.services.fi rewall.fwaas_plugin_v2.FirewallPluginV2 object at 0x7fe438d8f510>": No such file or directory 2019-01-09 07:44:22.958 5509 ERROR neutron.service [req-a792d15d-46ac-462b-adee-6ea9a3a8549a - - - - -] Unrecoverable error: please check log for details.: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "<neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2 object at 0x7fe438d8f510>": No such file or directory 2019-01-09 07:44:22.958 5509 ERROR neutron.service Traceback (most recent call last): 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/service.py", line 86, in serve_wsgi 2019-01-09 07:44:22.958 5509 ERROR neutron.service service.start() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/service.py", line 62, in start 2019-01-09 07:44:22.958 5509 ERROR neutron.service self.wsgi_app = _run_wsgi(self.app_name) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/service.py", line 291, in _run_wsgi 2019-01-09 07:44:22.958 5509 ERROR neutron.service app = config.load_paste_app(app_name) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/common/config.py", line 125, in load_paste_app 2019-01-09 07:44:22.958 5509 ERROR neutron.service app = loader.load_app(app_name) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_service/wsgi.py", line 353, in load_app 2019-01-09 07:44:22.958 5509 ERROR neutron.service return deploy.loadapp("config:%s" % self.config_path, name=name) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp 2019-01-09 07:44:22.958 5509 ERROR neutron.service return loadobj(APP, uri, name=name, **kw) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj 2019-01-09 07:44:22.958 5509 ERROR neutron.service return context.create() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.object_type.invoke(self) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2019-01-09 07:44:22.958 5509 ERROR neutron.service **context.local_conf) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call 2019-01-09 07:44:22.958 5509 ERROR neutron.service val = callable(*args, **kw) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/urlmap.py", line 31, in urlmap_factory 2019-01-09 07:44:22.958 5509 ERROR neutron.service app = loader.get_app(app_name, global_conf=global_conf) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app 2019-01-09 07:44:22.958 5509 ERROR neutron.service name=name, global_conf=global_conf).create() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.object_type.invoke(self) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2019-01-09 07:44:22.958 5509 ERROR neutron.service **context.local_conf) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call 2019-01-09 07:44:22.958 5509 ERROR neutron.service val = callable(*args, **kw) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/auth.py", line 47, in pipeline_factory 2019-01-09 07:44:22.958 5509 ERROR neutron.service app = loader.get_app(pipeline[-1]) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app 2019-01-09 07:44:22.958 5509 ERROR neutron.service name=name, global_conf=global_conf).create() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.object_type.invoke(self) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke 2019-01-09 07:44:22.958 5509 ERROR neutron.service return fix_call(context.object, context.global_conf, **context.local_conf) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call 2019-01-09 07:44:22.958 5509 ERROR neutron.service val = callable(*args, **kw) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/api/v2/router.py", line 25, in _factory 2019-01-09 07:44:22.958 5509 ERROR neutron.service return pecan_app.v2_factory(global_config, **local_config) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/pecan_wsgi/app.py", line 47, in v2_factory 2019-01-09 07:44:22.958 5509 ERROR neutron.service startup.initialize_all() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/pecan_wsgi/startup.py", line 39, in initialize_all 2019-01-09 07:44:22.958 5509 ERROR neutron.service manager.init() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 300, in init 2019-01-09 07:44:22.958 5509 ERROR neutron.service NeutronManager.get_instance() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 251, in get_instance 2019-01-09 07:44:22.958 5509 ERROR neutron.service cls._create_instance() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2019-01-09 07:44:22.958 5509 ERROR neutron.service return f(*args, **kwargs) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 237, in _create_instance 2019-01-09 07:44:22.958 5509 ERROR neutron.service cls._instance = cls() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 142, in __init__ 2019-01-09 07:44:22.958 5509 ERROR neutron.service self._load_service_plugins() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 204, in _load_service_plugins 2019-01-09 07:44:22.958 5509 ERROR neutron.service provider) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 167, in _get_plugin_instance 2019-01-09 07:44:22.958 5509 ERROR neutron.service return plugin_class() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin_v2.py", line 60, in __init__ 2019-01-09 07:44:22.958 5509 ERROR neutron.service fwaas_constants.FIREWALL_V2, self) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/services/service_base.py", line 47, in load_drivers 2019-01-09 07:44:22.958 5509 ERROR neutron.service provider['driver'], plugin 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object 2019-01-09 07:44:22.958 5509 ERROR neutron.service return import_class(import_str)(*args, **kwargs) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 79, in __init__ 2019-01-09 07:44:22.958 5509 ERROR neutron.service zone_per_port=self.CONNTRACK_ZONE_PER_PORT) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2019-01-09 07:44:22.958 5509 ERROR neutron.service return f(*args, **kwargs) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.py", line 58, in get_conntrack 2019-01-09 07:44:22.958 5509 ERROR neutron.service execute, namespace, zone_per_port) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.py", line 75, in __init__ 2019-01-09 07:44:22.958 5509 ERROR neutron.service self._populate_initial_zone_map() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.py", line 182, in _populate_initial_zone_map 2019-01-09 07:44:22.958 5509 ERROR neutron.service rules = self.get_rules_for_table_func('raw') 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 473, in get_rules_for_table 2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.execute(args, run_as_root=True).split('\n') 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 147, in execute 2019-01-09 07:44:22.958 5509 ERROR neutron.service returncode=returncode) 2019-01-09 07:44:22.958 5509 ERROR neutron.service ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "<neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2 object at 0x7fe438d8f510>": No such file or directory 2019-01-09 07:44:22.958 5509 ERROR neutron.service 2019-01-09 07:44:22.958 5509 ERROR neutron.service The scenario doc also lists a "[fwaas]" section in neutron.conf, however adding that section did not make a difference at any point. ## NEUTRON L3 AGENT The l3 agent was running the whole time, but I was not able to verify if it was properly configured for FWaaS v2. What I saw, was that it at least not configured as the scenario doc examples. l3_agent.ini lacks the following section form the scenario doc: [AGENT] extensions = fwaas_v2 But at the same time contains the "[fwaas]" section that the scenario doc places in the neutron.conf: [fwaas] agent_version = v2 driver = neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2.IptablesFwaasDriver enabled = True ## WAY FORWARD I would be happy to provide patches for the os_neutron ansible role to make FWaaS v2 work, but I struggle on getting the configuration right. Also the scenario doc does not seem to be up-to-date (or simply wrong). Also I wonder which network namespaces neutron-server is trying to access... if neutron-server needs access to the agents namespaces, then this will never work with the server in lxc... ## OUR SETUP OSA 18.1.0 Ubuntu 16.04 Neutron: OVS & DVR, neutron-server in lxc, neutron-agents on baremetal ## BUG Deploying FWaaS v2 should be fairly simple (according to the docs): https://docs.openstack.org/openstack-ansible-os_neutron/latest/configure-network-services.html#deploying-fwaas-v2 Unfortunately doing so sends neutron-server into a crash loop. The config changes (applied by os-neutron-install.yml) in neutron.conf and l3_agent.ini do not look like the examples in the FWaaS v2 scenario docs at https://docs.openstack.org/neutron/rocky/admin/fwaas-v2-scenario.html ## NEUTRON SERVER The neutron-server log file shows the following error message and restarts: 2019-01-04 19:36:47.052 130210 ERROR neutron.services.service_base [req-f679e2b1-85b5-45c0-b21b-95ca22256 8f7 - - - - -] No providers specified for 'FIREWALL_V2' service, exiting Adding the service provider as stated in the scenario doc also did not help. I had to add the "_V2" to the line as stated in the error message above. The config line that works is: [service_providers] # ... service_provider = FIREWALL_V2:Iptables:neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver:default That at least resolved the error above, but neutron-server is still crash looping. Now with a new error: 2019-01-09 07:44:22.956 5509 ERROR neutron.agent.linux.utils [req-a792d15d-46ac-462b-adee-6ea9a3a8549a - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "<neutron_fwaas.services.fi rewall.fwaas_plugin_v2.FirewallPluginV2 object at 0x7fe438d8f510>": No such file or directory 2019-01-09 07:44:22.958 5509 ERROR neutron.service [req-a792d15d-46ac-462b-adee-6ea9a3a8549a - - - - -] Unrecoverable error: please check log for details.: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "<neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2 object at 0x7fe438d8f510>": No such file or directory 2019-01-09 07:44:22.958 5509 ERROR neutron.service Traceback (most recent call last): 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/service.py", line 86, in serve_wsgi 2019-01-09 07:44:22.958 5509 ERROR neutron.service service.start() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/service.py", line 62, in start 2019-01-09 07:44:22.958 5509 ERROR neutron.service self.wsgi_app = _run_wsgi(self.app_name) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/service.py", line 291, in _run_wsgi 2019-01-09 07:44:22.958 5509 ERROR neutron.service app = config.load_paste_app(app_name) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/common/config.py", line 125, in load_paste_app 2019-01-09 07:44:22.958 5509 ERROR neutron.service app = loader.load_app(app_name) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_service/wsgi.py", line 353, in load_app 2019-01-09 07:44:22.958 5509 ERROR neutron.service return deploy.loadapp("config:%s" % self.config_path, name=name) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp 2019-01-09 07:44:22.958 5509 ERROR neutron.service return loadobj(APP, uri, name=name, **kw) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj 2019-01-09 07:44:22.958 5509 ERROR neutron.service return context.create() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.object_type.invoke(self) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2019-01-09 07:44:22.958 5509 ERROR neutron.service **context.local_conf) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call 2019-01-09 07:44:22.958 5509 ERROR neutron.service val = callable(*args, **kw) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/urlmap.py", line 31, in urlmap_factory 2019-01-09 07:44:22.958 5509 ERROR neutron.service app = loader.get_app(app_name, global_conf=global_conf) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app 2019-01-09 07:44:22.958 5509 ERROR neutron.service name=name, global_conf=global_conf).create() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.object_type.invoke(self) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2019-01-09 07:44:22.958 5509 ERROR neutron.service **context.local_conf) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call 2019-01-09 07:44:22.958 5509 ERROR neutron.service val = callable(*args, **kw) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/auth.py", line 47, in pipeline_factory 2019-01-09 07:44:22.958 5509 ERROR neutron.service app = loader.get_app(pipeline[-1]) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app 2019-01-09 07:44:22.958 5509 ERROR neutron.service name=name, global_conf=global_conf).create() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.object_type.invoke(self) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke 2019-01-09 07:44:22.958 5509 ERROR neutron.service return fix_call(context.object, context.global_conf, **context.local_conf) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call 2019-01-09 07:44:22.958 5509 ERROR neutron.service val = callable(*args, **kw) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/api/v2/router.py", line 25, in _factory 2019-01-09 07:44:22.958 5509 ERROR neutron.service return pecan_app.v2_factory(global_config, **local_config) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/pecan_wsgi/app.py", line 47, in v2_factory 2019-01-09 07:44:22.958 5509 ERROR neutron.service startup.initialize_all() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/pecan_wsgi/startup.py", line 39, in initialize_all 2019-01-09 07:44:22.958 5509 ERROR neutron.service manager.init() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 300, in init 2019-01-09 07:44:22.958 5509 ERROR neutron.service NeutronManager.get_instance() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 251, in get_instance 2019-01-09 07:44:22.958 5509 ERROR neutron.service cls._create_instance() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2019-01-09 07:44:22.958 5509 ERROR neutron.service return f(*args, **kwargs) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 237, in _create_instance 2019-01-09 07:44:22.958 5509 ERROR neutron.service cls._instance = cls() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 142, in __init__ 2019-01-09 07:44:22.958 5509 ERROR neutron.service self._load_service_plugins() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 204, in _load_service_plugins 2019-01-09 07:44:22.958 5509 ERROR neutron.service provider) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 167, in _get_plugin_instance 2019-01-09 07:44:22.958 5509 ERROR neutron.service return plugin_class() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin_v2.py", line 60, in __init__ 2019-01-09 07:44:22.958 5509 ERROR neutron.service fwaas_constants.FIREWALL_V2, self) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/services/service_base.py", line 47, in load_drivers 2019-01-09 07:44:22.958 5509 ERROR neutron.service provider['driver'], plugin 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object 2019-01-09 07:44:22.958 5509 ERROR neutron.service return import_class(import_str)(*args, **kwargs) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 79, in __init__ 2019-01-09 07:44:22.958 5509 ERROR neutron.service zone_per_port=self.CONNTRACK_ZONE_PER_PORT) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2019-01-09 07:44:22.958 5509 ERROR neutron.service return f(*args, **kwargs) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.py", line 58, in get_conntrack 2019-01-09 07:44:22.958 5509 ERROR neutron.service execute, namespace, zone_per_port) 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.py", line 75, in __init__ 2019-01-09 07:44:22.958 5509 ERROR neutron.service self._populate_initial_zone_map() 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.py", line 182, in _populate_initial_zone_map 2019-01-09 07:44:22.958 5509 ERROR neutron.service rules = self.get_rules_for_table_func('raw') 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 473, in get_rules_for_table 2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.execute(args, run_as_root=True).split('\n') 2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 147, in execute 2019-01-09 07:44:22.958 5509 ERROR neutron.service returncode=returncode) 2019-01-09 07:44:22.958 5509 ERROR neutron.service ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "<neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2 object at 0x7fe438d8f510>": No such file or directory 2019-01-09 07:44:22.958 5509 ERROR neutron.service 2019-01-09 07:44:22.958 5509 ERROR neutron.service The scenario doc also lists a "[fwaas]" section in neutron.conf, however adding that section did not make a difference at any point. ## NEUTRON L3 AGENT The l3 agent was running the whole time, but I was not able to verify if it was properly configured for FWaaS v2. What I see is that it is not configured as the scenario doc examples. l3_agent.ini lacks the following section form the scenario doc: [AGENT] extensions = fwaas_v2 But at the same time contains the "[fwaas]" section that the scenario doc places in the neutron.conf: [fwaas] agent_version = v2 driver = neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2.IptablesFwaasDriver enabled = True ## WAY FORWARD I would be happy to provide patches for the os_neutron ansible role to make FWaaS v2 work, but I struggle on getting the configuration right. Also the scenario doc does not seem to be up-to-date (or simply wrong). Also I wonder which network namespaces neutron-server is trying to access... if neutron-server needs access to the agents namespaces, then this will never work with the server in lxc...
2019-02-13 13:06:30 James Denton openstack-ansible: assignee James Denton (james-denton)
2019-02-13 15:05:44 James Denton openstack-ansible: status New In Progress
2019-02-14 20:12:14 OpenStack Infra openstack-ansible: status In Progress Fix Released
2019-05-13 11:50:54 Christian Zunker bug added subscriber Christian Zunker