neutron FWaaS v2 installation breaks neutron-server

[Daniel Marks]

## OUR SETUP

OSA 18.1.0
Ubuntu 16.04
Neutron: OVS & DVR, neutron-server in lxc, neutron-agents on baremetal

## BUG

Deploying FWaaS v2 should be fairly simple (according to the docs): https://docs.openstack.org/openstack-ansible-os_neutron/latest/configure-network-services.html#deploying-fwaas-v2
Unfortunately doing so sends neutron-server into a crash loop. The config changes (applied by os-neutron-install.yml) in neutron.conf and l3_agent.ini do not look like the examples in the FWaaS v2 scenario docs at https://docs.openstack.org/neutron/rocky/admin/fwaas-v2-scenario.html

## NEUTRON SERVER

The neutron-server log file shows the following error message and restarts:

2019-01-04 19:36:47.052 130210 ERROR neutron.services.service_base [req-f679e2b1-85b5-45c0-b21b-95ca22256
8f7 - - - - -] No providers specified for 'FIREWALL_V2' service, exiting

Adding the service provider as stated in the scenario doc also did not help. I had to add the "_V2" to the line as stated in the error message above. The config line that works is:

[service_providers]
# ...
service_provider = FIREWALL_V2:Iptables:neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver:default

That at least resolved the error above, but neutron-server is still crash looping. Now with a new error:

2019-01-09 07:44:22.956 5509 ERROR neutron.agent.linux.utils [req-a792d15d-46ac-462b-adee-6ea9a3a8549a - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "<neutron_fwaas.services.fi
rewall.fwaas_plugin_v2.FirewallPluginV2 object at 0x7fe438d8f510>": No such file or directory

2019-01-09 07:44:22.958 5509 ERROR neutron.service [req-a792d15d-46ac-462b-adee-6ea9a3a8549a - - - - -] Unrecoverable error: please check log for details.: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ;
Stderr: Cannot open network namespace "<neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2 object at 0x7fe438d8f510>": No such file or directory
2019-01-09 07:44:22.958 5509 ERROR neutron.service Traceback (most recent call last):
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/service.py", line 86, in serve_wsgi
2019-01-09 07:44:22.958 5509 ERROR neutron.service service.start()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/service.py", line 62, in start
2019-01-09 07:44:22.958 5509 ERROR neutron.service self.wsgi_app = _run_wsgi(self.app_name)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/service.py", line 291, in _run_wsgi
2019-01-09 07:44:22.958 5509 ERROR neutron.service app = config.load_paste_app(app_name)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/common/config.py", line 125, in load_paste_app
2019-01-09 07:44:22.958 5509 ERROR neutron.service app = loader.load_app(app_name)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_service/wsgi.py", line 353, in load_app
2019-01-09 07:44:22.958 5509 ERROR neutron.service return deploy.loadapp("config:%s" % self.config_path, name=name)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2019-01-09 07:44:22.958 5509 ERROR neutron.service return loadobj(APP, uri, name=name, **kw)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2019-01-09 07:44:22.958 5509 ERROR neutron.service return context.create()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.object_type.invoke(self)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2019-01-09 07:44:22.958 5509 ERROR neutron.service **context.local_conf)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2019-01-09 07:44:22.958 5509 ERROR neutron.service val = callable(*args, **kw)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/urlmap.py", line 31, in urlmap_factory
2019-01-09 07:44:22.958 5509 ERROR neutron.service app = loader.get_app(app_name, global_conf=global_conf)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2019-01-09 07:44:22.958 5509 ERROR neutron.service name=name, global_conf=global_conf).create()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.object_type.invoke(self)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2019-01-09 07:44:22.958 5509 ERROR neutron.service **context.local_conf)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2019-01-09 07:44:22.958 5509 ERROR neutron.service val = callable(*args, **kw)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/auth.py", line 47, in pipeline_factory
2019-01-09 07:44:22.958 5509 ERROR neutron.service app = loader.get_app(pipeline[-1])
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2019-01-09 07:44:22.958 5509 ERROR neutron.service name=name, global_conf=global_conf).create()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.object_type.invoke(self)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2019-01-09 07:44:22.958 5509 ERROR neutron.service return fix_call(context.object, context.global_conf, **context.local_conf)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2019-01-09 07:44:22.958 5509 ERROR neutron.service val = callable(*args, **kw)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/api/v2/router.py", line 25, in _factory
2019-01-09 07:44:22.958 5509 ERROR neutron.service return pecan_app.v2_factory(global_config, **local_config)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/pecan_wsgi/app.py", line 47, in v2_factory
2019-01-09 07:44:22.958 5509 ERROR neutron.service startup.initialize_all()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/pecan_wsgi/startup.py", line 39, in initialize_all
2019-01-09 07:44:22.958 5509 ERROR neutron.service manager.init()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 300, in init
2019-01-09 07:44:22.958 5509 ERROR neutron.service NeutronManager.get_instance()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 251, in get_instance
2019-01-09 07:44:22.958 5509 ERROR neutron.service cls._create_instance()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-01-09 07:44:22.958 5509 ERROR neutron.service return f(*args, **kwargs)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 237, in _create_instance
2019-01-09 07:44:22.958 5509 ERROR neutron.service cls._instance = cls()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 142, in __init__
2019-01-09 07:44:22.958 5509 ERROR neutron.service self._load_service_plugins()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 204, in _load_service_plugins
2019-01-09 07:44:22.958 5509 ERROR neutron.service provider)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/manager.py", line 167, in _get_plugin_instance
2019-01-09 07:44:22.958 5509 ERROR neutron.service return plugin_class()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin_v2.py", line 60, in __init__
2019-01-09 07:44:22.958 5509 ERROR neutron.service fwaas_constants.FIREWALL_V2, self)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/services/service_base.py", line 47, in load_drivers
2019-01-09 07:44:22.958 5509 ERROR neutron.service provider['driver'], plugin
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2019-01-09 07:44:22.958 5509 ERROR neutron.service return import_class(import_str)(*args, **kwargs)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 79, in __init__
2019-01-09 07:44:22.958 5509 ERROR neutron.service zone_per_port=self.CONNTRACK_ZONE_PER_PORT)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-01-09 07:44:22.958 5509 ERROR neutron.service return f(*args, **kwargs)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.py", line 58, in get_conntrack
2019-01-09 07:44:22.958 5509 ERROR neutron.service execute, namespace, zone_per_port)
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.py", line 75, in __init__
2019-01-09 07:44:22.958 5509 ERROR neutron.service self._populate_initial_zone_map()
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.py", line 182, in _populate_initial_zone_map
2019-01-09 07:44:22.958 5509 ERROR neutron.service rules = self.get_rules_for_table_func('raw')
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 473, in get_rules_for_table
2019-01-09 07:44:22.958 5509 ERROR neutron.service return self.execute(args, run_as_root=True).split('\n')
2019-01-09 07:44:22.958 5509 ERROR neutron.service File "/openstack/venvs/neutron-18.1.0/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 147, in execute
2019-01-09 07:44:22.958 5509 ERROR neutron.service returncode=returncode)
2019-01-09 07:44:22.958 5509 ERROR neutron.service ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "<neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2 object at 0x7fe438d8f510>": No such file or directory
2019-01-09 07:44:22.958 5509 ERROR neutron.service
2019-01-09 07:44:22.958 5509 ERROR neutron.service

The scenario doc also lists a "[fwaas]" section in neutron.conf, however adding that section did not make a difference at any point.

## NEUTRON L3 AGENT

The l3 agent was running the whole time, but I was not able to verify if it was properly configured for FWaaS v2. What I see is that it is not configured as the scenario doc examples.

l3_agent.ini lacks the following section form the scenario doc:

[AGENT]
extensions = fwaas_v2

But at the same time contains the "[fwaas]" section that the scenario doc places in the neutron.conf:

[fwaas]
agent_version = v2
driver = neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2.IptablesFwaasDriver
enabled = True

## WAY FORWARD

I would be happy to provide patches for the os_neutron ansible role to make FWaaS v2 work, but I struggle on getting the configuration right. Also the scenario doc does not seem to be up-to-date (or simply wrong).

Also I wonder which network namespaces neutron-server is trying to access... if neutron-server needs access to the agents namespaces, then this will never work with the server in lxc...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/636757

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/636757
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_neutron/commit/?id=41bd86b7bdd9b227e111363c3cd9afbf5d9147e1
Submitter: Zuul
Branch: master

commit 41bd86b7bdd9b227e111363c3cd9afbf5d9147e1
Author: James Denton <email address hidden>
Date: Wed Feb 13 21:02:28 2019 +0000

    Enable functional deployment of FWaaS v2

    This patch updates various vars and templates to enable a functioning
    deployment of FWaaS v2 on an Open vSwitch-based OSA cloud. A test is
    also included for verification.

    Change-Id: Ibfa2cbafd19f6870139c4ea3e9dfc80cf8c574e1
    Closes-Bug: #1811070

[Markus Küffner (mkuf)]

Any chance that this will be backported to Rocky?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_neutron stein-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_neutron train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_neutron ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_neutron yoga-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_neutron victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_neutron wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_neutron xena-eom release.