Obscure error message if missing 'ipset' utility

[Matt Kassawara]

In Liberty (7.0.0) and possibly prior releases, if you enable 'ipset' via the 'enable_ipset' option in the [securitygroup] section, a node running the Linux bridge agent that lacks the 'ipset' utility (e.g., due to a dependency issue) produces a rather obscure error message when attempting to launch an instance. Consider making this error message more useful.

INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-daeebd97-58ba-4f30-8184-9db4b54110d9 - - - - -] Agent out of sync with plugin!
INFO neutron.agent.securitygroups_rpc [req-daeebd97-58ba-4f30-8184-9db4b54110d9 - - - - -] Preparing filters for devices set(['tap7bdfbc9d-f6'])
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-daeebd97-58ba-4f30-8184-9db4b54110d9 - - - - -] Error in agent loop. Devices info: {'current': set(['tap7bdfbc9d-f6']), 'removed': set([]), 'added': set(['tap7bdfbc9d-f6']), 'updated': set([])}
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent Traceback (most recent call last):
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1105, in daemon_loop
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent sync = self.process_network_devices(device_info)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 947, in process_network_devices
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent device_info.get('updated'))
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 286, in setup_port_filters
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent self.prepare_devices_filter(new_devices)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 142, in decorated_function
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent *args, **kwargs)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 167, in prepare_devices_filter
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent security_groups, security_group_member_ips)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent self.gen.next()
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/firewall.py", line 110, in defer_apply
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent self.filter_defer_apply_off()
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 792, in filter_defer_apply_off
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent self.unfiltered_ports)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 210, in _setup_chains_apply
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent self._setup_chain(port, firewall.INGRESS_DIRECTION)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 237, in _setup_chain
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent self._add_rules_by_security_group(port, DIRECTION)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 479, in _add_rules_by_security_group
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent self._update_ipset_members(remote_sg_ids)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 515, in _update_ipset_members
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent self.ipset.set_members(sg_id, ip_version, current_ips)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 254, in inner
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent return f(*args, **kwargs)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 87, in set_members
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent self._create_set(set_name, ethertype)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 133, in _create_set
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent self._apply(cmd)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 143, in _apply
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent check_exit_code=fail_on_errors)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 117, in execute
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent execute_rootwrap_daemon(cmd, process_input, addl_env))
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 103, in execute_rootwrap_daemon
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent return client.execute(cmd, process_input)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line 137, in execute
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent res = proxy.run_one_command(cmd, stdin)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "<string>", line 2, in run_one_command
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent raise convert_to_error(kind, result)
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent RemoteError:
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent ---------------------------------------------------------------------------
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent Unserializable message: ('#ERROR', FilterMatchNotExecutable())
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent ---------------------------------------------------------------------------
ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/240265

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/240265
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a3f7d795b64ef913814afb9d9e17f3449a85fd3e
Submitter: Jenkins
Branch: master

commit a3f7d795b64ef913814afb9d9e17f3449a85fd3e
Author: Dongcan Ye <email address hidden>
Date: Thu Oct 29 20:50:43 2015 +0800

    Check missed IPSet utility using neutron-sanity-check

    In some case, host may lack ipset utility (e.g., due to a
    dependency issue)

    This patch allows checking IPSet utility support from CLI:
        neutron-sanity-check --ipset_installed

    Or using configuration options, for example:
        neutron-sanity-check --config-file /etc/neutron/neutron.conf
        --config-file /etc/neutron/plugins/ml2/ml2_conf.ini

    Closes-Bug: #1510680
    Change-Id: I2b9d6b13087a970bb0919a8217e428ce60d6e0c3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/250801

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/250801
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=91c17c045cc1625d6e21734816e43f6eafbf7ecd
Submitter: Jenkins
Branch: stable/liberty

commit 91c17c045cc1625d6e21734816e43f6eafbf7ecd
Author: Dongcan Ye <email address hidden>
Date: Thu Oct 29 20:50:43 2015 +0800

    Check missed IPSet utility using neutron-sanity-check

    In some case, host may lack ipset utility (e.g., due to a
    dependency issue)

    This patch allows checking IPSet utility support from CLI:
        neutron-sanity-check --ipset_installed

    Or using configuration options, for example:
        neutron-sanity-check --config-file /etc/neutron/neutron.conf
        --config-file /etc/neutron/plugins/ml2/ml2_conf.ini

    Closes-Bug: #1510680
    Change-Id: I2b9d6b13087a970bb0919a8217e428ce60d6e0c3
    (cherry picked from commit a3f7d795b64ef913814afb9d9e17f3449a85fd3e)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 7.0.2 release.

[Keith (openstack-z)]

I just hit exactly this bug installing Mitaka on a fresh CentOS 7 using the instructions at docs.openstack,org (networking option 2) - my controller node has ipset installed but my compute node did not - both had identical clean installations of CentOS 7 minimal + base group before I started.

I got the same error symptoms in my linuxbridge logs on the compute node when spinning upa new instance.

Fixed it with a simple 'yum install ipset', now everything is running fine.

[trakatelis (trakatelis)]

The same thing happens with networking option 1.
Configuration as Keith's.

[Dongcan Ye (hellochosen)]

Yes, we should install ipset manually.

[Stephen Mastrorocco (stephen-mastrorocco)]

Same issue confirmed as Keith. Fresh Mitaka/CentOS 7 install. Installing ipset on compute node resolved.

[Eric (e3gh75)]

I also encountered this issue with a fresh Centos 7.2.1511 install and Mitka. ipset was not installed on the compute node. VM would not spawn.

[trakatelis (trakatelis)]

1. Another solution (which of course should be avoided) is to omit
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver
in the [securitygroup] section of linuxbridge_agent.ini on compute node(s).

2. Not being able to spawn a VM is fatal.
I think this situation should be addressed as soon as possible.
Should it be reported as an error affecting mitaka precisely?
(stable/liberty is ok as mentioned above)

[Dongcan Ye (hellochosen)]

Hi, trakatelis. I had mad ipset installation in install guide docs:
https://review.openstack.org/#/c/313542/

[Matt Kassawara (ionosphere80)]

Actually, the enable_ipset option in the ml2_conf.ini file on the controller node triggers this issue if the compute nodes do not contain the ipset utility. This is a recurring RDO packaging bug rather than an installation guide bug. Thus, someone should open a bug on bugzilla.redhat.com and reference it in a comment near the workaround in the installation guide so we can track it.

[Matt Kassawara (ionosphere80)]

Also, because this issue involves packages, it isn't a neutron bug. Moving to openstack-manuals as we can at least provide a workaround for it.

[Pablo Schrader (pablo-schrader)]

Installing ipset solves the issue on Mitaka and was able to spin-up a VM on the corresponding network.
But testing a little bit further I saw this problem too when attaching a port to a new created VM instead of assigning a network even if ipset is installed on the compute node.

Down the errors that I saw on the linuxbridge-agent.log on the compute node

2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-d667c5cf-b953-4ca0-a31a-6bed2d387825 - - - - -] Error in agent loop. Devices info: {'current': set([]), 'timestamps': {}, 'removed': set(['tap0272f27b-aa', 'tapad0fb9de-9c']), 'added': set([]), 'updated': set([])}
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 417, in daemon_loop
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent sync = self.process_network_devices(device_info)
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 190, in process_network_devices
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent device_info.get('updated'))
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 315, in setup_port_filters
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.refresh_firewall(updated_devices)
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 152, in decorated_function
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent *args, **kwargs)
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 268, in refresh_firewall
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.firewall.update_port_filter(device)
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.gen.next()
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/site-packages/neutron/agent/firewall.py", line 128, in defer_apply
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.filter_defer_apply_off()
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 815, in filter_defer_apply_off
2016-05-09 09:52:02.771 1402 ERROR neutron.plugins.ml2.drivers.agent...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/313542
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=8a4ffb4224d438d68d642f93269002b06bf63502
Submitter: Jenkins
Branch: master

commit 8a4ffb4224d438d68d642f93269002b06bf63502
Author: Dongcan Ye <email address hidden>
Date: Fri May 6 20:53:08 2016 +0800

    [install-guide] Add ipset installation in compute node

    In some case, host may lack ipset utility(e.g., due to a
    dependency issue). This will cause create vm failed if we
    enable Neutron securitygroup in compute node.

    We had already fixed in Neutron side, we using neutron-sanity-check
    tool for check ipset installation. But this is insufficiency,
    some guys may not know that tool.

    So we install ipset in compute node.

    backport: Mitaka

    Change-Id: If071a9aa3d8bb5854de1abd4c9eb3eafb3b07817
    Related-Bug: #1510680

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/314927

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314927
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=5c070bcf0b1714b9e1f511d3374a81802a155be9
Submitter: Jenkins
Branch: stable/mitaka

commit 5c070bcf0b1714b9e1f511d3374a81802a155be9
Author: Dongcan Ye <email address hidden>
Date: Fri May 6 20:53:08 2016 +0800

    [install-guide] Add ipset installation in compute node

    In some case, host may lack ipset utility(e.g., due to a
    dependency issue). This will cause create vm failed if we
    enable Neutron securitygroup in compute node.

    We had already fixed in Neutron side, we using neutron-sanity-check
    tool for check ipset installation. But this is insufficiency,
    some guys may not know that tool.

    So we install ipset in compute node.

    Change-Id: If071a9aa3d8bb5854de1abd4c9eb3eafb3b07817
    Related-Bug: #1510680
    (cherry picked from commit 8a4ffb4224d438d68d642f93269002b06bf63502)