Comment 3 for bug 1491131

I am not sure how to reproduce this problem. It happens randomly (last time was August 19th). It actually happened on the same server and it also happened on the same request UUID (how is that possible?)

2015-08-19 15:13:04.547 6840 INFO neutron.agent.securitygroups_rpc [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Refresh firewall rules
2015-08-19 15:13:04.778 6840 ERROR neutron.agent.linux.utils [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ]
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ipset', 'del', 'NETIPv48a17c661-2071-4527-a', '10.22.248.16']
Exit code: 1
Stdin:
Stdout:
Stderr: ipset v6.11: Element cannot be deleted from the set: it's not added

[req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Error while processing VIF ports
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1605, in rpc_loop
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent ovs_restarted)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1367, in process_network_ports
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 375, in setup_port_filters
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.refresh_firewall(updated_devices)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 219, in decorated_function
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 331, in refresh_firewall
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent security_groups, security_group_member_ips)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/rh/python27/root/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.gen.next()
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/firewall.py", line 106, in defer_apply
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.filter_defer_apply_off()
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 658, in filter_defer_apply_off
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.unfiltered_ports)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 155, in _setup_chains_apply
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self._setup_chain(port, INGRESS_DIRECTION)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 182, in _setup_chain
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self._add_rules_by_security_group(port, DIRECTION)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 411, in _add_rules_by_security_group
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self._update_ipset_members(remote_sg_ids)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 448, in _update_ipset_members
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.ipset.set_members(sg_id, ip_version, current_ips)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 94, in set_members
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self._del_members_from_set(set_name, del_ips)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 158, in _del_members_from_set
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self._del_member_from_set(set_name, ip)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 123, in _del_member_from_set
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self._apply(cmd)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 138, in _apply
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.execute(cmd_ns, run_as_root=True, process_input=input)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 137, in execute
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent raise RuntimeError(m)
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError:
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ipset', 'del', 'NETIPv48a17c661-2071-4527-a', '10.22.248.16']
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 1
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdin:
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout:
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: ipset v6.11: Element cannot be deleted from the set: it's not added
2015-08-19 15:13:04.779 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent
2015-08-19 15:13:04.791 6840 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Agent out of sync with plugin!
2015-08-19 15:13:05.425 6840 INFO neutron.agent.securitygroups_rpc [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Preparing filters for devices set([u'6c204926-ed7d-46a7-8fd8-56f7e6b64523', u'd31c539d-7cbd-4742-8962-d6db822e4f10', u'93cf5c57-cc1e-4aee-bfeb-202acb53905b', u'1bc145ba-b350-43fc-bcc3-dc4e0483c83c', u'0ee2ce66-d53f-44b7-b438-ca3528fcd392', u'b8a0bdcc-e632-4e63-ac8e-90e870deb84f', u'8dad87dd-8f8a-4b9f-a6ab-35abb842f765', u'6d49218d-9a78-487d-bfd0-3649b9255f1b', u'3330de77-6ab2-4d7f-a2b3-20c204797a4e', u'93d9f991-232e-4e91-bdbe-b696ce1d2297', u'3f706749-f8bb-41ab-aa4c-a0925dc67bd4', u'629c6103-aef4-4d9a-8799-7bb0fe7217f9', u'a3ced370-67d9-470b-b258-9a0ef59e106c', u'90357c87-f35a-484e-890c-329094fb23b1', u'af18726d-e40a-45f6-b408-e17d160f8d3c', u'd945724e-78f7-4320-8ded-b7a4068946f4', u'39feda3c-e418-4a6f-a426-5091730ee0d2'])

In this cloud we have a few thousand vm's, all using security groups, that are tied to hundreds of different projects. So some of our security group rules are complex and others are pretty straight forward.