neutron

[Fullstack] iptables-restore fails with RuntimeError for ipset

Bug #1575033 reported by venkata anil on 2016-04-26

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Undecided	Jakub Libosvar

Bug Description

The following Trace is seen ovs_neutron_agent while running functional tests

http://logs.openstack.org/59/307159/5/check/gate-neutron-dsvm-fullstack/e1f25d4/logs/TestDVRL3Agent.test_snat_and_floatingip/neutron-openvswitch-agent--2016-04-22--12-33-51-032511.log.txt.gz

2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-f775b822-c40d-4d94-a2ec-005fb8b038fb - - - - -] Error while processing VIF ports
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1992, in rpc_loop
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, ovs_restarted)
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1623, in process_network_ports
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 292, in setup_port_filters
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 147, in decorated_function
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 172, in prepare_devices_filter
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.prepare_port_filter(device)
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.gen.next()
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/firewall.py", line 129, in defer_apply
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.filter_defer_apply_off()
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/iptables_firewall.py", line 824, in filter_defer_apply_off
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/iptables_firewall.py", line 824, in filter_defer_apply_off
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.iptables.defer_apply_off()
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/iptables_manager.py", line 468, in defer_apply_off
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply()
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/iptables_manager.py", line 482, in _apply
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return self._apply_synchronized()
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/iptables_manager.py", line 559, in _apply_synchronized
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent '\n'.join(log_lines))
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.force_reraise()
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent six.reraise(self.type_, self.value, self.tb)
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/iptables_manager.py", line 538, in _apply_synchronized
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent run_as_root=True)
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 137, in execute
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise RuntimeError(msg)
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: Exit code: 2; Stdin: # Generated by iptables_manager
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *filter
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent :ovs_agent.py-ib272c437-0 - [0:0]

2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent -I ovs_agent.py-PREROUTING 2 -m physdev --physdev-in tapb272c437-05 -j CT --zone 1
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent COMMIT
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent # Completed by iptables_manager
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ; Stdout: ; Stderr: iptables-restore v1.4.21: Set NIPv42a80b4e9-6d6f-4847-9e62- doesn't exist.
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Error occurred at line: 10
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Try `iptables-restore -h' or 'iptables-restore --help' for more information.
2016-04-22 12:34:33.670 27936 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent

Tags:

venkata anil (anil-venkata) on 2016-04-26

Changed in neutron:
assignee:	nobody → venkata anil (anil-venkata)

Revision history for this message

Doug Wiegley (dougwig) wrote on 2016-04-26:

What was it trying to restore? Could you do it by hand? Any rootwrap errors?

Changed in neutron:
status:	New → Incomplete

Revision history for this message

venkata anil (anil-venkata) wrote on 2016-05-02:

This is happening in fullstack tests for distributed ports(DVR, HA).

When I run fullstack tests for DVR, ipset rules for security groups are failing like below

http://logs.openstack.org/59/307159/6/check/gate-neutron-dsvm-fullstack/e171836/logs/TestDVRL3Agent.test_snat_and_floatingip/neutron-openvswitch-agent--2016-04-26--16-10-13-061698.log.txt.gz#_2016-04-26_16_10_24_711

http://marker.to/f7D2Pr

http://marker.to/FhN6Ye

In fullstack tests, vm is implemented by creating a namespace and attaching it's tap device to agent's integration bridge without linux bridge. As vm's interface is directly connected to integration bridge without linux bridge, I think we can't use security groups. So the solution is to disable security groups for full stack tests.

Need to check with Assaf.

tags:

added: fullstack

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-02: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/311766

Changed in neutron:
assignee:	venkata anil (anil-venkata) → Jakub Libosvar (libosvar)
status:	Incomplete → In Progress

Assaf Muller (amuller) on 2016-05-02

summary:

- iptables-restore fails with RuntimeError for ipset
+ [Fullstack] iptables-restore fails with RuntimeError for ipset

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-02: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/311766
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=77456735cbc9066dda8186937cb6ee2073833123
Submitter: Jenkins
Branch: master

commit 77456735cbc9066dda8186937cb6ee2073833123
Author: Jakub Libosvar <email address hidden>
Date: Mon May 2 17:35:49 2016 +0200

fullstack: Use noop firewall

    Currently fullstack tests don't use hybrid plugging but they use hybrid
    firewall by default. Using iptables is not segregated and OVS agents
    running in parallel may interfere between each other.

This patch removes using iptables in ovs agents per created port.

Change-Id: Ic15c942360a4a6ecf0eb5d8ecbbc54e59b10431d
Closes-Bug: #1575033

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-26: Fix proposed to neutron (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/321756

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-26: Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/321757

Revision history for this message

Doug Hellmann (doug-hellmann) wrote on 2016-06-03: Fix included in openstack/neutron 9.0.0.0b1

This issue was fixed in the openstack/neutron 9.0.0.0b1 development milestone.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-06: Fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/321757
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6d808cc6a7e36f220e5a80b774abd215b8ac4c54
Submitter: Jenkins
Branch: stable/liberty

commit 6d808cc6a7e36f220e5a80b774abd215b8ac4c54
Author: Jakub Libosvar <email address hidden>
Date: Mon May 2 17:35:49 2016 +0200

fullstack: Use noop firewall

This patch removes using iptables in ovs agents per created port.

    Change-Id: Ic15c942360a4a6ecf0eb5d8ecbbc54e59b10431d
    Closes-Bug: #1575033
    (cherry picked from commit 77456735cbc9066dda8186937cb6ee2073833123)

tags:	added: in-stable-liberty
tags:	added: in-stable-mitaka

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-06: Fix merged to neutron (stable/mitaka)

Reviewed: https://review.openstack.org/321756
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=804fc18cf292c2dc683327e58cdba0d3da23731d
Submitter: Jenkins
Branch: stable/mitaka

commit 804fc18cf292c2dc683327e58cdba0d3da23731d
Author: Jakub Libosvar <email address hidden>
Date: Mon May 2 17:35:49 2016 +0200

fullstack: Use noop firewall

This patch removes using iptables in ovs agents per created port.

    Change-Id: Ic15c942360a4a6ecf0eb5d8ecbbc54e59b10431d
    Closes-Bug: #1575033
    (cherry picked from commit 77456735cbc9066dda8186937cb6ee2073833123)