neutron

ARPSpoofOFCtlTestCase functional gate tests failing intermittently

Bug #1547486 reported by Sreekumar S
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Confirmed
High
Unassigned

Bug Description

Tests inside neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase seems to fail intermittently. The root cause seems like to be the same for all the tests...

Here are two different stack traces from two different patches...

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2016-02-19 09:49:54.288 | 2016-02-19 09:49:54.264 | ==============================
2016-02-19 09:49:54.312 | 2016-02-19 09:49:54.279 | Failed 1 tests - output below:
2016-02-19 09:49:54.321 | 2016-02-19 09:49:54.292 | ==============================
2016-02-19 09:49:54.332 | 2016-02-19 09:49:54.308 |
2016-02-19 09:49:54.335 | 2016-02-19 09:49:54.310 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_disable_network_port(native)
2016-02-19 09:49:54.339 | 2016-02-19 09:49:54.312 | ---------------------------------------------------------------------------------------------------------------
2016-02-19 09:49:54.339 | 2016-02-19 09:49:54.314 |
2016-02-19 09:49:54.343 | 2016-02-19 09:49:54.316 | Captured traceback:
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.317 | ~~~~~~~~~~~~~~~~~~~
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.322 | Traceback (most recent call last):
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.328 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 242, in test_arp_spoof_disable_network_port
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.351 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2)
2016-02-19 09:49:54.378 | 2016-02-19 09:49:54.354 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping
2016-02-19 09:49:54.380 | 2016-02-19 09:49:54.357 | dst_ip])
2016-02-19 09:49:54.385 | 2016-02-19 09:49:54.361 | File "neutron/agent/linux/ip_lib.py", line 885, in execute
2016-02-19 09:49:54.386 | 2016-02-19 09:49:54.363 | log_fail_as_error=log_fail_as_error, **kwargs)
2016-02-19 09:49:54.389 | 2016-02-19 09:49:54.365 | File "neutron/agent/linux/utils.py", line 140, in execute
2016-02-19 09:49:54.392 | 2016-02-19 09:49:54.368 | raise RuntimeError(msg)
2016-02-19 09:49:54.395 | 2016-02-19 09:49:54.371 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
2016-02-19 09:49:54.397 | 2016-02-19 09:49:54.374 |
2016-02-19 09:49:54.420 | 2016-02-19 09:49:54.376 | --- 192.168.0.2 ping statistics ---
2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.378 | 2 packets transmitted, 0 received, 100% packet loss, time 1005ms
2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.379 |
2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.382 | ; Stderr:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:55.965 | 2016-02-19 09:12:55.947 | ==============================
2016-02-19 09:12:55.972 | 2016-02-19 09:12:55.951 | Failed 1 tests - output below:
2016-02-19 09:12:55.977 | 2016-02-19 09:12:55.959 | ==============================
2016-02-19 09:12:55.986 | 2016-02-19 09:12:55.967 |
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.976 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_normal_traffic(native)
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.978 | ----------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:56.009 | 2016-02-19 09:12:55.990 |
2016-02-19 09:12:56.025 | 2016-02-19 09:12:56.006 | Captured pythonlogging:
2016-02-19 09:12:56.034 | 2016-02-19 09:12:56.016 | ~~~~~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.036 | 2016-02-19 09:12:56.018 | DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.038 | 2016-02-19 09:12:56.020 | DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.046 | 2016-02-19 09:12:56.028 |
2016-02-19 09:12:56.051 | 2016-02-19 09:12:56.030 |
2016-02-19 09:12:56.060 | 2016-02-19 09:12:56.036 | Captured traceback:
2016-02-19 09:12:56.062 | 2016-02-19 09:12:56.044 | ~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.066 | 2016-02-19 09:12:56.048 | Traceback (most recent call last):
2016-02-19 09:12:56.071 | 2016-02-19 09:12:56.053 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 145, in test_arp_spoof_doesnt_block_normal_traffic
2016-02-19 09:12:56.073 | 2016-02-19 09:12:56.055 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2)
2016-02-19 09:12:56.078 | 2016-02-19 09:12:56.058 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping
2016-02-19 09:12:56.080 | 2016-02-19 09:12:56.062 | dst_ip])
2016-02-19 09:12:56.082 | 2016-02-19 09:12:56.064 | File "neutron/agent/linux/ip_lib.py", line 885, in execute
2016-02-19 09:12:56.084 | 2016-02-19 09:12:56.066 | log_fail_as_error=log_fail_as_error, **kwargs)
2016-02-19 09:12:56.086 | 2016-02-19 09:12:56.068 | File "neutron/agent/linux/utils.py", line 140, in execute
2016-02-19 09:12:56.094 | 2016-02-19 09:12:56.076 | raise RuntimeError(msg)
2016-02-19 09:12:56.102 | 2016-02-19 09:12:56.080 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
2016-02-19 09:12:56.109 | 2016-02-19 09:12:56.091 |
2016-02-19 09:12:56.121 | 2016-02-19 09:12:56.101 | --- 192.168.0.2 ping statistics ---
2016-02-19 09:12:56.132 | 2016-02-19 09:12:56.113 | 2 packets transmitted, 0 received, 100% packet loss, time 1003ms
2016-02-19 09:12:56.137 | 2016-02-19 09:12:56.118 |
2016-02-19 09:12:56.138 | 2016-02-19 09:12:56.121 | ; Stderr:
2016-02-19 09:12:56.141 | 2016-02-19 09:12:56.123 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

2016-02-19 09:12:55.965 | 2016-02-19 09:12:55.947 | ==============================
2016-02-19 09:12:55.972 | 2016-02-19 09:12:55.951 | Failed 1 tests - output below:
2016-02-19 09:12:55.977 | 2016-02-19 09:12:55.959 | ==============================
2016-02-19 09:12:55.986 | 2016-02-19 09:12:55.967 |
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.976 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_normal_traffic(native)
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.978 | ----------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:56.009 | 2016-02-19 09:12:55.990 |
2016-02-19 09:12:56.025 | 2016-02-19 09:12:56.006 | Captured pythonlogging:
2016-02-19 09:12:56.034 | 2016-02-19 09:12:56.016 | ~~~~~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.036 | 2016-02-19 09:12:56.018 | DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.038 | 2016-02-19 09:12:56.020 | DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.046 | 2016-02-19 09:12:56.028 |
2016-02-19 09:12:56.051 | 2016-02-19 09:12:56.030 |
2016-02-19 09:12:56.060 | 2016-02-19 09:12:56.036 | Captured traceback:
2016-02-19 09:12:56.062 | 2016-02-19 09:12:56.044 | ~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.066 | 2016-02-19 09:12:56.048 | Traceback (most recent call last):
2016-02-19 09:12:56.071 | 2016-02-19 09:12:56.053 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 145, in test_arp_spoof_doesnt_block_normal_traffic
2016-02-19 09:12:56.073 | 2016-02-19 09:12:56.055 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2)
2016-02-19 09:12:56.078 | 2016-02-19 09:12:56.058 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping
2016-02-19 09:12:56.080 | 2016-02-19 09:12:56.062 | dst_ip])
2016-02-19 09:12:56.082 | 2016-02-19 09:12:56.064 | File "neutron/agent/linux/ip_lib.py", line 885, in execute
2016-02-19 09:12:56.084 | 2016-02-19 09:12:56.066 | log_fail_as_error=log_fail_as_error, **kwargs)
2016-02-19 09:12:56.086 | 2016-02-19 09:12:56.068 | File "neutron/agent/linux/utils.py", line 140, in execute
2016-02-19 09:12:56.094 | 2016-02-19 09:12:56.076 | raise RuntimeError(msg)
2016-02-19 09:12:56.102 | 2016-02-19 09:12:56.080 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
2016-02-19 09:12:56.109 | 2016-02-19 09:12:56.091 |
2016-02-19 09:12:56.121 | 2016-02-19 09:12:56.101 | --- 192.168.0.2 ping statistics ---
2016-02-19 09:12:56.132 | 2016-02-19 09:12:56.113 | 2 packets transmitted, 0 received, 100% packet loss, time 1003ms
2016-02-19 09:12:56.137 | 2016-02-19 09:12:56.118 |
2016-02-19 09:12:56.138 | 2016-02-19 09:12:56.121 | ; Stderr:
2016-02-19 09:12:56.141 | 2016-02-19 09:12:56.123 |

Tags: gate-failure
Revision history for this message
Rossella Sblendido (rossella-o) wrote :

In logstash I see 21 hits for the last 2 days

Changed in neutron:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Rossella Sblendido (rossella-o) wrote :

Fixed by https://review.openstack.org/#/c/286428/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.