Gate failures for neutron in TestGettingAddress

[Davanum Srinivas (DIMS)]

Logstash query:
message:"AssertionError: False is not true : Timed out waiting for 2003::1 to become reachable" AND voting:1

Example:
http://logs.openstack.org/95/262395/7/check/gate-tempest-dsvm-neutron-full/4293c5f/console.html#_2016-01-12_07_44_55_971

Jobs that fail:
gate-tempest-dsvm-neutron-full
gate-tempest-dsvm-neutron-dvr
gate-tempest-dsvm-neutron-src-taskflow
ate-tempest-dsvm-neutron-src-futurist

Tests that Fail include:
tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaa

Traceback:
2016-01-12 07:44:55.972 | Captured traceback:
2016-01-12 07:44:55.972 | ~~~~~~~~~~~~~~~~~~~
2016-01-12 07:44:55.972 | Traceback (most recent call last):
2016-01-12 07:44:55.972 | File "tempest/test.py", line 113, in wrapper
2016-01-12 07:44:55.972 | return f(self, *func_args, **func_kwargs)
2016-01-12 07:44:55.972 | File "tempest/scenario/test_network_v6.py", line 216, in test_dhcp6_stateless_from_os
2016-01-12 07:44:55.973 | self._prepare_and_test(address6_mode='dhcpv6-stateless')
2016-01-12 07:44:55.973 | File "tempest/scenario/test_network_v6.py", line 195, in _prepare_and_test
2016-01-12 07:44:55.973 | self.subnets_v6[i].gateway_ip)
2016-01-12 07:44:55.973 | File "tempest/scenario/test_network_v6.py", line 205, in _check_connectivity
2016-01-12 07:44:55.973 | (dest, source.ssh_client.host)
2016-01-12 07:44:55.973 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/unittest2/case.py", line 702, in assertTrue
2016-01-12 07:44:55.973 | raise self.failureException(msg)
2016-01-12 07:44:55.974 | AssertionError: False is not true : Timed out waiting for 2003::1 to become reachable from 172.24.5.111

[Davanum Srinivas (DIMS) (dims-v)]

Some of the jobs that fail include:

gate-tempest-dsvm-neutron-full
gate-tempest-dsvm-neutron-dvr
gate-tempest-dsvm-neutron-src-taskflow
ate-tempest-dsvm-neutron-src-futurist

[Ihar Hrachyshka (ihar-hrachyshka)]

Affects liberty too: http://logs.openstack.org/18/327018/4/gate/gate-tempest-dsvm-neutron-dvr/dfb03bd/logs/testr_results.html.gz

[Armando Migliaccio (armando-migliaccio)]

Don't see it occur on the gate queue for as long as logstash can remember.

[Launchpad Janitor (janitor)]

[Expired for tempest because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Kieran Spear (kspear)]

This is still happening:

2016-10-03 17:16:46.741866 | tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless[compute,id-cf1c4425-766b-45b8-be35-e2959728eb00,network]
2016-10-03 17:16:46.741911 | -------------------------------------------------------------------------------------------------------------------------------------------------------
2016-10-03 17:16:46.741918 |
2016-10-03 17:16:46.741929 | Captured traceback:
2016-10-03 17:16:46.741940 | ~~~~~~~~~~~~~~~~~~~
2016-10-03 17:16:46.741955 | Traceback (most recent call last):
2016-10-03 17:16:46.741972 | File "tempest/test.py", line 107, in wrapper
2016-10-03 17:16:46.741989 | return f(self, *func_args, **func_kwargs)
2016-10-03 17:16:46.742017 | File "tempest/scenario/test_network_v6.py", line 256, in test_dualnet_multi_prefix_dhcpv6_stateless
2016-10-03 17:16:46.742029 | dualnet=True)
2016-10-03 17:16:46.742053 | File "tempest/scenario/test_network_v6.py", line 203, in _prepare_and_test
2016-10-03 17:16:46.742069 | self.subnets_v6[i]['gateway_ip'])
2016-10-03 17:16:46.742102 | File "tempest/scenario/test_network_v6.py", line 213, in _check_connectivity
2016-10-03 17:16:46.742117 | (dest, source.ssh_client.host)
2016-10-03 17:16:46.742148 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/unittest2/case.py", line 702, in assertTrue
2016-10-03 17:16:46.742164 | raise self.failureException(msg)
2016-10-03 17:16:46.742192 | AssertionError: False is not true : Timed out waiting for 2003::1 to become reachable from 172.24.5.9

http://logs.openstack.org/45/371145/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial-newton/78ca06d/console.html

[Andrea Frittoli (andrea-frittoli)]

Two hits in the past 30dd.

[Andrea Frittoli (andrea-frittoli)]

It's not clear where the issue lies, but given the very low hit rate, I don't think it's worth investing much in investigating that. I will set this to low priority.

[Andrea Frittoli (andrea-frittoli)]

There's no indication of this being a bug in Tempest. Since it happens in different tests as well, I'm setting this as incomplete for Tempest.

[Jordan Pittier (jordan-pittier)]

I just saw this in test_dualnet_multi_prefix_dhcpv6_stateless. Maybe we should skip the test.

[Andrea Frittoli (andrea-frittoli)]

Looking at OpenStack-Health [0] the failure rate is still rather low (~1%) so I'm not sure we should skip the test just yet. However the issue definitely started showing on Nov 29th (the bug was filed on Dec 1st), so it may be worth looking in Neutron for changes merged on that date that could have caused this.

[0] http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless