Gate failures for neutron in test_dualnet_multi_prefix_slaac

[Davanum Srinivas (DIMS)]

24 hits in 7 days for logstash query : message:"in test_dualnet_multi_prefix_slaac" AND voting:1

2016-02-02 14:35:49.054 | Captured traceback:
2016-02-02 14:35:49.054 | ~~~~~~~~~~~~~~~~~~~
2016-02-02 14:35:49.054 | Traceback (most recent call last):
2016-02-02 14:35:49.054 | File "tempest/test.py", line 113, in wrapper
2016-02-02 14:35:49.055 | return f(self, *func_args, **func_kwargs)
2016-02-02 14:35:49.055 | File "tempest/scenario/test_network_v6.py", line 246, in test_dualnet_multi_prefix_slaac
2016-02-02 14:35:49.055 | dualnet=True)
2016-02-02 14:35:49.055 | File "tempest/scenario/test_network_v6.py", line 155, in _prepare_and_test
2016-02-02 14:35:49.055 | sshv4_1, ips_from_api_1, sid1 = self.prepare_server(networks=net_list)
2016-02-02 14:35:49.055 | File "tempest/scenario/test_network_v6.py", line 128, in prepare_server
2016-02-02 14:35:49.055 | username=username)
2016-02-02 14:35:49.056 | File "tempest/scenario/manager.py", line 390, in get_remote_client
2016-02-02 14:35:49.056 | linux_client.validate_authentication()
2016-02-02 14:35:49.056 | File "tempest/common/utils/linux/remote_client.py", line 63, in validate_authentication
2016-02-02 14:35:49.056 | self.ssh_client.test_connection_auth()
2016-02-02 14:35:49.056 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/tempest_lib/common/ssh.py", line 172, in test_connection_auth
2016-02-02 14:35:49.056 | connection = self._get_ssh_connection()
2016-02-02 14:35:49.056 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/tempest_lib/common/ssh.py", line 87, in _get_ssh_connection
2016-02-02 14:35:49.057 | password=self.password)
2016-02-02 14:35:49.057 | tempest_lib.exceptions.SSHTimeout: Connection to the 172.24.5.141 via SSH timed out.
2016-02-02 14:35:49.057 | User: cirros, Password: None

[Sean M. Collins (scollins)]

This is most likely a symptom of another bug. Tempest SSHs into a created instance over v4, then tries to SSH over v6 to a second compute node that was created.

[Armando Migliaccio (armando-migliaccio)]

All gate failures should be treated as critical issues and they cause serious disruption to the normal development routine.

[Armando Migliaccio (armando-migliaccio)]

Obviously there are some that are more or less severe (frequent) than others. To be hitting 24 times in the last few days it seems this is spiking and we should look into it. I wonder if this falls under the category of bugs:

bug 1509004
bug 1477192

[Armando Migliaccio (armando-migliaccio)]

Btw on second inspection, this failed once in the gate queue in the last 7 days.

[Brian Haley (brian-haley)]

http://logs.openstack.org/08/275508/1/check/gate-tempest-dsvm-neutron-full/5794f8e/logs/ is an example of this error still happening periodically.

And this is a link mtreinish posted in irc showing the status of the SLAAC test. Strangely this doesn't show the failure mentioned above, so more work is required:

http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac?groupKey=project&resolutionKey=hour&duration=P3M

[Armando Migliaccio (armando-migliaccio)]

0 gate occurrences in the past 7 days.

[Launchpad Janitor (janitor)]

[Expired for OpenStack-Gate because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Armando Migliaccio (armando-migliaccio)]

http://logs.openstack.org/83/339283/16/check/gate-tempest-dsvm-neutron-dvr/df8047b/logs/

[Armando Migliaccio (armando-migliaccio)]

22 occurrences during the last 7 days.

[Carl Baldwin (carl-baldwin)]

The query comes up empty for the last 7 days. I wonder if the partial fix for this one [1] got it. It is the same test and could be two manifestations of the same bug.

[1] https://bugs.launchpad.net/neutron/+bug/1609540

[Carl Baldwin (carl-baldwin)]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20test_dualnet_multi_prefix_slaac%5C%22%20AND%20voting%3A1

[Matthew Treinish (treinish)]

The test has been skipped in the gate since Aug. 3rd with: https://review.openstack.org/#/c/275457/ so I don't think the lack of elastic search hits is necessarily indicative of a fix.

[Carl Baldwin (carl-baldwin)]

I missed that the test was disabled. Changing to New and down-grading priority since it isn't a gate failure technically.

[Oleg Bondarev (obondarev)]

Trying to trigger a failure here: https://review.openstack.org/#/c/374106/

[Miguel Lavalle (minsel)]

I have been watching this bug in Kibana over the past 2 weeks, with a total of 43 hits. All of them were caused by:

1) Broken linuxbridge job, which was documented here: http://lists.openstack.org/pipermail/openstack-dev/2016-November/108164.html

2) The failures are due to bugs / errors in the corresponding patchset

As a consequence, marking it as incomplete

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for OpenStack-Gate because there has been no activity for 60 days.]