test_dhcp6_stateless_from_os fails to ping intermittently

[Matt Riedemann]

This appears to be spiking since 7/22:

http://logs.openstack.org/80/200380/3/gate/gate-tempest-dsvm-neutron-full/2b49b0d/console.html#_2015-07-22_09_57_50_146

2015-07-22 09:57:50.146 | tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac[compute,id-dec222b1-180c-4098-b8c5-cc1b8342d611,network]
2015-07-22 09:57:50.146 | ------------------------------------------------------------------------------------------------------------------------------------
2015-07-22 09:57:50.146 |
2015-07-22 09:57:50.146 | Captured traceback:
2015-07-22 09:57:50.146 | ~~~~~~~~~~~~~~~~~~~
2015-07-22 09:57:50.147 | Traceback (most recent call last):
2015-07-22 09:57:50.147 | File "tempest/test.py", line 126, in wrapper
2015-07-22 09:57:50.147 | return f(self, *func_args, **func_kwargs)
2015-07-22 09:57:50.147 | File "tempest/scenario/test_network_v6.py", line 183, in test_multi_prefix_slaac
2015-07-22 09:57:50.147 | self._prepare_and_test(address6_mode='slaac', n_subnets6=2)
2015-07-22 09:57:50.147 | File "tempest/scenario/test_network_v6.py", line 150, in _prepare_and_test
2015-07-22 09:57:50.147 | result = sshv4_2.ping_host(ips_from_api_1['4'])
2015-07-22 09:57:50.147 | File "tempest/common/utils/linux/remote_client.py", line 101, in ping_host
2015-07-22 09:57:50.147 | return self.exec_command(cmd)
2015-07-22 09:57:50.147 | File "tempest/common/utils/linux/remote_client.py", line 56, in exec_command
2015-07-22 09:57:50.147 | return self.ssh_client.exec_command(cmd)
2015-07-22 09:57:50.147 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/tempest_lib/common/ssh.py", line 146, in exec_command
2015-07-22 09:57:50.148 | strerror=''.join(err_data))
2015-07-22 09:57:50.148 | tempest_lib.exceptions.SSHExecCommandFailed: Command 'set -eu -o pipefail; PATH=$PATH:/sbin; ping -c1 -w1 -s56 10.100.0.3', exit status: 1, Error:

There are so many errors in the neutron logs for false negatives that I can't really tell what could be causing the failures, see related bug 1455320 and related bug 1477190.

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQ29tbWFuZCAnc2V0IC1ldSAtbyBwaXBlZmFpbDsgUEFUSD0kUEFUSDovc2JpbjsgcGluZyAtYzEgLXcxIC1zNTZcIiBBTkQgTk9UIGJ1aWxkX3F1ZXVlOlwiZXhwZXJpbWVudGFsXCIgQU5EIHRhZ3M6XCJjb25zb2xlXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0Mzc1NzU4NzgzNTYsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=

[Kyle Mestery (mestery)]

Henry, can you triage this one and have someone have a crack at fixing it once you've identified the cause? Thanks!

[Henry Gessau (gessau)]

Is that logstash query correct? It's hitting on some DEBUG logs.

When I search for SSHExecCommandFailed I get no hits.

[Matt Riedemann (mriedem)]

Maybe this is a better query?

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIHBpbmcgSVA6IDIwMDM6OjEgdmlhIGEgc3NoIGNvbm5lY3Rpb24gZnJvbTpcIiBBTkQgTk9UIGJ1aWxkX3F1ZXVlOlwiZXhwZXJpbWVudGFsXCIgQU5EIHRhZ3M6XCJjb25zb2xlXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0Mzk0MTE1NTAyNTgsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=

However, I'm seeing that for failures in test_dhcp6_stateless_from_os rather than test_multi_prefix_slaac:

http://logs.openstack.org/83/210483/9/check/gate-tempest-dsvm-neutron-linuxbridge/9fb8afe/console.html#_2015-08-11_14_31_08_616

[Kyle Mestery (mestery)]

Using Matt's query from #3, there are 813 hits in the last 7 days here, though it's a bit sporadic. Henry, any updates on this one given the number of hits it's seeing?

[Henry Gessau (gessau)]

Searching for "Failed to ping IP: 2003::1 via a ssh connection from:" results in 45 or more hits for a single failure since there are about that many retries before the test gives up. So the real hit rate is less than 20 in the last 7 days.

Nevertheless, I will look into this issue which as noted now seems to be in the test_dhcp6_stateless_from_os test.

[Kyle Mestery (mestery)]

Out of liberty-rc1.

[Armando Migliaccio (armando-migliaccio)]

I am baffled that this does not happen on the gate queue.

When I use this query:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIHBpbmcgSVA6IDIwMDM6OjEgdmlhIGEgc3NoIGNvbm5lY3Rpb24gZnJvbTpcIiBBTkQgdGFnczpcImNvbnNvbGVcIiBBTkQgYnVpbGRfbmFtZTpcImdhdGUtdGVtcGVzdC1kc3ZtLW5ldXRyb24tZnVsbFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDQzNDgwOTkzMDIwLCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

message:"Failed to ping IP: 2003::1 via a ssh connection from:" AND tags:"console" AND build_name:"gate-tempest-dsvm-neutron-full"

I could see only failures in the check queue.

Anyhow this should be telling:

https://review.openstack.org/#/c/228173/

[Armando Migliaccio (armando-migliaccio)]

weirdly enough none of them appear on Neutron changes...

[Armando Migliaccio (armando-migliaccio)]

I mean for those that run the gate-tempest-dsvm-neutron-full job only.

[Armando Migliaccio (armando-migliaccio)]

And by the way, this query is a lot better, as it gets rid of the duplicates:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQXNzZXJ0aW9uRXJyb3I6IEZhbHNlIGlzIG5vdCB0cnVlIDogVGltZWQgb3V0IHdhaXRpbmcgZm9yIDIwMDM6OjEgdG8gYmVjb21lIHJlYWNoYWJsZSBmcm9tXCIgQU5EIHRhZ3M6XCJjb25zb2xlXCIgQU5EIGJ1aWxkX25hbWU6XCJnYXRlLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWZ1bGxcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MzQ4MTU2OTg0NSwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

Or even better:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQXNzZXJ0aW9uRXJyb3I6IEZhbHNlIGlzIG5vdCB0cnVlIDogVGltZWQgb3V0IHdhaXRpbmcgZm9yIDIwMDM6OjEgdG8gYmVjb21lIHJlYWNoYWJsZSBmcm9tXCIgQU5EIHRhZ3M6XCJjb25zb2xlXCIgQU5EIGJ1aWxkX25hbWU6XCJnYXRlLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWZ1bGxcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MzQ4MTY0NzUwNywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

Three occurrences in the last 7 days.

[Armando Migliaccio (armando-migliaccio)]

Adjusting the title as it's no longer relevant.

[Armando Migliaccio (armando-migliaccio)]

0 occurrences in the last 7 days.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]