tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_in_tenant_traffic Timeout

[YAMAMOTO Takashi]

tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_in_tenant_traffic failed with:

AssertionError: Timed out waiting for 10.100.0.20 to become reachable

http://logs.openstack.org/24/88224/2/check/check-tempest-dsvm-neutron-2/dfb1367/console.html#_2014-04-21_17_12_36_575

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQXNzZXJ0aW9uRXJyb3I6IFRpbWVkIG91dCB3YWl0aW5nIGZvciAxMC4xMDAuMC4yMCB0byBiZWNvbWUgcmVhY2hhYmxlXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTgxMjMyNjczMjl9

[Matt Riedemann (mriedem)]

This is a better query:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQXNzZXJ0aW9uRXJyb3JcXDogVGltZWQgb3V0IHdhaXRpbmcgZm9yXCIgQU5EIG1lc3NhZ2U6XCJ0byBiZWNvbWUgcmVhY2hhYmxlXCIgQU5EIHRhZ3M6Y29uc29sZSIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwMTAyMjM1MDk0M30=

Saw that here:

http://logs.openstack.org/49/93649/4/check/check-tempest-dsvm-neutron/ff96580/console.html

It was a different test that failed.

Also seeing this:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiUHVibGljIG5ldHdvcmsgY29ubmVjdGl2aXR5IGNoZWNrIGZhaWxlZFwiIEFORCB0YWdzOmNvbnNvbGUiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDEwMjI0NjkxMDd9

[Matt Riedemann (mriedem)]

Also seeing this in the dump:

Fast TSC calibration failed

Don't know if that's related or would always show up:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFzdCBUU0MgY2FsaWJyYXRpb24gZmFpbGVkXCIgQU5EIHRhZ3M6Y29uc29sZSIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwMTAyMzQ4NDU5N30=

[Matt Riedemann (mriedem)]

This might be a variant of bug 1253896.

[Matt Riedemann (mriedem)]

Opened bug 1323658 for the SSH EOFError.

[Kyle Mestery (mestery)]

Are most of these failures in grenade? It appears that way. See this query:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiUHVibGljIG5ldHdvcmsgY29ubmVjdGl2aXR5IGNoZWNrIGZhaWxlZFwiIEFORCB0YWdzOmNvbnNvbGUgQU5EIGJ1aWxkX25hbWU6Y2hlY2stZ3JlbmFkZS1kc3ZtLW5ldXRyb24iLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDEyMjE2OTA5OTl9

[Eugene Nikanorov (enikanorov)]

Regarding the log in the first Matt's comment, I've found the following discrepancy:

The server to which tempest is trying to check the connectivity supposed to have 10.100.0.20 as fixed ip and 172.24.4.61 as floating.
Server output from nova doesn't show that server with fixed ip 10.100.0.20 has floating ip:
http://logs.openstack.org/24/88224/2/check/check-tempest-dsvm-neutron-2/dfb1367/console.html.gz#_2014-04-21_17_12_36_097

In the end of the log file another output from nova could be seen: http://logs.openstack.org/24/88224/2/check/check-tempest-dsvm-neutron-2/dfb1367/console.html.gz#_2014-04-21_17_12_36_564

Here you can see that VM has expected floating ip 172.24.4.61 but fixed ip is different: 10.100.0.18
If you search by 172.24.4.61 you'll see that it was only associated with 10.100.0.18 ever.
I'm not sure I'm interpreting logs correctly, at least it confuses me that logs related to the failing tests show nova output for different servers.

Anyway my conclusion is that there is a discrepancy between fixed/floating ips of the instance and test's expectation.
I was not able to find out the root cause yet.

[Eugene Nikanorov (enikanorov)]

Marking as incomplete as this have not been hit for quite a while

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]