test_list_security_groups_list_all_tenants_filter fails with nova-net "Security group 34 not found"

[Matt Riedemann]

http://logs.openstack.org/63/84363/2/check/check-grenade-dsvm/0e4dcac/console.html

Looks like this only fails in grenade jobs with nova-network:

2014-04-23 08:39:17.718 | Traceback (most recent call last):
2014-04-23 08:39:17.718 | File "tempest/api/compute/admin/test_security_groups.py", line 76, in test_list_security_groups_list_all_tenants_filter
2014-04-23 08:39:17.718 | resp, fetched_list = self.adm_client.list_security_groups(params=param)
2014-04-23 08:39:17.718 | File "tempest/services/compute/xml/security_groups_client.py", line 51, in list_security_groups
2014-04-23 08:39:17.718 | resp, body = self.get(url)
2014-04-23 08:39:17.718 | File "tempest/common/rest_client.py", line 204, in get
2014-04-23 08:39:17.718 | return self.request('GET', url, extra_headers, headers)
2014-04-23 08:39:17.718 | File "tempest/common/rest_client.py", line 452, in request
2014-04-23 08:39:17.719 | resp, resp_body)
2014-04-23 08:39:17.719 | File "tempest/common/rest_client.py", line 496, in _error_checker
2014-04-23 08:39:17.719 | raise exceptions.NotFound(resp_body)
2014-04-23 08:39:17.719 | NotFound: Object not found
2014-04-23 08:39:17.719 | Details: <itemNotFound code="404" xmlns="http://docs.openstack.org/compute/api/v1.1"><message>Security group 34 not found.</message></itemNotFound>

message:"Security group 34 not found" AND tags:screen-n-api.txt

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU2VjdXJpdHkgZ3JvdXAgMzQgbm90IGZvdW5kXCIgQU5EIHRhZ3M6c2NyZWVuLW4tYXBpLnR4dCIsImZpZWxkcyI6WyJ0YWdzIl0sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5ODI2MTE3ODE3OCwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

This doesn't always result in a failure when it shows up in the logs.

[Matt Riedemann (mriedem)]

Looking at the test:

http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/admin/test_security_groups.py#n47

It creates two security groups with a non-admin tenant and then 2 security groups with the admin tenant. Then it tries to list security groups for all tenants using the admin client, and that fails because one of the security groups isn't found.

From the logs, there are the creates for the non-admin secgroups:

2014-04-23 08:39:17.717 | 2014-04-23 08:28:47,300 Request (SecurityGroupsTestAdminXML:test_list_security_groups_list_all_tenants_filter): 200 POST http://127.0.0.1:5000/v2.0/tokens
2014-04-23 08:39:17.717 | 2014-04-23 08:28:47,472 Request (SecurityGroupsTestAdminXML:test_list_security_groups_list_all_tenants_filter): 200 POST http://127.0.0.1:8774/v2/4c13ccd94dfd4199a2a6f848f79b7337/os-security-groups 0.171s
2014-04-23 08:39:17.717 | 2014-04-23 08:28:47,532 Request (SecurityGroupsTestAdminXML:test_list_security_groups_list_all_tenants_filter): 200 POST http://127.0.0.1:8774/v2/4c13ccd94dfd4199a2a6f848f79b7337/os-security-groups 0.057s

These are the posts for the two admin secgroups:

2014-04-23 08:39:17.717 | 2014-04-23 08:28:47,765 Request (SecurityGroupsTestAdminXML:test_list_security_groups_list_all_tenants_filter): 200 POST http://127.0.0.1:5000/v2.0/tokens
2014-04-23 08:39:17.717 | 2014-04-23 08:28:47,925 Request (SecurityGroupsTestAdminXML:test_list_security_groups_list_all_tenants_filter): 200 POST http://127.0.0.1:8774/v2/b30338440c644778b54f7c3767c8c604/os-security-groups 0.160s
2014-04-23 08:39:17.717 | 2014-04-23 08:28:48,073 Request (SecurityGroupsTestAdminXML:test_list_security_groups_list_all_tenants_filter): 200 POST http://127.0.0.1:8774/v2/b30338440c644778b54f7c3767c8c604/os-security-groups 0.146s

Then the get fails:

2014-04-23 08:39:17.717 | 2014-04-23 08:28:48,138 Request (SecurityGroupsTestAdminXML:test_list_security_groups_list_all_tenants_filter): 404 GET http://127.0.0.1:8774/v2/b30338440c644778b54f7c3767c8c604/os-security-groups?all_tenants=true 0.064s

The server is returning a 200 on the creates so not sure why one of them wouldn't exist, but definitely a race.

[Matt Riedemann (mriedem)]

This is when the first of the two admin security groups is created:

http://logs.openstack.org/63/84363/2/check/check-grenade-dsvm/0e4dcac/logs/new/screen-n-api.txt.gz#_2014-04-23_08_28_47_893

This is where I think the second admin security group is created:

http://logs.openstack.org/63/84363/2/check/check-grenade-dsvm/0e4dcac/logs/new/screen-n-api.txt.gz#_2014-04-23_08_28_48_046

The weird thing is I see a third post here:

http://logs.openstack.org/63/84363/2/check/check-grenade-dsvm/0e4dcac/logs/new/screen-n-api.txt.gz#_2014-04-23_08_28_48_071

And then it fails here:

http://logs.openstack.org/63/84363/2/check/check-grenade-dsvm/0e4dcac/logs/new/screen-n-api.txt.gz#_2014-04-23_08_28_48_134