GET os-availability-zone/detail returns 500

[Joe Gordon]

http://logs.openstack.org/19/46219/6/check/gate-tempest-devstack-vm-postgres-full/f2a8775/logs/screen-n-api.txt.gz#_2013-09-17_19_17_28_163

2013-09-17 19:17:28.163 ERROR nova.api.openstack [req-8a9da02a-d0dd-4122-9387-3c353c1f61f9 AvailabilityZoneAdminTestJSON420684138-user AvailabilityZoneAdminTestJSON328161858-tenant] Caught error: 'nova'
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack Traceback (most recent call last):
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 119, in __call__
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack return req.get_response(self.application)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack application, catch_exc_info=False)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack return resp(environ, start_response)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/opt/stack/new/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 545, in __call__
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack return self.app(env, start_response)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack return resp(environ, start_response)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack return resp(environ, start_response)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack response = self.app(environ, start_response)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack return resp(environ, start_response)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 912, in __call__
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack content_type, body, accept)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 971, in _process_stack
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 1052, in dispatch
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack return method(req=request, **action_args)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/compute/contrib/availability_zone.py", line 158, in detail
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack return self._describe_availability_zones_verbose(context)
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/compute/contrib/availability_zone.py", line 127, in _describe_availability_zones_verbose
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack for host in zone_hosts[zone]:
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack KeyError: 'nova'
2013-09-17 19:17:28.163 20963 TRACE nova.api.openstack
2013-09-17 19:17:28.166 INFO nova.api.openstack [req-8a9da02a-d0dd-4122-9387-3c353c1f61f9 AvailabilityZoneAdminTestJSON420684138-user AvailabilityZoneAdminTestJSON328161858-tenant] http://127.0.0.1:8774/v2/401a55fcac9346e9a4651fa4998cf37e/os-availability-zone/detail returned with HTTP 500
2013-09-17 19:17:28.166 DEBUG nova.api.openstack.wsgi [req-8a9da02a-d0dd-4122-9387-3c353c1f61f9 AvailabilityZoneAdminTestJSON420684138-user AvailabilityZoneAdminTestJSON328161858-tenant] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:1198
2013-09-17 19:17:28.171 INFO nova.osapi_compute.wsgi.server [req-8a9da02a-d0dd-4122-9387-3c353c1f61f9 AvailabilityZoneAdminTestJSON420684138-user AvailabilityZoneAdminTestJSON328161858-tenant] 127.0.0.1 "GET /v2/401a55fcac9346e9a4651fa4998cf37e/os-availability-zone/detail HTTP/1.1" status: 500 len: 354 time: 0.0626421

2013-09-17 19:17:28.190 INFO nova.osapi_compute.wsgi.server [req-65257847-14d1-436b-a8c4-8f549c2237e3 FlavorsAdminTestXML1399969679-user FlavorsAdminTestXML1439560052-tenant] 127.0.0.1 "DELETE /v2/23b96b3d3887414e98826e0076812b6c/flavors/1365744031 HTTP/1.1" status: 202 len: 204 time: 0.0436058

From http://logs.openstack.org/19/46219/6/check/gate-tempest-devstack-vm-postgres-full/f2a8775/testr_results.html.gz

[Joe Gordon (jogo)]

logstash query: "KeyError: 'nova'" AND @fields.filename:"logs/screen-n-api.txt" AND @fields.build_status:"FAILURE" AND "_describe_availability_zones_verbose"

[Joe Gordon (jogo)]

Marking as critical since we are seeing this as a transient failure in tempest runs.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/47250

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/47250
Committed: http://github.com/openstack/nova/commit/d1d1515a610bee2c78e8275b63b09a0301dd9206
Submitter: Jenkins
Branch: master

commit d1d1515a610bee2c78e8275b63b09a0301dd9206
Author: Davanum Srinivas <email address hidden>
Date: Wed Sep 18 16:46:35 2013 -0400

    Fix for os-availability-zone/detail returning 500

    if there are no enabled_services, then zone_hosts is very likely
    to be empty, which means zone_hosts['nova'] will fail. So let's
    use get with a default of empty array to cover this case. Added
    a fix/test for v3 API as well.

    Fixes LP# 1226791

    Change-Id: I602c96242291fd9a4bdf67af6445aa935ec7f7e3