Handle exception in get_instance_sorted when scattering gather results from all cells.

Bug #1734012 reported by Yikun Jiang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Undecided
Yikun Jiang

Bug Description

Description
===========
Currently, when we get servers list in multi cell, we scatter gather results from cells, but if we get back exception or timeout from cell, we will get 500 error finally.

We should handle raise or timeout after getting back all results.

Maybe we could just skip the error result like quota stuff:
https://github.com/openstack/nova/blob/e9ce5c4c95edc869ab2cf82ca0733a2821c384ad/nova/quota.py#L1865

Steps to reproduce
==================
1. raise some error exception when gather results.

2. get server list
curl -g -i -X GET http://XXX/compute/v2.1/servers -H "OpenStack-API-Version: compute 2.53" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.53" -H "X-Auth-Token: $TOKEN"
HTTP/1.1 500 Internal Server Error
Date: Wed, 22 Nov 2017 07:11:39 GMT
Server: Apache/2.4.18 (Ubuntu)
OpenStack-API-Version: compute 2.53
X-OpenStack-Nova-API-Version: 2.53
Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version
Content-Type: application/json; charset=UTF-8
Content-Length: 193
x-openstack-request-id: req-429c02e7-84db-4c2d-98fd-dd1af0186383
x-compute-request-id: req-429c02e7-84db-4c2d-98fd-dd1af0186383
Connection: close

{"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<type 'exceptions.TypeError'>", "code": 500}}

2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 336, in wrapped
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 152, in index
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=False)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 308, in _get_servers
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2394, in get_all
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions context, filters, limit, marker, fields, sort_keys, sort_dirs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/instance_list.py", line 251, in get_instance_objects_sorted
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions expected_attrs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 1196, in _make_instance_list
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions for db_inst in db_inst_list:
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/instance_list.py", line 228, in get_instances_sorted
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions for i in heapq.merge(*results.values()):
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/heapq.py", line 373, in merge
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions for itnum, it in enumerate(map(iter, iterables)):
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions TypeError: 'object' object is not iterable
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions
2017-11-22 02:11:39.935 9792 INFO nova.api.openstack.wsgi [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>
2017-11-22 02:11:39.937 9792 DEBUG nova.api.openstack.wsgi [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1029
2017-11-22 02:11:39.938 9792 INFO nova.api.openstack.requestlog [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] 10.76.6.31 "GET /compute/v2.1/servers" status: 500 len: 193 microversion: 2.53 time: 0.746751

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/522425

Changed in nova:
assignee: nobody → Yikun Jiang (yikunkero)
status: New → In Progress
Revision history for this message
Yikun Jiang (yikunkero) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Yikun Jiang (Kero) (<email address hidden>) on branch: master
Review: https://review.openstack.org/522425

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.