Timeout of cell query action leads to result raise exception

[ramblejack]

2018-05-22 21:24:42.534 4170 INFO nova.api.openstack.wsgi [req-22dfdd81-5d54-4c17-a9c9-a2f52e668a9d f5e45135156e4249a1e3164e6168f592 72a46594e674471184ff7e6ab199c775 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>
2018-05-22 21:24:42.535 4170 INFO nova.osapi_compute.wsgi.server [req-22dfdd81-5d54-4c17-a9c9-a2f52e668a9d f5e45135156e4249a1e3164e6168f592 72a46594e674471184ff7e6ab199c775 - default default] 192.168.2.246 "GET /v2.1/servers/detail HTTP/1.1" status: 500 len: 626 time: 120.1441171
2018-05-22 22:03:57.815 4170 WARNING nova.context [req-bb1698d5-a798-4f0e-82f1-396110f52d62 f5e45135156e4249a1e3164e6168f592 72a46594e674471184ff7e6ab199c775 - default default] Timed out waiting for response from cell 00000000-0000-0000-0000-000000000000: CellTimeout: Timeout waiting for response from cell
2018-05-22 22:03:57.816 4170 WARNING nova.context [req-bb1698d5-a798-4f0e-82f1-396110f52d62 f5e45135156e4249a1e3164e6168f592 72a46594e674471184ff7e6ab199c775 - default default] Timed out waiting for response from cell 15130589-8865-4b06-a2b8-9aab7ee3f546: CellTimeout: Timeout waiting for response from cell
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi [req-bb1698d5-a798-4f0e-82f1-396110f52d62 f5e45135156e4249a1e3164e6168f592 72a46594e674471184ff7e6ab199c775 - default default] Unexpected exception in API method: TypeError: 'object' object is not iterable
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 788, in wrapped
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 165, in detail
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True)
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 308, in _get_servers
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2462, in get_all
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi context, filters, limit, marker, fields, sort_keys, sort_dirs)
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/instance_list.py", line 107, in get_instance_objects_sorted
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi expected_attrs)
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1199, in _make_instance_list
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi for db_inst in db_inst_list:
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/multi_cell_list.py", line 259, in get_records_sorted
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi for i in heapq.merge(*results.values()):
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi File "/usr/lib64/python2.7/heapq.py", line 372, in merge
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi for itnum, it in enumerate(map(iter, iterables)):
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi TypeError: 'object' object is not iterable
2018-05-22 22:03:57.816 4170 ERROR nova.api.openstack.wsgi
2018-05-22 22:03:57.816 4170 INFO nova.api.openstack.wsgi [req-bb1698d5-a798-4f0e-82f1-396110f52d62 f5e45135156e4249a1e3164e6168f592 72a46594e674471184ff7e6ab199c775 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>
2018-05-22 22:03:57.817 4170 INFO nova.osapi_compute.wsgi.server [req-bb1698d5-a798-4f0e-82f1-396110f52d62 f5e45135156e4249a1e3164e6168f592 72a46594e674471184ff7e6ab199c775 - default default] 192.168.2.246 "GET /v2.1/servers/detail HTTP/1.1" status: 500 len: 626 time: 180.5081770

[jichenjc (jichenjc)]

 Timed out waiting for response from cell 00000000-0000-0000-0000-000000000000

This might be the root cause

as far as I can tell all 0 cell is cell 0 ?

There's a FIXME in the code so I guess it's a known issue.....

        # FIXME(danms): If we raise or timeout on a cell we need to handle
        # that here gracefully. The below routine will provide sentinels
        # to indicate that, which will crash the merge below, but we don't
        # handle this anywhere yet anyway.
        if self.cells:
            results = context.scatter_gather_cells(ctx, self.cells, 60,
                                                   do_query)
        else:
            results = context.scatter_gather_all_cells(ctx, do_query)