Distributed Cloud: VIM failed to get image list on subcloud

[Andy]

Brief Description
-----------------
On subclouds of a Distributed Cloud system, VIM failed while trying to get
image list from glance-api.

Severity
--------
Major

Steps to Reproduce
------------------
- Deploy a Distributed Cloud with as least one subcloud.
- Check nfc-vim.log, there VIM failed to get image list.

2018-09-10T03:11:11.786 controller-0 VIM_Thread[30415] ERROR Caught exception while trying to get image list, error=[OpenStack Rest-API Exception: method=GET, url=http://192.168.121.2:9292/v2/images?limit=32, headers=None, body=None, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Date', 'Mon, 10 Sep 2018 03:11:11 GMT'), ('Content-Length', '186'), ('Content-Type', 'application/json'), ('X-Openstack-Request-Id', 'req-80884392-4da4-4b0e-99d5-ea0679f2ee16')], response_body={"message": "The server has either erred or is incapable of performing the requested operation.<br /><br />\n\n\n", "code": "500 Internal Server Error", "title": "Internal Server Error"}].
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_image_api.py", line 114, in get_images
    future.result = (yield)
OpenStackRestAPIException: [OpenStack Rest-API Exception: method=GET, url=http://192.168.121.2:9292/v2/images?limit=32, headers=None, body=None, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Date', 'Mon, 10 Sep 2018 03:11:11 GMT'), ('Content-Length', '186'), ('Content-Type', 'application/json'), ('X-Openstack-Request-Id', 'req-80884392-4da4-4b0e-99d5-ea0679f2ee16')], response_body={"message": "The server has either erred or is incapable of performing the requested operation.<br /><br />\n\n\n", "code": "500 Internal Server Error", "title": "Internal Server Error"}]

Expected Behavior
------------------
VIM get image list successfully.

Actual Behavior
----------------
VIM get image failed with Exception.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Observed on one node and two nodes subcloud.

Branch/Pull Time/Commit
-----------------------
master as of 2018-09-05_20-18-00

Timestamp/Logs
--------------

nfv-vim.log:
===========
2018-09-07T02:04:04.718 controller-0 VIM_Thread[12451] ERROR Caught exception while trying to get image list, error=[OpenStack Rest-API Exception: method=GET, url=http://192.168.121.2:9292/v2/images?limit=32, headers=None, body=None, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Date', 'Fri, 07 Sep 2018 02:04:04 GMT'), ('Content-Length', '186'), ('Content-Type', 'application/json'), ('X-Openstack-Request-Id', 'req-57ca82de-9de4-4cf9-8891-5f0fc16ba226')], response_body={"message": "The server has either erred or is incapable of performing the requested operation.<br /><br />\n\n\n", "code": "500 Internal Server Error", "title": "Internal Server Error"}].
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_image_api.py", line 114, in get_images
    future.result = (yield)
OpenStackRestAPIException: [OpenStack Rest-API Exception: method=GET, url=http://192.168.121.2:9292/v2/images?limit=32, headers=None, body=None, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Date', 'Fri, 07 Sep 2018 02:04:04 GMT'), ('Content-Length', '186'), ('Content-Type', 'application/json'), ('X-Openstack-Request-Id', 'req-57ca82de-9de4-4cf9-8891-5f0fc16ba226')], response_body={"message": "The server has either erred or is incapable of performing the requested operation.<br /><br />\n\n\n", "code": "500 Internal Server Error", "title": "Internal Server Error"}]
2018-09-07T02:04:04.718 controller-0 VIM_Thread[12451] ERROR _vim_nfvi_audits.py.550 Audit-Images callback, not completed, responses={'completed': False, 'reason': '', 'page-request-id': '57804519-0bd8-45e8-8e4e-11b96ac8a8e1'}.
2018-09-07T02:04:05.727 controller-0 VIM_Thread[12451] INFO nfvi_block_storage_api.py.118 Cinder service get-volumes not available.
2018-09-07T02:04:07.741 controller-0 VIM_Thread[12451] INFO nfvi_block_storage_api.py.501 Cinder service get-volume-snapshots not available.
2018-09-07T02:04:30.009 controller-0 VIM_Thread[12451] INFO _instance_director.py.1744 No hypervisors available, can't recover instances.
2018-09-07T02:04:54.259 controller-0 VIM_Thread[12451] DEBUG _host_tasks.py.473 Task (audit-disabled-host_controller-0) complete.
2018-09-07T02:05:00.441 controller-0 VIM_Thread[12451] INFO _instance_director.py.1744 No hypervisors available, can't recover instances.
2018-09-07T02:05:04.892 controller-0 VIM_Thread[12451] INFO _vim_nfvi_audits.py.858 Audit instances called, timer_id=14.

glance-api.log:
==============
2018-09-10 08:52:34.615 28214 INFO eventlet.wsgi.server [req-3a4cf22c-1077-4769-9646-aeac7073738a 7065939c45604f9c80431e824015afcf 1bd00c5984cc418faed011e95c5d7db1 - default default] 192.168.121.3 - - [10/Sep/2018 08:52:34] "GET /v2/images?limit=20 HTTP/1.1" 500 454 0.661956
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi [req-b24b5c7f-4320-4173-9d0e-3b0e968b6d58 b2b5595a87b04d9890c4775577dd5597 b98b7b7312d04665af82c3a3d50a06b0 - default default] Caught error: string indices must be integers, not str: TypeError: string indices must be integers, not str
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi Traceback (most recent call last):
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1269, in __call__
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi request, **action_args)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1308, in dispatch
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi return method(*args, **kwargs)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/v2/images.py", line 214, in index
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi member_status=member_status)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/authorization.py", line 111, in list
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi images = self.image_repo.list(*args, **kwargs)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 89, in list
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi items = self.base.list(*args, **kwargs)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 114, in list
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi return super(ImageRepoProxy, self).list(*args, **kwargs)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 89, in list
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi items = self.base.list(*args, **kwargs)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 89, in list
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi items = self.base.list(*args, **kwargs)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 89, in list
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi items = self.base.list(*args, **kwargs)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/db/__init__.py", line 101, in list
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi member_status=member_status, return_tag=True)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/db/registry/api.py", line 52, in wrapper
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi return func(client, *args, **kwargs)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/db/registry/api.py", line 131, in image_get_all
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi v1_mode=v1_mode)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/rpc.py", line 300, in method_proxy
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi return self.do_request(item, **kw)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/rpc.py", line 266, in do_request
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi 'kwargs': kwargs}])
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 71, in wrapped
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi return func(self, *args, **kwargs)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/rpc.py", line 252, in bulk_request
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi body)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 71, in wrapped
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi return func(self, *args, **kwargs)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 366, in do_request
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi self._authenticate()
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 344, in _authenticate
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi auth_plugin.authenticate()
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/auth.py", line 132, in authenticate
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi _authenticate(auth_url)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/auth.py", line 122, in _authenticate
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi self._v3_auth(token_url)
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/auth.py", line 260, in _v3_auth
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi endpoint = get_endpoint(resp_auth['serviceCatalog'],
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi TypeError: string indices must be integers, not str
2018-09-10 08:53:19.080 28214 ERROR glance.common.wsgi
2018-09-10 08:53:19.083 28214 INFO eventlet.wsgi.server [req-b24b5c7f-4320-4173-9d0e-3b0e968b6d58 b2b5595a87b04d9890c4775577dd5597 b98b7b7312d04665af82c3a3d50a06b0 - default default] 192.168.121.3 - - [10/Sep/2018 08:53:19] "GET /v2/images?limit=32 HTTP/1.1" 500 399 0.648814

[Ghada Khalil (gkhalil)]

Targeting stx.2018.10 as this affects glance functionality on distributed cloud

[Andy (andy.wrs)]

Change has been merged from the following pull request:

https://github.com/starlingx-staging/stx-glance/pull/1/commits/05c7fa59c8c9d922c28b4997970cb913e038b494