glance image-list is fine, but nova image-list with CEPH backend fails

[Sola]

Hi, all

I've configured HA Openstack Liberty with two controllers and compute/storage nodes with CEPH backend. OS installed everywhere is CentOS 7.1

Everything is fine, except one thing - this is output from the nova image-list command.

Seems that I have the same issue as described here
https://ask.openstack.org/en/question/86111/glance-image-list-is-fine-but-nova-image-list-error/ and here
https://ask.openstack.org/en/question/88386/nova-cannot-get-ceph-image-in-liberty/

    [root@vm-controller1 nova]# openstack image list
    +--------------------------------------+--------+
    | ID | Name |
    +--------------------------------------+--------+
    | 365a3e2b-7396-4c6d-a906-e59146ece2d4 | cirros |
    +--------------------------------------+--------+
    [root@vm-controller1 nova]# glance image-list
    +--------------------------------------+--------+
    | ID | Name |
    +--------------------------------------+--------+
    | 365a3e2b-7396-4c6d-a906-e59146ece2d4 | cirros |
    +--------------------------------------+--------+
    [root@vm-controller1 nova]# nova image-list
    ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
    <class 'glanceclient.exc.HTTPInternalServerError'> (HTTP 500) (Request-ID: req-f0c22157-0d40-4ee7-b0be-4d9bca51f143)

Tried following:

1) On the Controller nodes:
vi /etc/nova/nova.conf

[glance]
api_servers = http://10.7.0.139:9292

2) On the Controller nodes:
vi /etc/glance/glance-api.conf

[DEFAULT]
registry_host = 10.7.0.139

3) On the Controller nodes:
vi /etc/glance/glance-cache.conf

[DEFAULT]
registry_host = 10.7.0.139

4) Double checked all passwords,

5) Changed keystone API from V2 to V3 as described here - http://www.cloudkb.net/how-to-change-keystone-api-v2-v3/

Nothing helped and error still persists.
Output from the nova-api.log:

    2016-02-25 17:48:53.265 6274 INFO nova.osapi_compute.wsgi.server [req-f4b90591-3408-40c1-8c3a-8f6ea29f530e b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] 10.7.0.135 "GET /v2/ HTTP/1.1" status: 200 len: 583 time: 1.1227880
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions [req-f0c22157-0d40-4ee7-b0be-4d9bca51f143 b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] Unexpected exception in API method
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions Traceback (most recent call last):
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/images.py", line 145, in detail
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions **page_params)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 68, in get_all
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions return session.detail(context, **kwargs)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 284, in detail
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions for image in images:
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 254, in list
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions for image in paginate(params, return_request_id):
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 238, in paginate
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions images, resp = self._list(url, "images")
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 63, in _list
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions resp, body = self.client.get(url)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 280, in get
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions return self._request('GET', url, **kwargs)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 272, in _request
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions resp, body_iter = self._handle_response(resp)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 93, in _handle_response
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions raise exc.from_response(resp, resp.content)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions
    2016-02-25 17:48:53.769 6274 INFO nova.api.openstack.wsgi [req-f0c22157-0d40-4ee7-b0be-4d9bca51f143 b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
    <class 'glanceclient.exc.HTTPInternalServerError'>
    2016-02-25 17:48:53.771 6274 INFO nova.osapi_compute.wsgi.server [req-f0c22157-0d40-4ee7-b0be-4d9bca51f143 b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] 10.7.0.135 "GET /v2/292b43fbdc014727a4ec3746f7a7239e/images/detail HTTP/1.1" status: 500 len: 445 time: 0.2422621
    ~
    ~

Please, help, I've run out of any ideas... :(((((((

[Sola (sola2015)]

# systemctl stop openstack-glance-api openstack-glance-registry

# glance image-list
Error finding address for http://10.7.0.139:9292/v2/images?limit=20&sort_key=name&sort_dir=asc: ('Connection aborted.', BadStatusLine("''",))

# nova image-list
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'glanceclient.exc.CommunicationError'> (HTTP 500) (Request-ID: req-17197970-9216-4cd9-8c1a-3a2839cb7c87)

[Sola (sola2015)]

Update:

If I comment out this setting:

vi /etc/glance/glance-registry.conf

[paste_deploy]
# flavor = keystone

Then everything works as expected:

openstack image list
    +--------------------------------------+--------+
    | ID | Name |
    +--------------------------------------+--------+
    | 365a3e2b-7396-4c6d-a906-e59146ece2d4 | cirros |
    +--------------------------------------+--------+
glance image-list
    +--------------------------------------+--------+
    | ID | Name |
    +--------------------------------------+--------+
    | 365a3e2b-7396-4c6d-a906-e59146ece2d4 | cirros |
    +--------------------------------------+--------+
nova image-list
    +--------------------------------------+--------+--------+--------+
    | ID | Name | Status | Server |
    +--------------------------------------+--------+--------+--------+
    | 365a3e2b-7396-4c6d-a906-e59146ece2d4 | cirros | ACTIVE | |
    +--------------------------------------+--------+--------+--------+

[Sean Dague (sdague)]

Please attach nova.conf, glance.conf. This really looks like just a keystone credentials issues.

[Sola (sola2015)]

Hi, Sean

Please, see files attached.
Don't know how to do a baunch attachment in one comment, sorry :)

[Sola (sola2015)]

Anybody, help!