openstack image list throwing 500 error

[Madhu]

I can list service catalog without issues
openstack catalog list
+----------+----------+-----------------------------------------+
| Name | Type | Endpoints |
+----------+----------+-----------------------------------------+
| keystone | identity | RegionOne |
| | | public: http://controller1:5000/v3/ |
| | | RegionOne |
| | | admin: http://controller1:35357/v3/ |
| | | RegionOne |
| | | internal: http://controller1:5000/v3/ |
| | | |
| glance | image | RegionOne |
| | | admin: http://controller1:9292 |
| | | RegionOne |
| | | public: http://controller1:9292 |
| | | RegionOne |
| | | internal: http://controller1:9292 |
| | | |
+----------+----------+-----------------------------------------+

but when I use openstack image list it is throwing 500 error, following is from glance-api log.

Installation is on Ubuntu17.04 + octa release

2017-09-07 17:17:41.096 19450 DEBUG eventlet.wsgi.server [-] (19450) accepted ('192.168.1.45', 43908) server /usr/lib/python2.7/dist-packages/eventlet/wsgi.py:868
2017-09-07 17:17:41.099 19450 DEBUG glance.api.middleware.version_negotiation [-] Determining version of request: GET /v2/images Accept: */* process_request /usr/local/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:46
2017-09-07 17:17:41.099 19450 DEBUG glance.api.middleware.version_negotiation [-] Using url versioning process_request /usr/local/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:58
2017-09-07 17:17:41.100 19450 DEBUG glance.api.middleware.version_negotiation [-] Matched version: v2 process_request /usr/local/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:70
2017-09-07 17:17:41.100 19450 DEBUG glance.api.middleware.version_negotiation [-] new path /v2/images process_request /usr/local/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:71
2017-09-07 17:17:41.244 19450 INFO eventlet.wsgi.server [-] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/glance/common/wsgi.py", line 804, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 108, in __call__
    return request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
    response = self.process_request(req)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 623, in process_request
    resp = super(AuthProtocol, self).process_request(request)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
    allow_expired=allow_expired)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 435, in _do_fetch_token
    data = self.fetch_token(token, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 762, in fetch_token
    allow_expired=allow_expired)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/_identity.py", line 217, in verify_token
    auth_ref = self._request_strategy.verify_token(
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/_identity.py", line 168, in _request_strategy
    strategy_class = self._get_strategy_class()
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/_identity.py", line 190, in _get_strategy_class
    if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 176, in get_endpoint
    return self.session.get_endpoint(auth or self.auth, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 856, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 216, in get_endpoint
    service_name=service_name)
  File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 190, in url_for
    raise exceptions.EmptyCatalog('The service catalog is empty.')
EmptyCatalog: The service catalog is empty.

2017-09-07 17:17:41.245 19450 INFO eventlet.wsgi.server [-] 192.168.1.45 - - [07/Sep/2017 17:17:41] "GET /v2/images HTTP/1.1" 500 139 0.146946

[Madhu (mssunkara)]

output of following command with debug option

openstack image list --debug

Making authentication request to http://controller1:35357/v3/auth/tokens
"POST /v3/auth/tokens HTTP/1.1" 201 1645
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "e09e9f01a2924283856b59e82f3296fb", "name": "admin"}], "expires_at": "2017-09-08T02:38:45.000000Z", "pro\
ject": {"domain": {"id": "default", "name": "Default"}, "id": "173d0a72eebd44a79633f7d8a8a7b302", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://controller1:5000/v\
3/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "20b4007c32834892b82596bc26c943c1"}, {"url": "http://controller1:35357/v3/", "interface": "adm\
in", "region": "RegionOne", "region_id": "RegionOne", "id": "3201afe42cf247eeaca8c205c87a2ffc"}, {"url": "http://controller1:5000/v3/", "interface": "internal", "region": "Regio\
nOne", "region_id": "RegionOne", "id": "e2c3ad4225c04ded9014ceeca17034f1"}], "type": "identity", "id": "ad82e0a0be544800b041a433a1aa52b5", "name": "keystone"}, {"endpoints": [{"\
url": "http://controller1:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "588d09bb63db4344ab2efcfe85b087fa"}, {"url": "http://controller1:92\
92", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "9ea27634d05d4d17874434e5d8732ba2"}, {"url": "http://controller1:9292", "interface": "internal\
", "region": "RegionOne", "region_id": "RegionOne", "id": "d5c5aca5e09f4420ba1cec4529a06a8b"}], "type": "image", "id": "b0aba851dbc14efbbd183cee8f52908d", "name": "glance"}], "u\
ser": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "d24cae7c524b42fcb9fa11a41326e0a1"}, "audit_ids": ["z8mA0VytSve8bvE2JT\
eQnA"], "issued_at": "2017-09-08T01:38:45.000000Z"}}
Instantiating image api: <class 'openstackclient.api.image_v2.APIv2'>
REQ: curl -g -i -X GET "http://controller1:9292/v2/images?marker=None" -H "User-Agent: osc-lib/1.3.0 keystoneauth1/2.18.0 python-requests/2.10.0 CPython/2.7.13" -H "X-Auth-Token\
: {SHA1}2ee688187b65bad14acb8e27e9a63c25434cce52"
Starting new HTTP connection (1): controller1
"GET /v2/images HTTP/1.1" 500 0
RESP: [500] Content-Type: text/plain Content-Length: 0 Date: Fri, 08 Sep 2017 01:38:46 GMT Connection: close
RESP BODY: Omitted, Content-Type is set to text/plain. Only application/json responses have their bodies logged.

Request returned failure status: 500

[Lance Bragstad (lbragstad)]

From comment #1, it looks like there is at least a service catalog in the debug logging. Can you share the configuration you have for keystoneauth in glance?

[Madhu (mssunkara)]

glance-api.conf

[keystone_authtoken]
auth_uri = http://controller1:5000
auth_url = http://controller1:35357
memcached_servers = controller1:11211
auth_type = password
project_domain_name = default
user_domain_name = default
projecct_name = service
username = glance
password = <_password_>

from glance-registry.conf

[keystone_authtoken]

auth_uri = http://controller1:5000
auth_url = http://controller1:35357
memcached_servers = controller1:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = glance
password = <_password_>

[Lance Bragstad (lbragstad)]

Are you able to get a token as a regular user and does it have a service catalog in it? Are you able to do the same with the glance service user? Keystone populated a service catalog in the debug logging of osc, so I don't think this is an issue with keystone server. More likely a configuration issue of some sort.

[Lance Bragstad (lbragstad)]

Also, is:

 projecct_name = service

a typo or was that copy/pasted directly from glance-api.conf?

[Madhu (mssunkara)]

project_name = service is correct, I just followed ubuntu install doc from openstack,

I am able to get token as admin, demo and I am running image list as admin

as admin:

"POST /v3/auth/tokens HTTP/1.1" 201 1645
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "e09e9f01a2924283856b59e82f3296fb", "name": "admin"}], "expires_at": "2017-09-08T16:27:13.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "173d0a72eebd44a79633f7d8a8a7b302", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://controller1:5000/v3/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "20b4007c32834892b82596bc26c943c1"}, {"url": "http://controller1:35357/v3/", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "3201afe42cf247eeaca8c205c87a2ffc"}, {"url": "http://controller1:5000/v3/", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "e2c3ad4225c04ded9014ceeca17034f1"}], "type": "identity", "id": "ad82e0a0be544800b041a433a1aa52b5", "name": "keystone"}, {"endpoints": [{"url": "http://controller1:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "588d09bb63db4344ab2efcfe85b087fa"}, {"url": "http://controller1:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "9ea27634d05d4d17874434e5d8732ba2"}, {"url": "http://controller1:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "d5c5aca5e09f4420ba1cec4529a06a8b"}], "type": "image", "id": "b0aba851dbc14efbbd183cee8f52908d", "name": "glance"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "d24cae7c524b42fcb9fa11a41326e0a1"}, "audit_ids": ["evSeR1I7THq7AhLG1CqS2g"], "issued_at": "2017-09-08T15:27:13.000000Z"}}
run(Namespace(columns=[], formatter='table', max_width=0, noindent=False, prefix='', print_empty=False, variables=[]))
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires | 2017-09-08T16:27:13+0000 |
| id | gAAAAABZsrbRtgHljHTkDLGX9yY6Q-BIqSc1b5uEPuNYNlVgdZ4a5bNbvZo37bb9igWT9uY8SzXGu0d15pAQXTNwPHP7bfve3dngN32gRw8jKo- |
| | 7IpRpSyr8nGhtUbZmAIGVB8bPBUGIg15V8fErEJob3S9xgmhreeyqag0RhMMBPJX3WyQD6cY |
| project_id | 173d0a72eebd44a79633f7d8a8a7b302 ...

[Madhu (mssunkara)]

as glance user

openstack --os-auth-url http://controller1:35357/v3 --os-project-domain-name default --os-user-domain-name default --os-project-name service --os-username glance token issue --debug

Making authentication request to http://controller1:35357/v3/auth/tokens
"POST /v3/auth/tokens HTTP/1.1" 201 1648
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "e09e9f01a2924283856b59e82f3296fb", "name": "admin"}], "expires_at": "2017-09-08T17:03:54.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "fe5c898642384e9bac1df200944770b2", "name": "service"}, "catalog": [{"endpoints": [{"url": "http://controller1:5000/v3/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "20b4007c32834892b82596bc26c943c1"}, {"url": "http://controller1:35357/v3/", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "3201afe42cf247eeaca8c205c87a2ffc"}, {"url": "http://controller1:5000/v3/", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "e2c3ad4225c04ded9014ceeca17034f1"}], "type": "identity", "id": "ad82e0a0be544800b041a433a1aa52b5", "name": "keystone"}, {"endpoints": [{"url": "http://controller1:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "588d09bb63db4344ab2efcfe85b087fa"}, {"url": "http://controller1:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "9ea27634d05d4d17874434e5d8732ba2"}, {"url": "http://controller1:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "d5c5aca5e09f4420ba1cec4529a06a8b"}], "type": "image", "id": "b0aba851dbc14efbbd183cee8f52908d", "name": "glance"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "glance", "id": "1ce03ce5e5ef47f988456c5fb9dd9027"}, "audit_ids": ["FtT5LK6sT5G2g56_QqFT_Q"], "issued_at": "2017-09-08T16:03:54.000000Z"}}
run(Namespace(columns=[], formatter='table', max_width=0, noindent=False, prefix='', print_empty=False, variables=[]))
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires | 2017-09-08T17:03:54+0000 |
| id | gAAAAABZsr9q_79c7UV57GCWgoM3Lght5oE77A4lnpySEJorrGby2wFW2QWlbCdTZY5gXrnBUzpif8Tw-E_s3cZdztwB8B2GSTmWotsZ9xtT0TbUXzZGb61y6nnX7cR4RoC- |
| | AwIEhXo4JsS8OWrQwDfgthSWAR0E0AzIdgms150CR2hIpQrzUJE |
| project_id | fe5c898642384e9bac1df20094...

[Lance Bragstad (lbragstad)]

It doesn't look like keystone is the problem here since catalogs are being issued to multiple different users.

The reason why I asked about project_name in the keystoneauth config is because projecct_name (from comment #3) would be ignored by oslo when parsing configuration options. It should be project_name.

[Morgan Fainberg (mdrnstm)]

With no project name, you'd get an unscoped token, which (IIRC) would be without a catalog since catalog is somewhat scope specific (historical). This lines up with what lance said in #8

[Madhu (mssunkara)]

It is indeed a typo, couldn't realize there is extra 'c' even when you asked.
Thanks, All working now after changing.

[Madhu (mssunkara)]

Thanks for your time again