v3 tenant 404

[Ross Krumbeck]

Issue is with:
CentOS Linux release 7.1.1503 (Core)
openstack-ceilometer-notification-2015.1.0-6.el7.noarch
openstack-ceilometer-alarm-2015.1.0-6.el7.noarch
openstack-ceilometer-common-2015.1.0-6.el7.noarch
openstack-ceilometer-collector-2015.1.0-6.el7.noarch
openstack-ceilometer-central-2015.1.0-6.el7.noarch
python-ceilometerclient-1.0.13-1.el7.noarch
python-ceilometer-2015.1.0-6.el7.noarch

Have configured ceilometer.conf to use Identity v2.

Can confirm that ceilometer auth is successful but there seems to be errors every ten minutes in the keystone admin access log:

10.1.1.1 - - [07/Aug/2015:11:52:09 +0000] "GET /v3/tenants HTTP/1.1" 404 93 "-" "python-keystoneclient"
10.1.1.1 - - [07/Aug/2015:11:52:09 +0000] "GET /v3/tenants HTTP/1.1" 404 93 "-" "python-keystoneclient"
10.1.1.1 - - [07/Aug/2015:11:52:09 +0000] "GET /v3/tenants HTTP/1.1" 404 93 "-" "python-keystoneclient"

This should be:

/v2.0/tenants

[gordon chung (chungg)]

are there errors in compute agent or central agent logs??

[Ross Krumbeck (rkrum)]

From ceilometer central log:

2015-08-12 13:20:13.080 22270 ERROR ceilometer.agent.base [-] Unable to discover resources: The resource could not be found. (HTTP 404) (Request-ID: req-2cb27819-f7ce-45e6-9392-b0619dca0ff5) (HTTP 404)
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base Traceback (most recent call last):
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/ceilometer/agent/base.py", line 346, in discover
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base discovered = discoverer.discover(self, param)
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/ceilometer/agent/discovery/tenant.py", line 34, in discover
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base tenants = manager.keystone.tenants.list()
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/keystoneclient/v2_0/tenants.py", line 123, in list
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base tenant_list = self._list('/tenants%s' % query, 'tenants')
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 113, in _list
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base resp, body = self.client.get(url, **kwargs)
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 170, in get
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base return self.request(url, 'GET', **kwargs)
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base return self.session.request(url, method, **kwargs)
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 397, in request
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base raise exceptions.from_response(resp, method, url)
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base NotFound: The resource could not be found. (HTTP 404) (Request-ID: req-2cb27819-f7ce-45e6-9392-b0619dca0ff5)
2015-08-12 13:20:13.080 22270 TRACE ceilometer.agent.base

[Ross Krumbeck (rkrum)]

I changed back the keystone endpoints back to v2.0 and it is doing the correct request again:

/v2.0/tenants

So seems to be only when keystone endpoints are set to v3.

[ZhiQiang Fan (aji-zqfan)]

caused by swift related pollsters, they require tenant discover

[ZhiQiang Fan (aji-zqfan)]

currently, we are using keystone v2 by default

if environment is set to v3, then there is no way for end user to specify to using v2 for ceilometer, finally keystoneclient will use the v3 endpoint, then there always will be 404

to solve it, we need keystone v3 support, this will be handled by the blueprints: https://blueprints.launchpad.net/ceilometer/+spec/support-keystone-v3, and we need to set it as default, since v2 is deprecated from kilo version,

a quick fix might be that we add the endpoint_override option, then we can always use v2 endpoint, as end user wishes

[ZhiQiang Fan (aji-zqfan)]

even though we can use endpoint_override to solve this issue, but after further thought, I think maybe we should fix it in python-keystoneclient upstream, because we're using keystoneclient.v2_0.client.Client but it just use v3 endpoint, which makes non-sense, keystoneclient.v2_0 should ensure itself is using v2 endpoint, because end user already explicitly use v2 sdk

@gordon chung
what's your opinion?

[Jesse Pretorius (jesse-pretorius)]

FYI: https://review.openstack.org/206333

[Shinobu Kinjo (shinobu-kj)]

@Jesse,

Does that patch set fix this bug?

[gordon chung (chungg)]

https://review.openstack.org/#/c/237537/