Hi everyone,
At work we've been trying to figure out how to get Ceilometer on the compute node to authenticate agasint Keystone but to no avail.
Here's the output of the keystone-wsgi-public log (controller node)
2016-04-29 18:54:19.074 26606 INFO keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] POST http://controller:5000/v2.0/tokens
2016-04-29 18:54:19.081 26606 WARNING keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] Authorization failed. The request you have made requires authentication. from xx.xx.xx.xx
2016-04-29 18:54:19.090 26604 DEBUG keystone.middleware.auth [req-08ae6a8a-4584-4033-9bd0-904d20cb6f1f - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71
And here's the output of the ceilometer-compute.log (compute node)
2016-04-29 18:35:26.708 24510 DEBUG ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Polling config file: /etc/ceilometer/pipeline.yaml _setup_polling_manager /usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:804
2016-04-29 18:35:26.723 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Pipeline config: {'sources': [{'interval': 600, 'meters': ['*'], 'name': 'meter_source', 'sinks': ['meter_sink']}, {'interval': 600, 'meters': ['cpu'], 'name': 'cpu_source', 'sinks': ['cpu_sink', 'cpu_delta_sink']}, {'interval': 600, 'meters': ['disk.read.bytes', 'disk.read.requests', 'disk.write.bytes', 'disk.write.requests', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests'], 'name': 'disk_source', 'sinks': ['disk_sink']}, {'interval': 600, 'meters': ['network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets'], 'name': 'network_source', 'sinks': ['network_sink']}], 'sinks': [{'publishers': ['notifier://'], 'transformers': None, 'name': 'meter_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'delta', 'parameters': {'target': {'name': 'cpu.delta'}, 'growth_only': True}}], 'name': 'cpu_delta_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': '(disk\\.device|disk)\\.(read|write)\\.(bytes|requests)', 'unit': '(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name': '\\1.\\2.\\3.rate', 'unit': '\\1/s'}}}}], 'name': 'disk_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': 'network\\.(incoming|outgoing)\\.(bytes|packets)', 'unit': '(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name': 'network.\\1.\\2.rate', 'unit': '\\1/s'}}}}], 'name': 'network_sink'}]}
2016-04-29 18:35:26.724 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] detected decoupled pipeline config format
2016-04-29 18:35:26.728 24510 DEBUG keystoneauth.identity.v2 [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Making authentication request to http://controller:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py:63
2016-04-29 18:35:29.135 24510 DEBUG keystoneauth.session [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Request returned failure status: 401 request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:466
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client search_opts=search_opts))
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client "servers")
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client resp, body = self.api.client.get(url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client return self.request(url, 'GET', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 89, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client return self.session.request(url, method, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client auth_headers = self.get_auth_headers(auth)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client return auth.get_headers(self, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client token = self.get_token(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client return self.get_access(session).auth_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client self.auth_ref = self.get_auth_ref(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client authenticated=False, log=False)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client return self.request(url, 'POST', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client raise exceptions.from_response(resp, method, url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client
2016-04-29 18:35:29.143 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.read.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.packets, no resources found this cycle
2016-04-29 18:35:29.145 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.iops, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.incoming.packets, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.requests.rate, no resources found this cycle
This is the keystone auth config on ceilometer.conf
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer
password = CEILOMETER_PASS
And the service credentials
os_auth_url = http://controller:5000/v2.0
os_username = ceilometer
os_tenant_name = service
os_password = CEILOMETER_PASS
interface = internalURL
region_name = RegionOne
BTW this is for the Mitaka release. I'm following the installation guide for Ubuntu.
docs.openstack.org/mitaka/install-guide-ubuntu
I was able to resolve my issue by migrating the ceilometer auth to keystone to v3. This is how the ceilometer service_credentials section looks now
[service_credentials]
auth_url = http://controller:5000/v3
username = ceilometer
tenant_name = service
password = xxxxxx
interface = internalURL
region_name = RegionOne
project_name = service
project_domain_id = xxxxxxxx
user_domain_id = xxxxxxxx
auth_type = password
With this configuration I was able to authenticate to keystone successfully.
Regards
-----------------------------------
Release: 0.1 on 2016-05-10 07:10
SHA: e2b52e0d3070a2accca3634052473b385aaea69f
Source: http://git.openstack.org/cgit/openstack/openstack-manuals/tree/doc/install-guide/source/index.rst
URL: http://docs.openstack.org/mitaka/install-guide-ubuntu/
All services should use the Identity v3 API, so we need to fix this issue. I suspect it primarily involves removal of the default domain late in the Mitaka cycle and change from 'project_domain_id = default' to 'project_