Cinder not configured to provide telemetry to ceilometer

[Frode Nordahl]

Metrics are currently not collected for Cinder volume when using Ceilometer in a deployment.

Cinder needs a few additions to its configuration file and the 'cinder-volume-usage-audit' program needs to be run periodically on cinder units.

I would suggest that this either is added to the cinder-charm and activated with a relation to ceilometer or by creating a new subordinate charm like the ceilometer-agent subordinate used for nova-compute.

[Edward Hope-Morley (hopem)]

If i'm not mistaken the following simple change to cinder.conf does the trick:

[oslo_messaging_notifications]
driver = messagingv2

As for cinder-volume-usage-audit would it be ill-advised to just always run it? It seems like a fairly low-touch action since it runs every 5 minutes. If so we could just add this to the cinder charm natively and not have to worry about extra relations, config and subordinates.

[Frode Nordahl (fnordahl)]

Sounds about right. Good input on a more simplistic way to solve it.

I have these concerns for such an approach:
a) Notifications sent by cinder-volume-usage-audit will never be consumed and might at some point create trouble on the end users rabbitmq-cluster if they do not have ceilometer running.
b) The audit gathering process might put an extra toll on the cinder nodes and become a unnecessary resource drain for end users not using ceilometer.

For a):
If ceilometer is designed in such a way that the consuming agent creates the queue and not the producer, I guess the messages will not be sent/stored unless the central ceilometer agent is present. We will probably have to check for problematic and unnecessary error logging on the cinder nodes though?

For b):
I guess we will have to evaluate and measure what amount of resources this process hogs in a sizable cluster.

[Edward Hope-Morley (hopem)]

I just ried a test. The cinder charm is already configured to provide notifications ("[oslo_messaging_notifications] driver = messagingv2") but I am not seeing all the expected meters as per http://docs.openstack.org/admin-guide/telemetry-measurements.html#openstack-block-storage:

ubuntu@xenial$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| d154a1e8-c903-45d1-867c-8c82dc5f9c2d | available | - | 1 | - | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
ubuntu@xenial$ ceilometer meter-list| grep volume
| volume.size | gauge | GB | d154a1e8-c903-45d1-867c-8c82dc5f9c2d | 0a91d38c85c9437caa92b8ac10e1eebe | 71aaf5f27079464fa6f82e23b0360a4a |
ubuntu@xenial$ ceilometer sample-list| grep volume
| 09458900-e3d9-11e6-991e-fa163e150a2a | d154a1e8-c903-45d1-867c-8c82dc5f9c2d | volume.size | gauge | 1.0 | GB | 2017-01-26T15:06:39.201000 |
| 08656f00-e3d9-11e6-991e-fa163e150a2a | d154a1e8-c903-45d1-867c-8c82dc5f9c2d | volume.size | gauge | 1.0 | GB | 2017-01-26T15:06:37.723000 |
ubuntu@xenial$ nova volume-attach bb5a6774-8620-4a7d-aa85-e590c6dfa6b0 d154a1e8-c903-45d1-867c-8c82dc5f9c2d
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | d154a1e8-c903-45d1-867c-8c82dc5f9c2d |
| serverId | bb5a6774-8620-4a7d-aa85-e590c6dfa6b0 |
| volumeId | d154a1e8-c903-45d1-867c-8c82dc5f9c2d |
+----------+--------------------------------------+
ubuntu@xenial$ ceilometer meter-list| grep volume
| volume.size | gauge | GB | d154a1e8-c903-45d1-867c-8c82dc5f9c2d | 0a91d38c85c9437caa92b8ac10e1eebe |
juju ssh cinder/0 sudo cinder-volume-usage-audit
Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
2017-01-26 15:25:24.008 13018 WARNING oslo_config.cfg [req-21c4fbd0-ee80-4b19-8ea1-a978c354cb6e - - - - -] Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py:241: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
Connection to 2001:db8:0:1:f816:3eff:fef0:165c closed.
ubuntu@xenial$ ceilometer meter-list| grep volume
| volume.size | gauge | GB | d154a1e8-c903-45d1-867c-8c82dc5f9c2d | 0a91d38c85c9437caa92b8ac10e1eebe |

[Edward Hope-Morley (hopem)]

I enabled debug logging for the volume audit and it is apparently not detecting any volumes even though i do have one:

2017-01-26 16:23:13.212 19385 DEBUG cinder [req-e4437c8d-3d19-4bb7-aab7-7903ee61bdfd - - - - -] Found 0 volumes main /usr/lib/python2.7/dist-packages/cinder/cmd/volume_usage_audit.py:106
2017-01-26 16:23:13.246 19385 DEBUG cinder [req-e4437c8d-3d19-4bb7-aab7-7903ee61bdfd - - - - -] Found 0 snapshots main /usr/lib/python2.7/dist-packages/cinder/cmd/volume_usage_audit.py:177

[Edward Hope-Morley (hopem)]

My bad, the default audit window is too short. This works:

root@juju-853dc7-default-4:~# cinder-volume-usage-audit --end_time "2017-02-01 00:00:00" 2>&1| grep Found
2017-01-26 21:25:23.582 18440 DEBUG cinder [req-90f87894-ba96-48d6-be43-f1c9ff8e1c8b - - - - -] Found 1 volumes main /usr/lib/python2.7/dist-packages/cinder/cmd/volume_usage_audit.py:106
2017-01-26 21:25:23.888 18440 DEBUG cinder [req-90f87894-ba96-48d6-be43-f1c9ff8e1c8b - - - - -] Found 0 snapshots main /usr/lib/python2.7/dist-packages/cinder/cmd/volume_usage_audit.py:177

Still only have volume.size meter though so something still not quite right.

[Edward Hope-Morley (hopem)]

Ok something is definitely out of kilter here. While the openstack docs say that there are numerous meters defined for volume, the trunk meters.yaml file only defines one:

https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml#L33

Which presumably explains why no matter what events i trigger i only get volume.size as a meter.

[Edward Hope-Morley (hopem)]

ftr disk usage meters seems to be working fine - http://pastebin.ubuntu.com/23874037/

[Edward Hope-Morley (hopem)]

I completely missed the fact that a cron entry for running cinder-volume-usage-audit was added to the 16.10 charms (see LP: #1623144). This allows the user to adjust the size of the volume audit window via charm config but does not, however, provide any configurability for the cron frequency.

[Edward Hope-Morley (hopem)]

I am going to close this bug since as far as I can tell the cinder charm is adequately configuring cinder to provide metrics to ceilometer. Please re-open if you disagree.