Heat autoscaling with gnocchi based aodh alarms requires use of naive instance_discovery_method setting with ceilometer compute agents?

[Gary Mackenzie]

OpenStack Release: Ocata
Gnocchi Version: 3.1.9
Ubuntu Release: Xenial
Deployment method: Charms

Using default settings, using heat autoscaling with gnocchi based alarms in aodh, the metadata for server group does not appear in libvirt-metadata, so 'gnocchi measures aggregation --resource-type instance --query server_group="<<group-uuid>>" --aggregation mean -m cpu_util' returns no output and alarms stay in 'insufficient data' state.

Changing 'instance_discovery_method = libvirt_metadata' to 'instance_discovery_method = naive' resolves and alarms work, however this is against upstream recommendations per https://docs.openstack.org/releasenotes/ceilometer/ocata.html

[James Page (james-page)]

See Ocata release notes for Ceilometer:

  https://docs.openstack.org/releasenotes/ceilometer/ocata.html

"If you are using Gnocchi as backend it’s strongly recommended to switch [compute]/instance_discovery_method to libvirt_metadata. This will reduce the load on the Nova API especially if you have many compute nodes."

However if you use the sample heat autoscaling template from:

  https://github.com/openstack/heat-templates/blob/master/hot/autoscaling.yaml

which includes the use of aodh with gnocchi based alarms, if ceilometer compute agents are configured with the default [compute]/instance_discovery_method @ Ocata, metadata is not retrieved by the agent and as a result the attribute is never set on the instance resource records in gnocchi.

Switching to naive resolves this issue, however it obviously generates load on the Nova API for metadata retrieval.

[James Page (james-page)]

We'll need to make a deployment tool based decision in the charm as to which setting to use; use of naive will restore functionality from Ocata onwards for anyone using Heat/Aodh/Gnocchi to drive autoscaling.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/545289

[gordon chung (chungg)]

the server_group attribute is captured in ocata even if discovery method is libvirt_metadata... it gathers this data via notifications from nova. basically, the source of this data is not the polling agent but nova notifications

[gordon chung (chungg)]

for example, our integration gate tests this scenario specifically: https://github.com/openstack/ceilometer/blob/stable/ocata/ceilometer/tests/integration/gabbi/gabbits-live/create_stack.json#L64

[James Page (james-page)]

I wonder whether we're missing something in charm deployments in the nova configuration then.

[James Page (james-page)]

@gordon

Is this configure for the nova-compute daemon using:

[DEFAULT]
...
instance_usage_audit = True
instance_usage_audit_period = hour
notify_on_state_change = vm_and_task_state

[oslo_messaging_notifications]
...
driver = messagingv2

?

[gordon chung (chungg)]

yes, those should be added to each compute node.

[James Page (james-page)]

that was my guess - in-fact we already do that (the ceilometer-agent charm passes this to the nova-compute charm to write into its configuration file).

'''
instance_usage_audit = True
instance_usage_audit_period = hour
notify_on_state_change = vm_and_task_state
'''

[OpenStack Infra (hudson-openstack)]

Change abandoned by James Page (<email address hidden>) on branch: master
Review: https://review.openstack.org/545289
Reason: Change not required - ceilometer collects this data from nova notifications which are correctly configured by charm deployments - autoscaling issues where caused by mismatch in granularity between ceilometer and gnocchi polling/measures.

[James Page (james-page)]

Issue was caused by bug 1750531 (mismatch between ceilometer and gnocchi granularity on polling and measures).