Heat incorrectly creates alarm metadata when the meter parameter is not in NOVA_METERS

Bug #1589415 reported by Peter Razumovsky
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Low
Peter Razumovsky

Bug Description

I try to create a new stack with only one resource which is a Ceilometer alarm. If a "meter" parameter is defined in a ceilometer, alarm is created with a correct query string: "metadata.user_metadata.stack == <stack_id>". There are situations when I need to create an alarm for a "meter", defined in pipeline.yaml. In this situation query string is changed to the "metadata.metering.metadata.user_metadata.stack == <stack_id>" value.

Steps to reproduce:
# cat alarm.yaml
heat_template_version: 2015-10-15

resources:

  mem_alarm_high:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-up if the average memory usage >= 80% for 1 minute
      meter_name: memory
      statistic: avg
      period: 60
      evaluation_periods: 1
      threshold: 80
      alarm_actions:
      matching_metadata: {"metadata.user_metadata.stack": {get_param: "OS::stack_id"}}
      comparison_operator: gt

# heat stack-create -f alarm.yaml test-05311438
# ceilometer alarm-show 06ca7544-0c80-4029-8880-b573944a2102
+---------------------------+----------------------------------------------------------------------+
| Property | Value |
+---------------------------+----------------------------------------------------------------------+
| alarm_actions | [] |
| alarm_id | 06ca7544-0c80-4029-8880-b573944a2102 |
| comparison_operator | gt |
| description | Scale-up if the average memory usage >= 80% for 1 minute |
| enabled | True |
| evaluation_periods | 1 |
| exclude_outliers | False |
| insufficient_data_actions | None |
| meter_name | memory |
| name | test-05311438-mem_alarm_high-nxaoa6jqvcqr |
| ok_actions | None |
| period | 60 |
| project_id | d400f2e065f84b58829cbc5b0dc30961 |
| query | metadata.user_metadata.stack == 34fa084a-a72b-43df-947e-32e82a6f2c55 |
| repeat_actions | True |
| severity | low |
| state | insufficient data |
| statistic | avg |
| threshold | 80.0 |
| type | threshold |
| user_id | f46b8fa2d32c42b6b1e9f6c2eaca145e |
+---------------------------+----------------------------------------------------------------------+

So, all works fine. Now use memory_util:

# cat alarm.yaml
heat_template_version: 2015-10-15

resources:

  mem_alarm_high:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-up if the average memory usage >= 80% for 1 minute
      meter_name: memory_util
      statistic: avg
      period: 60
      evaluation_periods: 1
      threshold: 80
      alarm_actions:
      matching_metadata: {"metadata.user_metadata.stack": {get_param: "OS::stack_id"}}
      comparison_operator: gt

# heat stack-create -f alarm.yaml test-05311439
# ceilometer alarm-show 397b1304-5d76-4f2c-b108-a1fe0b16b770
+---------------------------+------------------------------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------------------------------+
| alarm_actions | [] |
| alarm_id | 397b1304-5d76-4f2c-b108-a1fe0b16b770 |
| comparison_operator | gt |
| description | Scale-up if the average memory usage >= 80% for 1 minute |
| enabled | True |
| evaluation_periods | 1 |
| exclude_outliers | False |
| insufficient_data_actions | None |
| meter_name | memory_util |
| name | test-05311439-mem_alarm_high-cnwkc45vc27g |
| ok_actions | None |
| period | 60 |
| project_id | d400f2e065f84b58829cbc5b0dc30961 |
| query | metadata.metering.metadata.user_metadata.stack == 7decbaf8-c2ed- |
| | 42f0-9210-37e521704201 |
| repeat_actions | True |
| severity | low |
| state | insufficient data |
| statistic | avg |
| threshold | 80.0 |
| type | threshold |
| user_id | f46b8fa2d32c42b6b1e9f6c2eaca145e |
+---------------------------+------------------------------------------------------------------+

Changed in heat:
status: New → In Progress
Revision history for this message
Peter Razumovsky (prazumovsky) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/323256
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=f4486930e199da11ef132728c4f63085bb233da6
Submitter: Jenkins
Branch: master

commit f4486930e199da11ef132728c4f63085bb233da6
Author: Peter Razumovsky <email address hidden>
Date: Tue May 31 12:42:49 2016 +0300

    Fix building query key for ceilometer alarm

    Current building of query key doesn't check prefix
    correctly, so fix this issue.

    Change-Id: I002838d1dd57a48bd8e40f4c2c59f310e1c07987
    Closes-bug: #1589415

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/338195

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/heat 7.0.0.0b2

This issue was fixed in the openstack/heat 7.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/mitaka)

Reviewed: https://review.openstack.org/338195
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ea8d94ae6841acef999e45d22a770e41ee2ff0b2
Submitter: Jenkins
Branch: stable/mitaka

commit ea8d94ae6841acef999e45d22a770e41ee2ff0b2
Author: Peter Razumovsky <email address hidden>
Date: Tue May 31 12:42:49 2016 +0300

    Fix building query key for ceilometer alarm

    Current building of query key doesn't check prefix
    correctly, so fix this issue.

    Change-Id: I002838d1dd57a48bd8e40f4c2c59f310e1c07987
    Closes-bug: #1589415
    (cherry picked from commit f4486930e199da11ef132728c4f63085bb233da6)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 6.1.0

This issue was fixed in the openstack/heat 6.1.0 release.

Revision history for this message
Jeremy Pugh (pugh-jeremy) wrote :

I'm still not sure this is working as intended, because I now see alarms with queries like this after I added the patch and created new stacks:

metadata.metering.stack == 0321cb4a-ee98-4caa-bc23-118654b61ceb

Revision history for this message
Jeremy Pugh (pugh-jeremy) wrote :

to be clear, I was looking at memory.resident in mitaka for generating alarms and hit this after the patch. If I add memory.resident to NOVA_METERS in alarm.py, I notice the query is correct and the alarm will then trigger. Honestly, I'm not sure yet what this means...if memory.resident is supposed to be in the NOVA_METERS list?

Revision history for this message
Jeremy Pugh (pugh-jeremy) wrote :

hmm, when I look in the ceilometer polling logs for our nodes which run nova-compute, it appears that this is indeed a NOVA derived meter, so I'm not sure why this one isn't in the list of NOVA_METERS. I'll just open a new bug ticket.

2016-09-14 17:14:14.447 629 INFO ceilometer.agent.manager [req-9dff4666-7ab6-4c43-9ab6-1c13587a5f81 admin - - - -] Polling pollster memory.resident in the context of meter_source
2016-09-14 17:14:14.459 629 INFO ceilometer.agent.manager [req-9dff4666-7ab6-4c43-9ab6-1c13587a5f81 admin - - - -] Polling pollster disk.device.allocation in the context of meter_source
2016-09-14 17:14:14.470 629 INFO ceilometer.agent.manager [req-9dff4666-7ab6-4c43-9ab6-1c13587a5f81 admin - - - -] Polling pollster memory.usage in the context of meter_source

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

This issue was fixed in the openstack/heat 6.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.