combination alarm should be evaluatored by dependency order

Bug #1310500 reported by ZhiQiang Fan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Aodh
Won't Fix
Undecided
Unassigned
Ceilometer
Won't Fix
Undecided
Unassigned

Bug Description

currently, the GET /alarms returns alarms ordered by id, but think about a complex scenarion, such as

create threshold alarm ta1
create threshold alarm ta2
create threshold alarm ta3
create combination alarm ca1, alarm_ids = ta1 and ta2
create combination alarm ca2, alarm_ids = ta1 or ta2
create combination alarm ca3, alarm_ids = ca1 and ta3

then alarm-evaluator get a list of alarms and evaluator them one by one, the order can be ca3, ca2, ca1, ta3, ta2, ta1
so ta1 and ta2 are properly updated in period 1
then ca1 and ca2 are properly updated in period 2
then ca3 is properly updated in period 3, then user get notified after 2 periods

the longer dependency chain is, the later user will get notified

I think we could update all alarms in one single period, which means if
ta1 and ta2 and ta3 are all triggerd, then ca3 should be triggerd in
same period.

I have another bug reported to solve dead loop in dependency chain, so in this bug fix, the cycled alarms may be ignored directly, this can be discussed in patch review

ZhiQiang Fan (aji-zqfan)
Changed in ceilometer:
assignee: nobody → ZhiQiang Fan (aji-zqfan)
ZhiQiang Fan (aji-zqfan)
summary: - combination alarm should be evaluatored in one single period
+ combination alarm should be evaluatored in less periods
Revision history for this message
ZhiQiang Fan (aji-zqfan) wrote : Re: combination alarm should be evaluatored in less periods

when write test code, I find that there is no guarantee for evaluate all alarms in one periods,

that because, even we can decide the right order when evaluate alarms, but the for loop is very quick and the evaluate process is very slow compare to the loop, so, there is a high possibility that the depended alarm is being evaluated and the state value is not flushed into database, the meanwhile, the alarm get request has been sent, so we get the state of last period of that alarm

description: updated
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/89330

Changed in ceilometer:
status: New → In Progress
ZhiQiang Fan (aji-zqfan)
summary: - combination alarm should be evaluatored in less periods
+ combination alarm should be evaluatored by dependency order
Revision history for this message
ZhiQiang Fan (aji-zqfan) wrote :

after a deep thought, I think currently alarm-evaluator only has one work, and everything works in sequence, which means it is safe, my previous comment only valid in parallel or asychronous scenarions

description: updated
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix proposed to ceilometer (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/89335

gordon chung (chungg)
Changed in ceilometer:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ceilometer (master)

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/89330
Reason: marking abandoned as this is crazy old. please reopen if you feel like it's still worthwhile.

gordon chung (chungg)
Changed in ceilometer:
status: In Progress → Triaged
assignee: ZhiQiang Fan (aji-zqfan) → nobody
Revision history for this message
ZhiQiang Fan (aji-zqfan) wrote :

@gordon I think we can remove the tag for Ceilometer because alarm code is going to be removed in this project.

ZhiQiang Fan (aji-zqfan)
Changed in ceilometer:
status: Triaged → Won't Fix
importance: Medium → Undecided
Changed in aodh:
importance: Undecided → Medium
assignee: nobody → ZhiQiang Fan (aji-zqfan)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to aodh (master)

Fix proposed to branch: master
Review: https://review.openstack.org/228154

Changed in aodh:
status: New → In Progress
ZhiQiang Fan (aji-zqfan)
Changed in aodh:
status: In Progress → Won't Fix
importance: Medium → Undecided
assignee: ZhiQiang Fan (aji-zqfan) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on aodh (master)

Change abandoned by ZhiQiang Fan (<email address hidden>) on branch: master
Review: https://review.openstack.org/228154
Reason: too old

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.