Extra instance is created if evaluation period set short

Bug #1375156 reported by Hang Liu
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Medium
Yanyan Hu
Juno
Fix Released
Medium
Zane Bitter
Kilo
Fix Released
Undecided
Unassigned

Bug Description

If evaluation period is set shorter than the instance starting time. And 'repeat_actions' is set to True.
Extra instances would be created, even the cooldown time has been set long enough.

Possible Reasons:

    1) No guarding logic at alarm signals in ASG code.
    2) If the 2nd instance starts up slow, a 2nd alarm may arrive before the 2nd instance is ready.
    3) The very first 'cooldown' timestamp is not computed/set because the 2nd instance is not ready. So the 3rd instance (maybe the 4th, 5th, ... instances are quickly launched).

Qiming Teng (tengqim)
Changed in heat:
assignee: nobody → Qiming Teng (tengqim)
status: New → Confirmed
Angus Salkeld (asalkeld)
Changed in heat:
importance: Undecided → Medium
Revision history for this message
Deliang Fan (vanderliang) wrote :

Should we add a para to indicate the status of ASG, for example, PROGRESS, COMPLETE. If the status of ASG resource is process autoscaling, then any signal will be ignored. Only if that the ASG status is COMPLETE and out of cool down time can scale.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/186659

Changed in heat:
assignee: Qiming Teng (tengqim) → Yanyan Hu (yanyanhu)
status: Confirmed → In Progress
Changed in heat:
status: In Progress → Fix Committed
tags: added: juno-backport-potential kilo-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/202660

Thierry Carrez (ttx)
Changed in heat:
milestone: none → liberty-2
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/kilo)

Reviewed: https://review.openstack.org/202660
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=034c58261194eacfc320e9174d4240e12224cd66
Submitter: Jenkins
Branch: stable/kilo

commit 034c58261194eacfc320e9174d4240e12224cd66
Author: yanyanhu <email address hidden>
Date: Fri May 29 01:43:03 2015 -0400

    ASG scaling account for cooldown timestamp & in-progress

    There are cases where it takes a long time to create a new resource as
    requested by the scaling operation on an ASG resource, for instance, a
    nova server creation followed by a complex SoftwareDeployment. During
    this process, additional alarms may come in but failed to be blocked
    by the current cooldown checking mechanism because the very first
    timestamp has yet to be generated. This is leading to unexpected size
    adjustment to the ASG.

    This patch augments the existing cooldown checking mechanism with a
    scaling-in-progress test so that additional alarms arriving during
    the very first scaling operation will be ignored.

    Closes-Bug: #1375156

    Conflicts:
     heat/engine/resources/openstack/heat/scaling_policy.py
     heat/scaling/cooldown.py
     heat/tests/autoscaling/test_heat_scaling_group.py
     heat/tests/autoscaling/test_heat_scaling_policy.py
     heat/tests/autoscaling/test_scaling_group.py
     heat/tests/autoscaling/test_scaling_policy.py

    (cherry-picked from b76881b8bcb3e381c5b993a0c84afd37abd8ea5b)
    Change-Id: Ib8aa83eed366df7097c9cbb9247eca866ae4b620

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/224753

tags: removed: kilo-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/231798

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (stable/juno)

Change abandoned by Zane Bitter (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/224753
Reason: I finally got this working, so to reduce future confusion I reproposed with the correct Change-Id at https://review.openstack.org/#/c/231798/

Thierry Carrez (ttx)
Changed in heat:
milestone: liberty-2 → 5.0.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/juno)

Reviewed: https://review.openstack.org/231798
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=e182a57c9a8ba2f5f4203fac3d2aa04d45dffb5e
Submitter: Jenkins
Branch: stable/juno

commit e182a57c9a8ba2f5f4203fac3d2aa04d45dffb5e
Author: yanyanhu <email address hidden>
Date: Thu Sep 17 09:29:50 2015 -0400

    ASG scaling account for cooldown timestamp & in-progress

    There are cases where it takes a long time to create a new resource as
    requested by the scaling operation on an ASG resource, for instance, a
    nova server creation followed by a complex SoftwareDeployment. During
    this process, additional alarms may come in but failed to be blocked
    by the current cooldown checking mechanism because the very first
    timestamp has yet to be generated. This is leading to unexpected size
    adjustment to the ASG.

    This patch augments the existing cooldown checking mechanism with a
    scaling-in-progress test so that additional alarms arriving during
    the very first scaling operation will be ignored.

    Change-Id: Ib8aa83eed366df7097c9cbb9247eca866ae4b620
    Co-Authored-By: Zane Bitter <email address hidden>
    Closes-Bug: #1375156

Zane Bitter (zaneb)
tags: removed: juno-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.