Extra instance is created if evaluation period set short (backport upstream)

[Denis Klepikov]

Backport upstream fix for bug
https://bugs.launchpad.net/heat/+bug/1375156
to fuel 5.1, 5.1.1, 6.0, 6.1

[Vitaly Sedelnik (vsedelnik)]

Denis, this fix has Medium importance in upstream. We are not including bugfixes for Medium bugs into maintenance updates. Please provide more information to raise the importance to High - which customer is affected, what is the impact, etc.

[Denis Klepikov (dklepikov)]

HEAT autoscaling creates 3 new instances, instead of 1 (scaling adjustment is set to 1).
This bug happens every time, for all tenants.
If you delete the stack after autoscaling is performed it will only delete 2 instances. 2 instances will remain; it's like HEAT doesn't know about those two instances.

[Eugene Nikanorov (enikanorov)]

The impact is 50% increased resource and quota usage and confusion for the users.
It's pretty severe.

If you don't want to add it to updates - please just backport.

[Miroslav Anashkin (manashkin)]

This bug is customer-found. We should consider all such bugs as high.
We are not allowed to mention the customer name until we make this bug private.

[Aleksander Mogylchenko (amogylchenko)]

mos-heat should release a fix for 7.0 branch.

[Pavlo Shchelokovskyy (pshchelo)]

backport to stable/kilo branch is on review in upstream
https://review.openstack.org/#/c/202660/

we will get it with next stable/kilo sync or we will apply the patch by ourselves if schedule becomes tight

[Vitaly Sedelnik (vsedelnik)]

Ok, confirmed for 5.1/5.1.1-updates

[Vitaly Sedelnik (vsedelnik)]

Backport is on review - https://review.fuel-infra.org/#/c/9722/

[Sergey Kraynev (skraynev)]

Vitaly: what about 6.1 and 6.0 ?

[Sergey Kraynev (skraynev)]

Patch for MOS 7.0 was merged https://review.fuel-infra.org/#/c/9533/

[Sergey Kraynev (skraynev)]

Wait, when unittests for https://review.fuel-infra.org/#/c/9722/ will be fixed and then we can merge it too.

[Vitaly Sedelnik (vsedelnik)]

Nominated for 6.0 and 6.1 maint updates

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-5.1-updates/2014.1.1
Change author: yanyanhu <email address hidden>
Review: https://review.fuel-infra.org/9897

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/9897
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1-updates/2014.1.1

Commit: 8b90f6c853edda18f24a220358191b18b7464e14
Author: yanyanhu <email address hidden>
Date: Mon Jul 27 21:30:33 2015

ASG scaling account for cooldown timestamp & in-progress

There are cases where it takes a long time to create a new resource as
requested by the scaling operation on an ASG resource, for instance, a
nova server creation followed by a complex SoftwareDeployment. During
this process, additional alarms may come in but failed to be blocked
by the current cooldown checking mechanism because the very first
timestamp has yet to be generated. This is leading to unexpected size
adjustment to the ASG.

This patch augments the existing cooldown checking mechanism with a
scaling-in-progress test so that additional alarms arriving during
the very first scaling operation will be ignored.

Closes-Bug: #1474332
Co-Authored-By: yanyanhu <email address hidden>

Conflicts:
 heat/engine/resources/openstack/heat/scaling_policy.py
 heat/scaling/cooldown.py
 heat/tests/autoscaling/test_heat_scaling_group.py
 heat/tests/autoscaling/test_heat_scaling_policy.py
 heat/tests/autoscaling/test_scaling_group.py
 heat/tests/autoscaling/test_scaling_policy.py

(cherry-picked from Ib8aa83eed366df7097c9cbb9247eca866ae4b620)
Change-Id: Ib8aa83eed366df7097c9cbb9247eca866ae4b620

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/9722
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1.1-updates/2014.1.1

Commit: f0edfd3b0a9a64a4e5b13694348d2c58a4e6ca6a
Author: yanyanhu <email address hidden>
Date: Mon Jul 27 20:23:14 2015

ASG scaling account for cooldown timestamp & in-progress

There are cases where it takes a long time to create a new resource as
requested by the scaling operation on an ASG resource, for instance, a
nova server creation followed by a complex SoftwareDeployment. During
this process, additional alarms may come in but failed to be blocked
by the current cooldown checking mechanism because the very first
timestamp has yet to be generated. This is leading to unexpected size
adjustment to the ASG.

This patch augments the existing cooldown checking mechanism with a
scaling-in-progress test so that additional alarms arriving during
the very first scaling operation will be ignored.

Closes-Bug: #1474332
Co-Authored-By: yanyanhu <email address hidden>

Conflicts:
 heat/engine/resources/openstack/heat/scaling_policy.py
 heat/scaling/cooldown.py
 heat/tests/autoscaling/test_heat_scaling_group.py
 heat/tests/autoscaling/test_heat_scaling_policy.py
 heat/tests/autoscaling/test_scaling_group.py
 heat/tests/autoscaling/test_scaling_policy.py

(cherry-picked from Ib8aa83eed366df7097c9cbb9247eca866ae4b620)
Change-Id: Ib8aa83eed366df7097c9cbb9247eca866ae4b620

[Vitaly Sedelnik (vsedelnik)]

Moved to MU-6 for 6.0-updates because mos-infra-ci jobs need to be fixed in order to get the change merged.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10084
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 67c896aaed878a8b3596f49934911379767054f8
Author: yanyanhu <email address hidden>
Date: Tue Sep 8 17:02:14 2015

ASG scaling account for cooldown timestamp & in-progress

There are cases where it takes a long time to create a new resource as
requested by the scaling operation on an ASG resource, for instance, a
nova server creation followed by a complex SoftwareDeployment. During
this process, additional alarms may come in but failed to be blocked
by the current cooldown checking mechanism because the very first
timestamp has yet to be generated. This is leading to unexpected size
adjustment to the ASG.

This patch augments the existing cooldown checking mechanism with a
scaling-in-progress test so that additional alarms arriving during
the very first scaling operation will be ignored.

Closes-Bug: #1474332
Co-Authored-By: yanyanhu <email address hidden>
Change-Id: Ib8aa83eed366df7097c9cbb9247eca866ae4b620

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Alexey Khivin <email address hidden>
Review: https://review.fuel-infra.org/11329

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11329
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 039701f29f1a4316155e5dd6e0a60228819c51f8
Author: Alexey Khivin <email address hidden>
Date: Wed Sep 9 11:45:53 2015

Fix broken test

Test broken in change (Ib8aa83eed366df7097c9cbb9247eca866ae4b620)

Change-Id: I7d8ba1527a53e61f3af8bd1cb012dc59caad45b8
Related-bug: #1474332

[Vitaly Gusev (vgusev)]

Verified on 6.0 with packets *heat*2014.2-fuel6.0~mira6_all.deb from mirror http://172.18.82.150:82/ubuntu-fuel-6.0-updates-stable/ubuntu/all/

[Vitaly Sedelnik (vsedelnik)]

The fix was committed to 5.1.1-updates branch, need to make sure it landed to 5.1.1 branch after merge

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10083
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: a34198c6aaa99710f20025eed677f9b9654c4461
Author: yanyanhu <email address hidden>
Date: Wed Sep 9 11:29:14 2015

ASG scaling account for cooldown timestamp & in-progress

There are cases where it takes a long time to create a new resource as
requested by the scaling operation on an ASG resource, for instance, a
nova server creation followed by a complex SoftwareDeployment. During
this process, additional alarms may come in but failed to be blocked
by the current cooldown checking mechanism because the very first
timestamp has yet to be generated. This is leading to unexpected size
adjustment to the ASG.

This patch augments the existing cooldown checking mechanism with a
scaling-in-progress test so that additional alarms arriving during
the very first scaling operation will be ignored.

Closes-Bug: #1474332
Co-Authored-By: yanyanhu <email address hidden>
Change-Id: Ib8aa83eed366df7097c9cbb9247eca866ae4b620

[Vadim Rovachev (vrovachev)]

Verified on 6.1 Ubuntu wirh packages:
heat-api,heat-common,heat-docker,heat-api-cfn,heat-api-cloudwatch,heat-engine,python-heat
Version:
2014.2.2-1~u14.04+mos11+git.a34198c.0ee6b26

[Vitaly Sedelnik (vsedelnik)]

Confirmed for 5.1.1-updates as the fix is not present is in updates repo

[Alexander Nagovitsyn (gluk12189)]

Unconfirmed on 7.0. All works fine
Used autoskaling template ( https://github.com/pshchelo/stackdev/edit/master/templates/autoscaling/asg.yaml )
And Cpuload sсript ( https://github.com/pshchelo/stackdev/blob/master/scripts/cpuload )

[Vitaly Sedelnik (vsedelnik)]

It seems that backporting the fix for the bug 1474332 (upstream bug 1375156) to MOS 5/6 was not done correctly.

Using vanilla MOS 6.0 with Ceilometer deployed (is needed for Heat's autoscaling feature):

- one controller, one compute, one mongo
  - did not bother with setting up heat domain etc, was doing things from admin

- Heat's autoscaling working as expected

- installed updated heat packages from MOS 6 updates (http://mirror.fuel-infra.org/fwm/6.0/updates/ubuntu/pool/main/h/heat/)
  - stopped heat services, installed updated heat packages (all of them, python-heat, heat-common, heat-engine, heat-api*)
  - cleaned *.pyc files
  - restarted all heat services

- autoscaling stopped working
  - ASG always thinks it is already scaling_in_progress judging by its metadata
  - grep "NOT performing scaling adjustment, cooldown" /var/log/heat/heat-engine.log

Our backport is here: https://review.fuel-infra.org/#/c/10084/
There is now a backport for the same bug in upstream stable/juno: https://review.openstack.org/#/c/231798/
Note that our backport was done before the upstream one, and comparison reveals some differences between them.

When I applied the upstream backport to the env above, Heat's autoscaling started to work again.

Therefore I propose that we revert the commits with backport in our branches and cherry-pick the upstream one:
6.0 - definitely, confirmed
6.1 - most probably, has to be verified
5.1 - most probably, but an extra effort and care has to be paid for backporting the upstream stable/juno fix to our Icehouse-based Heat of 5.1
7.0 - mos probably not (AFAIU for 7.0 fix in our branch is actually cherry-picked from stable/kilo), but better be verified.

Testing the Heat's autoscaling:
- create a stack from provided test template
  - modify parameters to your actual setup
- one nova server will be created
- assign a floating IP to this server
- login to the server over SSH with key you have specified in the template
- stress server's CPU
- after some time ceilometer alarm for high CPU usage will fire off and another server will be created
  - if stress is not released, 3rd one will be created too (max size of the autoscaling group in the test template is hard-coded to 3)
- release the CPU load
- after some time only 1 server will be left

[Vitaly Sedelnik (vsedelnik)]

Reopened (i.e. Confirmed for 6.0 and 6.1)

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: yanyanhu <email address hidden>
Review: https://review.fuel-infra.org/13914

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: yanyanhu <email address hidden>
Review: https://review.fuel-infra.org/13916

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13914
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 1c1340e2fb3dcc41ade19a5cf1f4cc1316b52f77
Author: yanyanhu <email address hidden>
Date: Fri Nov 13 15:08:42 2015

ASG scaling account for cooldown timestamp & in-progress

There are cases where it takes a long time to create a new resource as
requested by the scaling operation on an ASG resource, for instance, a
nova server creation followed by a complex SoftwareDeployment. During
this process, additional alarms may come in but failed to be blocked
by the current cooldown checking mechanism because the very first
timestamp has yet to be generated. This is leading to unexpected size
adjustment to the ASG.

This patch augments the existing cooldown checking mechanism with a
scaling-in-progress test so that additional alarms arriving during
the very first scaling operation will be ignored.

Co-Authored-By: Zane Bitter <email address hidden>
Closes-Bug: #1474332

Change-Id: I73d5deafe7baa8d3ed49ff0e93a7a80137b8ad6a

[Roman Podoliaka (rpodolyaka)]

Heat team, could you please double-check if we have the fix in 7.0 branch?

[Pavlo Shchelokovskyy (pshchelo)]

setting as Invalid for 7.0 as per comment #25

[Denis Meltsaykin (dmeltsaykin)]

Returning to the Triaged for 5.1.1-updates, as the fix was reverted and needs to be reviewed.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13916
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: b610937a910f2f562fc5692e2dac4cb341e674d3
Author: yanyanhu <email address hidden>
Date: Fri Nov 13 16:31:08 2015

ASG scaling account for cooldown timestamp & in-progress

There are cases where it takes a long time to create a new resource as
requested by the scaling operation on an ASG resource, for instance, a
nova server creation followed by a complex SoftwareDeployment. During
this process, additional alarms may come in but failed to be blocked
by the current cooldown checking mechanism because the very first
timestamp has yet to be generated. This is leading to unexpected size
adjustment to the ASG.

This patch augments the existing cooldown checking mechanism with a
scaling-in-progress test so that additional alarms arriving during
the very first scaling operation will be ignored.

Co-Authored-By: Zane Bitter <email address hidden>
Closes-Bug: #1474332
cherry-pick from e182a57c9a8ba2f5f4203fac3d2aa04d45dffb5e

Change-Id: I73d5deafe7baa8d3ed49ff0e93a7a80137b8ad6a

[Vadim Rovachev (vrovachev)]

Verified on 6.0.
packages:
heat-api,heat-api-cfn,heat-api-cloudwatch,heat-common,heat-engine,python-heat
version:
2014.2-fuel6.0~mira10

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-5.1/2014.1
Change author: yanyanhu <email address hidden>
Review: https://review.fuel-infra.org/14530

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Oleksii Chuprykov <email address hidden> on branch: openstack-ci/fuel-5.1/2014.1
Review: https://review.fuel-infra.org/14530
Reason: Not needed.

[Vadim Rovachev (vrovachev)]

Template for verify this bug:
https://github.com/pshchelo/stackdev/blob/master/templates/autoscaling/asg.yaml
https://github.com/pshchelo/stackdev/blob/master/scripts/cpuload

[Vadim Rovachev (vrovachev)]

Verified on Ubuntu 6.1
Packages:
heat-api,heat-api-cfn,heat-api-cloudwatch,heat-common,heat-docker,heat-engine,python-heat
Version:
2014.2.2-1~u14.04+mos18

[Anton Chevychalov (achevychalov)]

Won't fix for 5.1 series because end of support of that version.