Heat stack stops scaling after "can't compare datetime.datetime to NoneType"

[Dmitry Sutyagin]

Confirmed on MOS 6.0 MU7, Ubuntu. Heat package version - 2014.2-fuel6.0~mira10.
Reported by Heat team to be related to https://bugs.launchpad.net/mos/+bug/1474332

Issue summary:
Heat stack stops scaling after several failed attempts (due to quotas / other non-heat errors).
All subsequent alarm triggers result in the following log messages:

<134>Mar 3 16:43:26 node-13 heat-engine 2016-03-03 16:43:26.723 10851 INFO heat.engine.resources.openstack.scaling_policy [req-65d7f565-c8f7-474d-8eac-8d6a058d2d45 None] scale_up_policy Alarm, adjusting Group my_asg with id march3rd-my_asg-qyq2gbppzjck by 1
<134>Mar 3 16:43:26 node-13 heat-engine 2016-03-03 16:43:26.724 10851 INFO heat.engine.resources.autoscaling [req-65d7f565-c8f7-474d-8eac-8d6a058d2d45 None] my_asg NOT performing scaling adjustment, cooldown 60

This behavior starts after the following traceback:

<131>Mar 3 16:40:26 node-13 heat-engine 2016-03-03 16:40:26.659 10851 ERROR heat.engine.resource [req-23cf5695-68b9-4ae7-a15f-39414a9c666f None] signal AutoScalingPolicy "scale_up_policy" [d7186b9aa0d94f22bb6719c3829dd852] Stack "march3rd" [2443de4f-ca1f-4bb3-9102-bfa3f863f444] : can't compare datetime.datetime to NoneType
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource Traceback (most recent call last):
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 1032, in signal
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource signal_result = self.handle_signal(details)
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/scaling_policy.py", line 153, in handle_signal
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource adjustment_type)
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resources/autoscaling.py", line 622, in adjust
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource capacity = len(self.get_instances())
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resources/autoscaling.py", line 208, in get_instances
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource return sorted(resources, key=lambda r: (r.created_time, r.name))
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource TypeError: can't compare datetime.datetime to NoneType
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource

Attached are extracts from mysql heat database.

[Dmitry Sutyagin (dsutyagin)]

Logs (for Mirantis staff) https://drive.google.com/a/mirantis.com/folderview?id=0B2VydA-aQqI9M20xaU90UHhhVk0&usp=sharing

[Dmitry Sutyagin (dsutyagin)]

Probably related to https://bugs.launchpad.net/heat/+bug/1480183

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: huangtianhua <email address hidden>
Review: https://review.fuel-infra.org/18087

[Sergey Kraynev (skraynev)]

Looks like issue, which was fixed in Liberty.
There is a similar bug: https://bugs.launchpad.net/heat/+bug/1480183
With fix: https://review.openstack.org/#/c/208386/

[Dmitry Sutyagin (dsutyagin)]

Tested with https://review.fuel-infra.org/18087 - could not reproduce, so I think the fix is good.

[Sergey Kraynev (skraynev)]

Looks like Mitaka and Liberty contain necessary fixes, so we may just check, that it works correct without porting and applying any fixes.

[Dina Belova (dbelova)]

Sergey, once you'll verify needed fixes are presented in L/M please move to Invalid for appropriate series.

[Alexey Stupnikov (astupnikov)]

Steps to reproduce:
1. Modify project's quota, limit number of volumes allowed.

2. Install and configure heat CLI client

3. Copy attached files to your current dir

4. Start the stack:
heat stack-create simple -f simple.yaml -e environment.yaml

5. Wait for 20 minutes until the messages from bug description will come up in heat-all.log

[Alexander Nagovitsyn (gluk12189)]

On latest 9.0 - could not reproduce

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: huangtianhua <email address hidden>
Review: https://review.fuel-infra.org/20416

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: huangtianhua <email address hidden>
Review: https://review.fuel-infra.org/20417

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20417
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 231adfda976713d13957e34ff710d39f00a259ef
Author: huangtianhua <email address hidden>
Date: Tue May 10 07:54:10 2016

Fix incorrect resource's information while describing

We can't get the resource'infos from db once the resource
is deleted. All information of a deleted resource come
from the initialization. So this change will modify some
infos when resource init:
1. set the action of resource to stack'action
2. set the created_time to stack's created_time
3. set the updated_time to stack's updated_time

Then the information will be reset for normal resources.

Change-Id: Iafdc99c572c5dab06bf1843842ed47554d6de128
Closes-Bug: #1557013

[Alexey Stupnikov (astupnikov)]

Setting this bug's status to Invalid for MOS8, since the fix is already in MOS8 code.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20416
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 0ad90080053d64007436e9edc8a4fbc19cc7d0a7
Author: huangtianhua <email address hidden>
Date: Tue May 10 08:11:32 2016

Fix incorrect resource's information while describing

We can't get the resource'infos from db once the resource
is deleted. All information of a deleted resource come
from the initialization. So this change will modify some
infos when resource init:
1. set the action of resource to stack'action
2. set the created_time to stack's created_time
3. set the updated_time to stack's updated_time

Then the information will be reset for normal resources.

Change-Id: Iafdc99c572c5dab06bf1843842ed47554d6de128
Closes-Bug: #1557013

[Alexander Gromov (agromov)]

Verified on MOS 6.1 + mu6 updates.

Steps to reproduce are described above.

Notes:
1) limit of number of volumes was set to 3.
2) log can appear later then 20 minutes

Without updates the trace from description was found on one controller.

After updates the trace wasn't found. Error "OverLimit: VolumeLimitExceeded" continued to appear.

[Alexander Gromov (agromov)]

Verified on MOS 7.0 + mu4 updates.

Steps to reproduce are described above.

[Sergey Kraynev (skraynev)]

It's invalid for any release after Liberty, because fix is already in upstream code.

[Sergey Kraynev (skraynev)]

Fix for 6.0 was proposed, so please just review it https://review.fuel-infra.org/#/c/18087/