test_alarm fails on gate

[Dmitriy]

Gates fail on test_alarm with the following traceback:

2016-06-13 13:04:48.107 | 2016-06-13 13:04:48.085 | heat_integrationtests.scenario.test_ceilometer_alarm.CeilometerAlarmTest.test_alarm
2016-06-13 13:04:48.107 | 2016-06-13 13:04:48.087 | -----------------------------------------------------------------------------------
2016-06-13 13:04:48.109 | 2016-06-13 13:04:48.089 |
2016-06-13 13:04:48.110 | 2016-06-13 13:04:48.090 | Captured traceback:
2016-06-13 13:04:48.112 | 2016-06-13 13:04:48.092 | ~~~~~~~~~~~~~~~~~~~
2016-06-13 13:04:48.115 | 2016-06-13 13:04:48.094 | Traceback (most recent call last):
2016-06-13 13:04:48.115 | 2016-06-13 13:04:48.095 | File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 352, in _stack_delete
2016-06-13 13:04:48.117 | 2016-06-13 13:04:48.097 | success_on_not_found=True)
2016-06-13 13:04:48.119 | 2016-06-13 13:04:48.099 | File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 342, in _wait_for_stack_status
2016-06-13 13:04:48.121 | 2016-06-13 13:04:48.101 | raise exceptions.TimeoutException(message)
2016-06-13 13:04:48.124 | 2016-06-13 13:04:48.103 | heat_integrationtests.common.exceptions.TimeoutException: Request timed out
2016-06-13 13:04:48.136 | 2016-06-13 13:04:48.105 | Details: Stack CeilometerAlarmTest-1612240712 failed to reach DELETE_COMPLETE status within the required time (1200 s).

[Pavlo Shchelokovskyy (pshchelo)]

soo, I've got this too. Judging from the logs:

start of stack delete:

http://logs.openstack.org/96/279096/3/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2/ac1c829/logs/screen-h-eng.txt.gz#_2016-06-21_09_27_44_071

failure to acquire resource lock:

http://logs.openstack.org/96/279096/3/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2/ac1c829/logs/screen-h-eng.txt.gz#_2016-06-21_09_27_44_388

but right before the failure I see that there was still something going on with stack not related to delete - Setting metadata for AutoScalingPolicy "scaleup_policy":

http://logs.openstack.org/96/279096/3/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2/ac1c829/logs/screen-h-eng.txt.gz#_2016-06-21_09_27_44_359

So it seems there is some race during delete in convergence engine, not all internal resource actions are stopped before the delete starts. Or, there should be a retry mechanism in these places.

[Rabi Mishra (rabi)]

It's same as bug #1607814. Though this was reported earlier, we've number of discussions and patches on that bug. So marking this as duplicate.