Unable to delete failed stack

[Endre Karlson]

http://pastebin.com/Y9tFYxRm < see log for details. Tried both deletion in CLI and via the Horizon Thermal plugin. Also with or without admin privileges.

[Endre Karlson (endre-karlson)]

Heat engine.log is attached.

[Zane Bitter (zaneb)]

The relevant portion of the log:

2012-12-12 13:19:32 WARNING [heat.engine.resources.instance] Instance IP address not found (The resource could not be found. (HTTP 404) (Request-ID: req-b795c3a5-bab1-4677-b9f6-996cf8353144))
2012-12-12 13:19:32 INFO [heat.engine.resources.instance] OpenShiftBrokerServer.GetAtt(PublicIp) == 0.0.0.0
2012-12-12 13:19:32 WARNING [heat.engine.resources.instance] Instance IP address not found (The resource could not be found. (HTTP 404) (Request-ID: req-deaa184a-d4fa-4764-abe1-d00b65f70785))
2012-12-12 13:19:32 INFO [heat.engine.resources.instance] OpenShiftBrokerServer.GetAtt(PublicIp) == 0.0.0.0
2012-12-12 13:19:32 INFO [heat.engine.resource] deleting WaitCondition "WaitCondition" (inst:None db_id:None)
2012-12-12 13:19:32 ERROR [heat.engine.resource] Delete WaitCondition "WaitCondition"
Traceback (most recent call last):
  File "/root/heat/heat/engine/resource.py", line 291, in delete
    self.handle_delete()
  File "/root/heat/heat/engine/resources/wait_condition.py", line 200, in handle_delete
    self._get_handle_resource_id()
  File "/root/heat/heat/engine/resources/wait_condition.py", line 157, in _get_handle_resource_id
    handle_id = identifier.ResourceIdentifier.from_arn_url(handle_url)
  File "/root/heat/heat/common/identifier.py", line 76, in from_arn_url
    raise ValueError('"%s" is not a valid URL' % url)
ValueError: "WaitHandle" is not a valid URL
2012-12-12 13:19:32 ERROR [heat.engine.parser] Failed to delete WaitCondition "WaitCondition" error: "WaitHandle" is not a valid URL
2012-12-12 13:19:32 INFO [heat.engine.resource] deleting Instance "OpenShiftBrokerServer" (inst:None db_id:None)

[Zane Bitter (zaneb)]

This is due to a change in the internal implementation where upgrading heat breaks stacks with WaitConditions that were created prior to the upgrade. We're currently not offering any guarantees about upgrades (particularly in development versions) not breaking things.

To get rid of this, clear out the database and recreate it.

[Endre Karlson (endre-karlson)]

Kind of the same problem post db-refresh:
2012-12-12 14:26:07 DEBUG [heat.openstack.common.rpc.amqp] unpacked context: {'username': None, 'service_user': u'heat', 'service_tenant': u'service', 'roles': u'admin,Member', 'aws_auth_uri': None, 'tenant_id': u'c97027dd880d4c129ae7a4ba7edade05', 'auth_token': '<SANITIZED>', 'service_password': u'svcp4ss', 'auth_url': u'http://127.0.0.1:5000/v2.0', 'is_admin': True, 'password': None, 'aws_creds': None, 'tenant': u'Smart'}
2012-12-12 14:26:07 INFO [heat.engine.service] deleting stack master
2012-12-12 14:26:07 INFO [heat.engine.resource] deleting SecurityGroup "EC2SecurityGroup" (inst:None db_id:None)
2012-12-12 14:26:08 INFO [heat.engine.resource] deleting WaitCondition "PuppetMasterWaitCondition" (inst:None db_id:None)
2012-12-12 14:26:08 ERROR [heat.engine.resource] Delete WaitCondition "PuppetMasterWaitCondition"
Traceback (most recent call last):
  File "/root/heat/heat/engine/resource.py", line 291, in delete
    self.handle_delete()
  File "/root/heat/heat/engine/resources/wait_condition.py", line 200, in handle_delete
    self._get_handle_resource_id()
  File "/root/heat/heat/engine/resources/wait_condition.py", line 157, in _get_handle_resource_id
    handle_id = identifier.ResourceIdentifier.from_arn_url(handle_url)
  File "/root/heat/heat/common/identifier.py", line 76, in from_arn_url
    raise ValueError('"%s" is not a valid URL' % url)
ValueError: "PuppetMasterWaitHandle" is not a valid URL
2012-12-12 14:26:08 ERROR [heat.engine.parser] Failed to delete WaitCondition "PuppetMasterWaitCondition" error: "PuppetMasterWaitHandle" is not a valid URL
2012-12-12 14:26:08 INFO [heat.engine.resource] deleting Instance "PuppetMasterInstance" (inst:None db_id:None)
2012-12-12 14:26:08 WARNING [heat.engine.resources.user] could not get secret for master.CFNInitUser Error:resource_id not yet set
2012-12-12 14:26:08 INFO [heat.engine.resources.user] master.CFNKeys.GetAtt(SecretAccessKey) == <SANITIZED>
2012-12-12 14:26:08 WARNING [heat.engine.resources.user] could not get secret for master.CFNInitUser Error:resource_id not yet set
2012-12-12 14:26:08 INFO [heat.engine.resources.user] master.CFNKeys.GetAtt(SecretAccessKey) == <SANITIZED>
2012-12-12 14:26:08 INFO [heat.engine.resource] deleting AccessKey "CFNKeys" (inst:None db_id:None)
2012-12-12 14:26:08 INFO [heat.engine.resource] deleting User "CFNInitUser" (inst:None db_id:None)
2012-12-12 14:26:08 ERROR [heat.engine.resources.user] Cannot delete User resource before user created!
2012-12-12 14:26:08 INFO [heat.engine.resource] deleting WaitConditionHandle "PuppetMasterWaitHandle" (inst:None db_id:None)
2012-12-12 14:26:08 INFO [heat.engine.resource] deleting SecurityGroup "PuppetGroup" (inst:None db_id:None)
2012-12-12 14:26:08 DEBUG [amqplib] Closed channel #1
2012-12-12 14:26:08 DEBUG [amqplib] using channel_id: 1
2012-12-12 14:26:08 DEBUG [amqplib] Channel open
2012-12-12 14:26:08 DEBUG [amqplib] Closed channel #1
2012-12-12 14:26:08 DEBUG [amqplib] using channel_id: 1
2012-12-12 14:26:08 DEBUG [amqplib] Channel open
2012-12-12 14:26:08 DEBUG [heat.openstack.common.rpc....

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/18240

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/18240
Committed: http://github.com/openstack/heat/commit/50cbd80650655338b7563966083388b564f8567c
Submitter: Jenkins
Branch: master

commit 50cbd80650655338b7563966083388b564f8567c
Author: Zane Bitter <email address hidden>
Date: Mon Dec 17 19:56:22 2012 +0100

    Fix issues with deleting a WaitCondition

    Previously we were ignoring the reference to the WaitConditionHandle stored
    in the database by always setting it to None on load. This meant that we
    had to try to recreate it on delete, which would fail if e.g. the
    WaitConditionHandle had already been deleted.

    bug 1089354

    Change-Id: Ibc4145348a91b455e97ba27748ff72451b0c7bce
    Signed-off-by: Zane Bitter <email address hidden>