stack delete overcloud fails on Delete AccessKey "NovaCompute0Key"

[Gregory Haynes]

Running heat stack-delete overcloud results in deleted nova instances, but DELETE_FAILED stack state.

The following traceback is found in heat-engine.log:
2014-03-11 21:08:02.098 2939 ERROR heat.engine.resource [-] Delete AccessKey "NovaCompute0Key" [cc09945f29424dd48c42008a5a3fb2e7] Stack "overcloud" [6317749d-d991-4af4-b3d4-ad71ca3437bb]
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource Traceback (most recent call last):
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource File "/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/resource.py", line 683, in delete
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource handle_data = self.handle_delete()
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource File "/opt/stack/venvs/heat/lib/python2.7/site-packages/heat/engine/resources/user.py", line 209, in handle_delete
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource user._delete_keypair()
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource File "/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/stack_user.py", line 138, in _delete_keypair
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource credential_id = db_api.resource_data_get(self, 'credential_id')
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource File "/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/db/api.py", line 62, in resource_data_get
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource return IMPL.resource_data_get(resource, key)
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource File "/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 157, in resource_data_get
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource key)
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource File "/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 188, in resource_data_get_by_key
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource raise exception.NotFound(_('No resource data found'))
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource NotFound: No resource data found
2014-03-11 21:08:02.098 2939 TRACE heat.engine.resource

[Robert Collins (lifeless)]

(undercloud)robertc@lifelesswks:~/work$ heat event-list overcloud
+------------------------------+--------------------------------------+-----------------------------------------------------------------------+--------------------+----------------------+
| resource_name | id | resource_status_reason | resource_status | event_time |
+------------------------------+--------------------------------------+-----------------------------------------------------------------------+--------------------+----------------------+
| notCompute0CompletionHandle | ec61c18c-4af4-448a-9e19-692c9ba0dd79 | state changed | CREATE_IN_PROGRESS | 2014-03-12T18:46:07Z |
| NovaCompute0CompletionHandle | 51f41789-5622-461b-ac47-d33170c9edd5 | state changed | CREATE_IN_PROGRESS | 2014-03-12T18:46:07Z |
| AccessPolicy | ee32432a-6ff9-445d-9023-4e912ad0ff6f | state changed | CREATE_IN_PROGRESS | 2014-03-12T18:46:07Z |
| RabbitCookie | 3cc4b8fc-ae4a-48f7-98e6-fbe22728710d | state changed | CREATE_IN_PROGRESS | 2014-03-12T18:46:07Z |
| NovaCompute1CompletionHandle | f6b84a01-dd93-4c7d-bd2c-6fb90ac748fa | state changed | CREATE_IN_PROGRESS | 2014-03-12T18:46:07Z |
| ComputeAccessPolicy | cef14755-45ac-44c3-b2f7-eda4465a3419 | state changed | CREATE_IN_PROGRESS | 2014-03-12T18:46:07Z |
| NovaCompute0CompletionHandle | b6950433-a024-4efa-8f32-cd35a29c594d | state changed | CREATE_COMPLETE | 2014-03-12T18:46:07Z |
| ComputeAccessPolicy | 92342287-2fe8-4966-b1bc-83d5ef3b0767 | state changed | CREATE_COMPLETE | 2014-03-12T18:46:07Z |
| AccessPolicy | c4ed0c12-3168-47b3-be4f-e6e0b3c2a4f1 | state changed | CREATE_COMPLETE | 2014-03-12T18:46:07Z |
| notCompute0CompletionHandle | 4d48d664-7e14-4a9b-88f2-ce304d78794a | state changed | CREATE_COMPLETE | 2014-03-12T18:46:07Z |
| RabbitCookie | e0755776-dbc8-46d6-8581-0d5591b15dba | state changed | CREATE_COMPLETE | 2014-03-12T18:46:07Z |
| NovaCompute1CompletionHandle | 0b75010b-67ea-4479-b7ef-e2f4f7eeaf2a | state changed | CREATE_COMPLETE | 2014-03-12T18:46:07Z |
| User | 63f1cfe5-7746-4a57-b3e2-dffaca5ba91e | state changed | CREATE_IN_PROGRESS | 2014-03-12T18:46:07Z |
| ComputeUser | 40f6eb62-8414-4009-82a1-e466701f1c5b | state changed ...

[Clint Byrum (clint-fewbar)]

It is a bit confusing why the access keys are not there, but we should not fail to delete on a "not found" message as that is the target state anyway.

[Steve Baker (steve-stevebaker)]

Is there likely a Heat bug to fix here?

[Gregory Haynes (greghaynes)]

Yes, just pointed the review at this ticket.

[Steve Baker (steve-stevebaker)]

https://review.openstack.org/#/c/80116/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/80116
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=efeb365f7bdc79705ab956c16455550b664ab569
Submitter: Jenkins
Branch: master

commit efeb365f7bdc79705ab956c16455550b664ab569
Author: Gregory Haynes <email address hidden>
Date: Wed Mar 12 16:41:50 2014 -0700

    Catch NotFound exception on user delete_key

    Deleting a key which doesn't exist shouldn't cause a failure. Catch
    NotFound exceptions so delete can succeed in this case.

    Change-Id: If651a2936862f8ed2168c2c216ef6df8e944d6ba
    Closes-Bug: #1291060