Unable to delete a stack because a resource is not found

[Gaëtan Trellu]

Hi,

I tried to delete a stack but when i did it a get nothing else than: DELETE_FAILED
The only (bad) way to delete the stack was to delete the entry in the heat database...

May be could we add something to the code to avoid a failed if the resource is not found ?

Please find below the stacktrace:

REQ: curl -i https://ca.prospere.com:9696//v2.0/floatingips/7604b118-6ba9-480f-bc97-5d275642918e.json -X DELETE -H "X-Auth-Token: a20cefe401624de4b643dd6ab8f665c3" -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-neutronclient"
 http_log_req /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:173

==> api-cfn.log <==
8065: DEBUG heat.api.middleware.version_negotiation [-] Processing request: OPTIONS / Accept: process_request /usr/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:53

==> engine.log <==
7955: DEBUG neutronclient.client [-] RESP:{'date': 'Wed, 15 Oct 2014 18:47:55 GMT', 'status': '404', 'content-length': '144', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-5936de6a-0db7-4c30-848a-5cffe747d658'} {"NeutronError": {"message": "Floating IP 7604b118-6ba9-480f-bc97-5d275642918e could not be found", "type": "FloatingIPNotFound", "detail": ""}}
 http_log_resp /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:179
7955: DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": {"message": "Floating IP 7604b118-6ba9-480f-bc97-5d275642918e could not be found", "type": "FloatingIPNotFound", "detail": ""}} _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:1125
7955: DEBUG heat.engine.scheduler [-] Task destroy from None starting start /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:153
7955: DEBUG heat.engine.scheduler [-] Task destroy from None running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:186
7955: INFO heat.engine.resource [-] deleting LoadBalancer "my_lb" Stack "stack01" [46a0a37d-6aa1-49ae-a5d1-bc822005e47b]
7955: ERROR heat.engine.resource [-] Delete LoadBalancer "my_lb" Stack "stack01" [46a0a37d-6aa1-49ae-a5d1-bc822005e47b]
7955: TRACE heat.engine.resource Traceback (most recent call last):
7955: TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 707, in delete
7955: TRACE heat.engine.resource handle_data = self.handle_delete()
7955: TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/neutron/loadbalancer.py", line 617, in handle_delete
7955: TRACE heat.engine.resource member_id = db_api.resource_data_get(self, member)
7955: TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/db/api.py", line 62, in resource_data_get
7955: TRACE heat.engine.resource return IMPL.resource_data_get(resource, key)
7955: TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 157, in resource_data_get
7955: TRACE heat.engine.resource key)
7955: TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 188, in resource_data_get_by_key
7955: TRACE heat.engine.resource raise exception.NotFound(_('No resource data found'))
7955: TRACE heat.engine.resource NotFound: No resource data found
7955: TRACE heat.engine.resource
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.stack_lock [-] Engine 241047ea-f6bf-46a8-85e9-6cfcac9828d4 released lock on stack 46a0a37d-6aa1-49ae-a5d1-bc822005e47b release /usr/lib/python2.7/site-packages/heat/engine/stack_lock.py:122

[Sergey Kraynev (skraynev)]

@goldyfruit: it's look strange, because fix for this is presented more then 1 year ago and now it looks like [1]

[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/neutron/floatingip.py#L160-L162

Could you you provide information about what release of OpenStack do you use?

[Qiming Teng (tengqim)]

Seems that this is related to python-neutronclient as well.

There should be a class declaration of FloatingIPNotFoundClient in https://github.com/openstack/python-neutronclient/blob/master/neutronclient/common/exceptions.py.

Then heat can handle this as a known not-found exception here:

http://git.openstack.org/cgit/openstack/heat/tree/heat/engine/clients/os/neutron.py#n51

[Gaëtan Trellu (goldyfruit)]

@skraynev Sorry, i missed to give you those informations... I'm running with IceHouse on RHEL 7 + OSP5

Thanks for your help guys !!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/130567

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130567
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=60cbd5bfc424ddc7586048ba8aadb3ef911d9f1a
Submitter: Jenkins
Branch: stable/icehouse

commit 60cbd5bfc424ddc7586048ba8aadb3ef911d9f1a
Author: Peter Razumovsky <email address hidden>
Date: Thu Oct 23 18:53:40 2014 +0400

    Catch NotFound error during loadbalancer deleting

    There is a situation, when we request resource data
    from db during loadbalancer resource deleting. If
    there is no such data, NotFound exception raised and
    stack delete failed with this error. To prevent this
    behaviour, there is a need to catch NotFound exception
    in delete method of loadbalancer.

    Change-Id: Ie52bc3059e91bd25effc53bbd255bce6948509ac
    Closes-bug: #1381700