Unable to delete a stack because a resource is not found

Bug #1381700 reported by Gaëtan Trellu on 2014-10-15
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Invalid
High
Peter Razumovsky
Icehouse
Fix Released
High
Peter Razumovsky

Bug Description

Hi,

I tried to delete a stack but when i did it a get nothing else than: DELETE_FAILED
The only (bad) way to delete the stack was to delete the entry in the heat database...

May be could we add something to the code to avoid a failed if the resource is not found ?

Please find below the stacktrace:

REQ: curl -i https://ca.prospere.com:9696//v2.0/floatingips/7604b118-6ba9-480f-bc97-5d275642918e.json -X DELETE -H "X-Auth-Token: a20cefe401624de4b643dd6ab8f665c3" -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-neutronclient"
 http_log_req /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:173

==> api-cfn.log <==
8065: DEBUG heat.api.middleware.version_negotiation [-] Processing request: OPTIONS / Accept: process_request /usr/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:53

==> engine.log <==
7955: DEBUG neutronclient.client [-] RESP:{'date': 'Wed, 15 Oct 2014 18:47:55 GMT', 'status': '404', 'content-length': '144', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-5936de6a-0db7-4c30-848a-5cffe747d658'} {"NeutronError": {"message": "Floating IP 7604b118-6ba9-480f-bc97-5d275642918e could not be found", "type": "FloatingIPNotFound", "detail": ""}}
 http_log_resp /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:179
7955: DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": {"message": "Floating IP 7604b118-6ba9-480f-bc97-5d275642918e could not be found", "type": "FloatingIPNotFound", "detail": ""}} _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:1125
7955: DEBUG heat.engine.scheduler [-] Task destroy from None starting start /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:153
7955: DEBUG heat.engine.scheduler [-] Task destroy from None running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:186
7955: INFO heat.engine.resource [-] deleting LoadBalancer "my_lb" Stack "stack01" [46a0a37d-6aa1-49ae-a5d1-bc822005e47b]
7955: ERROR heat.engine.resource [-] Delete LoadBalancer "my_lb" Stack "stack01" [46a0a37d-6aa1-49ae-a5d1-bc822005e47b]
7955: TRACE heat.engine.resource Traceback (most recent call last):
7955: TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 707, in delete
7955: TRACE heat.engine.resource handle_data = self.handle_delete()
7955: TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/neutron/loadbalancer.py", line 617, in handle_delete
7955: TRACE heat.engine.resource member_id = db_api.resource_data_get(self, member)
7955: TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/db/api.py", line 62, in resource_data_get
7955: TRACE heat.engine.resource return IMPL.resource_data_get(resource, key)
7955: TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 157, in resource_data_get
7955: TRACE heat.engine.resource key)
7955: TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 188, in resource_data_get_by_key
7955: TRACE heat.engine.resource raise exception.NotFound(_('No resource data found'))
7955: TRACE heat.engine.resource NotFound: No resource data found
7955: TRACE heat.engine.resource
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.scheduler [-] Task destroy from None cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
7955: DEBUG heat.engine.stack_lock [-] Engine 241047ea-f6bf-46a8-85e9-6cfcac9828d4 released lock on stack 46a0a37d-6aa1-49ae-a5d1-bc822005e47b release /usr/lib/python2.7/site-packages/heat/engine/stack_lock.py:122

description: updated
Sergey Kraynev (skraynev) wrote :

@goldyfruit: it's look strange, because fix for this is presented more then 1 year ago and now it looks like [1]

[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/neutron/floatingip.py#L160-L162

Could you you provide information about what release of OpenStack do you use?

Qiming Teng (tengqim) wrote :

Seems that this is related to python-neutronclient as well.

There should be a class declaration of FloatingIPNotFoundClient in https://github.com/openstack/python-neutronclient/blob/master/neutronclient/common/exceptions.py.

Then heat can handle this as a known not-found exception here:

http://git.openstack.org/cgit/openstack/heat/tree/heat/engine/clients/os/neutron.py#n51

Changed in heat:
status: New → Confirmed
Changed in heat:
assignee: nobody → Razumovsky Peter (prazumovsky)
Gaëtan Trellu (goldyfruit) wrote :

@skraynev Sorry, i missed to give you those informations... I'm running with IceHouse on RHEL 7 + OSP5

Thanks for your help guys !!

Changed in heat:
importance: Undecided → High
status: Confirmed → Invalid

Reviewed: https://review.openstack.org/130567
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=60cbd5bfc424ddc7586048ba8aadb3ef911d9f1a
Submitter: Jenkins
Branch: stable/icehouse

commit 60cbd5bfc424ddc7586048ba8aadb3ef911d9f1a
Author: Peter Razumovsky <email address hidden>
Date: Thu Oct 23 18:53:40 2014 +0400

    Catch NotFound error during loadbalancer deleting

    There is a situation, when we request resource data
    from db during loadbalancer resource deleting. If
    there is no such data, NotFound exception raised and
    stack delete failed with this error. To prevent this
    behaviour, there is a need to catch NotFound exception
    in delete method of loadbalancer.

    Change-Id: Ie52bc3059e91bd25effc53bbd255bce6948509ac
    Closes-bug: #1381700

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers