OpenStack Heat

Stack DELETE FAILED (xxx): Error deleting project: Could not find project: b3b992f748ef4814a2fc10a6a0dc6c02 (HTTP 404)

Bug #1410698 reported by TonyWang
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Undecided
Ethan Lynn
OpenStack Heat 2015.1.0 "kilo"
Juno
Fix Released
Undecided
Steven Hardy
OpenStack Heat 2014.2.3

Bug Description

This may be a regression bug of bug 1365332:
https://bugs.launchpad.net/heat/+bug/1365332

I deleted a stack, and failed because of the authentication issue:
2015-01-12 16:07:10.346 6147 INFO heat.engine.stack [-] Stack DELETE COMPLETE (xxx): Stack DELETE completed successfully
2015-01-12 16:07:11.454 6147 ERROR heat.engine.stack [-] A user and password or token is required.
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack Traceback (most recent call last):
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 915, in delete
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack trust_id)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/common/heat_keystoneclient.py", line 300, in delete_trust
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack self.client.trusts.delete(trust_id)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/v3/contrib/trusts.py", line 94, in delete
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack return super(TrustManager, self).delete(trust_id=base.getid(trust))
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 71, in func
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack return f(*args, **new_kwargs)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 373, in delete
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack self.build_url(dict_args_in_out=kwargs))
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 209, in _delete
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack return self.client.delete(url, **kwargs)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 644, in delete
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack return self._cs_request(url, 'DELETE', **kwargs)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 621, in _cs_request
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack return self.request(url, method, **kwargs)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 596, in request
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack resp = super(HTTPClient, self).request(url, method, **kwargs)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/baseclient.py", line 21, in request
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack return self.session.request(url, method, **kwargs)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack return func(*args, **kwargs)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 266, in request
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack token = self.get_token(auth)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 506, in get_token
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack return auth.get_token(self)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 275, in get_token
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack return self.auth_token
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 283, in auth_token
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack self.authenticate()
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack return func(*args, **kwargs)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 425, in authenticate
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack resp = self.get_raw_token_from_identity_service(**kwargs)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/v3/client.py", line 254, in get_raw_token_from_identity_service
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack raise exceptions.AuthorizationFailure(msg)
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack AuthorizationFailure: A user and password or token is required.
2015-01-12 16:07:11.454 6147 TRACE heat.engine.stack

Although the trust was failed to delete, the stack_domain_project was deleted successfully:
"/usr/lib/python2.7/site-packages/heat/engine/stack.py"
Stack.delete
            if self.user_creds_id:
                user_creds = db_api.user_creds_get(self.user_creds_id)
                # If we created a trust, delete it
                if user_creds is not None:
                    trust_id = user_creds.get('trust_id')
                    if trust_id:
                        try:
                            # If the trustor doesn't match the context user the
                            # we have to use the stored context to cleanup the
                            # trust, as although the user evidently has
                            # permission to delete the stack, they don't have
                            # rights to delete the trust unless an admin
                            trustor_id = user_creds.get('trustor_user_id')
                            if self.context.user_id != trustor_id:
                                LOG.debug('Context user_id doesn\'t match '
                                          'trustor, using stored context')
                                sc = self.stored_context()
                                sc.clients.client('keystone').delete_trust(
                                    trust_id)
                            else:
                                self.clients.client('keystone').delete_trust(
                                    trust_id) <<<<<Failed to delete the trust, but there is an following "except".
                        except Exception as ex:
                            LOG.exception(ex)
                            stack_status = self.FAILED
                            reason = ("Error deleting trust: %s" %
                                      six.text_type(ex))

...

            # If the stack has a domain project, delete it
            if self.stack_user_project_id and not abandon:
                try:
                    keystone = self.clients.client('keystone')
                    keystone.delete_stack_domain_project(
                        project_id=self.stack_user_project_id) <<<<<Deleted the stack_domain_project.
                except Exception as ex:
                    LOG.exception(ex)
                    stack_status = self.FAILED
                    reason = "Error deleting project: %s" % six.text_type(ex)

Then I deleted the stack again, and failed by the following reason:
2015-01-14 15:28:23.056 6147 ERROR heat.engine.stack [-] Could not find project: b3b992f748ef4814a2fc10a6a0dc6c02 (HTTP 404)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack Traceback (most recent call last):
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 942, in delete
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack project_id=self.stack_user_project_id)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/common/heat_keystoneclient.py", line 506, in delete_stack_domain_project
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack project = self.domain_admin_client.projects.get(project=project_id)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/v3/projects.py", line 83, in get
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack project_id=base.getid(project))
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 71, in func
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack return f(*args, **new_kwargs)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 333, in get
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack self.key)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 135, in _get
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack resp, body = self.client.get(url, **kwargs)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 629, in get
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack return self._cs_request(url, 'GET', **kwargs)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 621, in _cs_request
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack return self.request(url, method, **kwargs)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 596, in request
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack resp = super(HTTPClient, self).request(url, method, **kwargs)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/baseclient.py", line 21, in request
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack return self.session.request(url, method, **kwargs)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack return func(*args, **kwargs)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 346, in request
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack raise exceptions.from_response(resp, method, url)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack NotFound: Could not find project: b3b992f748ef4814a2fc10a6a0dc6c02 (HTTP 404)
2015-01-14 15:28:23.056 6147 TRACE heat.engine.stack
2015-01-14 15:28:23.072 6147 INFO heat.engine.stack [-] Stack DELETE FAILED (xxx): Error deleting project: Could not find project: b3b992f748ef4814a2fc10a6a0dc6c02 (HTTP 404)

The stack can't be deleted by this error.
I think the fix of bug 1308812 can fix this issue:
https://bugs.launchpad.net/heat/+bug/1308812

When heat delete a stack_domain_project, it should pass if the project hadn't been existed.
The fix of bug 1365332 doesn't ignore exception kc_exception.NotFound when heat fails to get the project before the delete action.

See original description
TonyWang (tw-nova)
description: updated
Revision history for this message
Marcin Zbik (zbikmarc+launchpad) wrote :
Download full text (6.9 KiB)

I confirm that behavior.
In first try stack is not deleted due to:

2015-01-21 11:52:25.153 23101 DEBUG heat.engine.stack [-] Context user_id doesn't match trustor, using stored context _delete_credentials /usr/lib/python2.7/site-packages/heat/engine/stack.py:904
2015-01-21 11:52:25.158 23101 DEBUG keystoneclient.auth.identity.v3 [req-e37aec2d-9ceb-47de-bb7a-7f4e91224758 ] Making authentication request to http://10.20.71.252:5000/v3/auth/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3.py:117
2015-01-21 11:52:25.178 23101 DEBUG keystoneclient.session [req-e37aec2d-9ceb-47de-bb7a-7f4e91224758 ] Request returned failure status: 401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:353
2015-01-21 11:52:25.179 23101 DEBUG keystoneclient.v3.client [req-e37aec2d-9ceb-47de-bb7a-7f4e91224758 ] Authorization failed. get_raw_token_from_identity_service /usr/lib/python2.7/site-packages/keystoneclient/v3/client.py:268
2015-01-21 11:52:25.179 23101 ERROR heat.common.keystoneclient [req-e37aec2d-9ceb-47de-bb7a-7f4e91224758 None] Keystone client authentication failed
2015-01-21 11:52:25.180 23101 ERROR heat.engine.stack [req-e37aec2d-9ceb-47de-bb7a-7f4e91224758 None] Authorization failed.

However it deletes projects:
2015-01-21 11:52:25.233 23101 DEBUG keystoneclient.session [req-e37aec2d-9ceb-47de-bb7a-7f4e91224758 ] REQ: curl -i -X DELETE http://10.20.71.252:5000/v3/projects/575ca8909fa04ddd863fc5e3603ea5f2 -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: {SHA1}dda52cd0e0b79ce391720850ec07e93ea82ad321" _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:162
2015-01-21 11:52:25.280 23101 DEBUG keystoneclient.session [req-e37aec2d-9ceb-47de-bb7a-7f4e91224758 ] RESP: [204] date: Wed, 21 Jan 2015 10:51:12 GMT content-length: 0 vary: X-Auth-Token _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:190

And just after this:

2015-01-21 11:52:25.281 23101 DEBUG keystoneclient.httpclient [req-e37aec2d-9ceb-47de-bb7a-7f4e91224758 ] No body was returned. _decode_body /usr/lib/python2.7/site-packages/keystoneclient/httpclient.py:578
2015-01-21 11:52:25.285 23101 INFO heat.engine.stack [req-e37aec2d-9ceb-47de-bb7a-7f4e91224758 None] Stack DELETE FAILED (ubuntu_14b0bd88d0f): Error deleting trust: Authorization failed.
2015-01-21 11:52:25.291 23101 DEBUG heat.engine.stack_lock [req-e37aec2d-9ceb-47de-bb7a-7f4e91224758 None] Engine fc9cd09a-1d12-4f23-8deb-134d12e3e4b9 released lock on stack 454d3449-0a9d-4caf-a9ea-044d039f44b8 release /usr/lib/python2.7/site-packages/heat/engine/stack_lock.py:123

So I try to delete again and:

2015-01-21 12:01:40.241 23101 DEBUG keystoneclient.auth.identity.v3 [-] Making authentication request to http://10.20.71.252:5000/v3/auth/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3.py:117
2015-01-21 12:01:40.370 23101 DEBUG keystoneclient.session [-] REQ: curl -i -X GET http://10.20.71.252:5000/v3/projects/575ca8909fa04ddd863fc5e3603ea5f2 -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: {SHA1}ee2dce50712deea0bb573cdd996d7a225eafdd96" _http_log_request /usr/lib/python2.7/site-packages/k...

Read more...

Revision history for this message
Ethan Lynn (ethanlynn) wrote :

What version of openstack are you using?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/152406

Changed in heat:
assignee: nobody → Ethan Lynn (ethanlynn)
status: New → In Progress
Revision history for this message
TonyWang (tw-nova) wrote :

I used Juno version

Revision history for this message
Marcin Zbik (zbikmarc+launchpad) wrote :

Proposed fix works for me.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/152406
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=6046182dfad858349fa647b8777657f535a629c7
Submitter: Jenkins
Branch: master

commit 6046182dfad858349fa647b8777657f535a629c7
Author: Ethan Lynn <email address hidden>
Date: Tue Feb 3 14:15:40 2015 +0800

    Catch notfound exception when deleting project

    When heat trying to delete a non-exist project, will catch a
    notfound error, original code place the catcher in the wrong
    place, this patch fixes it.

    Change-Id: I68269838d1cef311c3f41f5846deae4ed94aef4c
    Closes-Bug: #1410698

Changed in heat:
status: In Progress → Fix Committed
Steven Hardy (shardy)
tags: added: juno-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/156559

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/juno)

Reviewed: https://review.openstack.org/156559
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=848c59ba1c4978040d82566c0f8667623a04b93f
Submitter: Jenkins
Branch: stable/juno

commit 848c59ba1c4978040d82566c0f8667623a04b93f
Author: Ethan Lynn <email address hidden>
Date: Tue Feb 3 14:15:40 2015 +0800

    Catch notfound exception when deleting project

    When heat trying to delete a non-exist project, will catch a
    notfound error, original code place the catcher in the wrong
    place, this patch fixes it.

    Change-Id: I68269838d1cef311c3f41f5846deae4ed94aef4c
    Closes-Bug: #1410698
    (cherry picked from commit 6046182dfad858349fa647b8777657f535a629c7)

Thierry Carrez (ttx)
Changed in heat:
milestone: none → kilo-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: kilo-3 → 2015.1.0
Zane Bitter (zaneb)
tags: added: in-stable-juno
removed: juno-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.