Stacks can get stuck DELETE_COMPLETE

[Steven Hardy]

First reported here:
https://bugzilla.redhat.com/show_bug.cgi?id=1021989

I've just reproduced:

2013-11-01 17:06:12.298 22929 DEBUG requests.packages.urllib3.connectionpool [-] "DELETE /v3/OS-TRUST/trusts/75e16ba4d55a43e9ac9761b3fb34a83f HTTP/1.1" 403 116 _make_request /usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:344
2013-11-01 17:06:12.299 22929 DEBUG keystoneclient.httpclient [-] Request returned failure status: 403 request /usr/lib/python2.7/site-packages/keystoneclient/httpclient.py:123
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/heat/engine/parser.py", line 574, in delete
    self.clients.keystone().delete_trust(trust_id)
  File "/usr/lib/python2.7/site-packages/heat/common/heat_keystoneclient.py", line 209, in delete_trust
    self.client_v3.trusts.delete(trust_id)
  File "/usr/lib/python2.7/site-packages/keystoneclient/v3/contrib/trusts.py", line 86, in delete
    return super(TrustManager, self).delete(trust_id=base.getid(trust))
  File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 70, in func
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 361, in delete
    self.build_url(dict_args_in_out=kwargs))
  File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 201, in _delete
    return self.client.delete(url)
  File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 670, in delete
    return self._cs_request(url, 'DELETE', **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 651, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 610, in request
    **request_kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 124, in request
    raise exceptions.from_response(resp, method, url)
Forbidden: You are not authorized to perform the requested action. (HTTP 403)
Removing descriptor: 14

Results in:
$ heat list
+--------------------------------------+---------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+---------------+--------------------+----------------------+
| bd4231d8-1746-4223-8624-bf051fc61670 | wordpress_u1 | CREATE_IN_PROGRESS | 2013-11-01T15:31:07Z |
| 102dd603-07d6-4054-b79e-fb96a16926de | wordpress_u11 | DELETE_COMPLETE | 2013-11-01T17:01:13Z |
+--------------------------------------+---------------+--------------------+----------------------+

It seems that this is not specific to trusts, but if anything happens during delete related to keystone resources (a previous report failed deleting a User resource) we can end up setting the state to DELETE_COMPLETE, and not setting the deleted_at field in the stack table correctly

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/58582

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/58582
Committed: http://github.com/openstack/heat/commit/214ba503757e5bd9bf5a5fab6692c4e94d0536fa
Submitter: Jenkins
Branch: master

commit 214ba503757e5bd9bf5a5fab6692c4e94d0536fa
Author: Steven Hardy <email address hidden>
Date: Wed Nov 27 19:08:27 2013 +0000

    Catch error deleting trust on stack delete

    When deleting a stack, it's possible for deleting the trust to fail,
    for example if the user deleting the stack is not the user who created
    it, or an admin (which raises a Forbidden error), or due to some
    other transient error e.g connection to keystone interrupted.

    Currently in this case, we fail to mark the stack deleted in the DB
    and leave the status "DELETE, COMPLETE", which is misleading.

    Change-Id: Ie8e9ea48bc4f44e56ff4764123fcca733f5bd458
    Partial-Bug: #1247200

[Steven Hardy (shardy)]

So the two reporters of this issue can no longer reproduce, and the only failure I could reproduce is now fixed by the patch above, so marking this Fix Committed.

Ref https://bugzilla.redhat.com/show_bug.cgi?id=1021989

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70679
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=4d213f8b082938ec2587b72b5b920ce211200c46
Submitter: Jenkins
Branch: stable/havana

commit 4d213f8b082938ec2587b72b5b920ce211200c46
Author: Steven Hardy <email address hidden>
Date: Fri Nov 22 15:24:55 2013 +0000

    Improve coverage of storing credentials in parser.Stack

    Currently the trusts path is not directly tested, so add coverage
    and ensure the correct credentials are stored in each case.

    Related-Bug: #1247200
    Conflicts:
     heat/tests/test_parser.py

    (cherry picked from commit 5397e94292dcbf61778bdaf8abdd3c14be728458)
    Change-Id: I0aa999e01015046946f242a9b52b484522dc6d72

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70680
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9279833b8d331392c13a45b563904e18d3a3461e
Submitter: Jenkins
Branch: stable/havana

commit 9279833b8d331392c13a45b563904e18d3a3461e
Author: Steven Hardy <email address hidden>
Date: Fri Nov 22 15:46:36 2013 +0000

    Add coverage for trusts parser.Stack delete path

    Related-Bug: #1247200
    (cherry picked from commit 9904be6febc4acd39fb86afe119aa6427e890b9a)
    Change-Id: Ic55030be389ac71ec999e08533fa9d5fc05b5bd1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70681
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ab5d961efd062662544218f36ae64277d39763fd
Submitter: Jenkins
Branch: stable/havana

commit ab5d961efd062662544218f36ae64277d39763fd
Author: Steven Hardy <email address hidden>
Date: Wed Nov 27 19:08:27 2013 +0000

    Catch error deleting trust on stack delete

    When deleting a stack, it's possible for deleting the trust to fail,
    for example if the user deleting the stack is not the user who created
    it, or an admin (which raises a Forbidden error), or due to some
    other transient error e.g connection to keystone interrupted.

    Currently in this case, we fail to mark the stack deleted in the DB
    and leave the status "DELETE, COMPLETE", which is misleading.

    Conflicts:
     heat/tests/test_parser.py

    Closes-Bug: #1247200
    (cherry picked from commit 214ba503757e5bd9bf5a5fab6692c4e94d0536fa)
    Change-Id: Ie8e9ea48bc4f44e56ff4764123fcca733f5bd458