Error deleting stack, deleting trust fails

[Thomas Herve]

Seen here: http://logs.openstack.org/15/374115/1/check/gate-tempest-dsvm-heat-apache/ffef527/

Error deleting trust
Traceback (most recent call last):
  File "/opt/stack/new/heat/heat/engine/stack.py", line 1658, in _delete_credentials
    trust_id)
  File "/opt/stack/new/heat/heat/engine/clients/os/keystone/heat_keystoneclient.py", line 231, in delete_trust
    self.client.trusts.delete(trust_id)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/v3/contrib/trusts.py", line 97, in delete
    return super(TrustManager, self).delete(trust_id=base.getid(trust))
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 74, in func
    return f(*args, **new_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 415, in delete
    self.build_url(dict_args_in_out=kwargs))
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 218, in _delete
    return self.client.delete(url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 202, in delete
    return self.request(url, 'DELETE', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 344, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 112, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 570, in request
    raise exceptions.from_response(resp, method, url)
Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-752da2b8-713b-47a2-a256-76aaa3990726)

Deleting trust fails because of authentication issue apparently.

[venkatamahesh (venkatamaheshkotha)]

Hi,

Now the jenkins got success : http://logs.openstack.org/15/374115/1/check/gate-tempest-dsvm-heat-apache/2344e40/console.html

[Thomas Herve (therve)]

Yeah, that's an intermittent failure.

[Rabi Mishra (rabi)]

Looks like a race condition with multiple DELETE requests(delete in the test and test teardown) like bug #1626173 as the stack is already in DELETE_COMPLETE.

It seems to me that the user is already deleted by the earlier request/thread, as I can see invalid user token and subsequent authorization failure in keystone logs[1] when deleting the trust.

IMO, https://review.openstack.org/#/c/374440/ should fix this too.

[1] http://logs.openstack.org/15/374115/1/check/gate-tempest-dsvm-heat-apache/ffef527/logs/apache/keystone.txt.gz#_2016-09-21_13_13_45_993

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/374440
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=e1f161a19a9eef3edb05612aad905e2e7ae7c674
Submitter: Jenkins
Branch: master

commit e1f161a19a9eef3edb05612aad905e2e7ae7c674
Author: Crag Wolfe <email address hidden>
Date: Wed Sep 21 19:06:36 2016 -0400

    Do not attempt deletion of a DELETE_COMPLETE stack in service api

    A stack may be in transient state where it is DELETE_COMPLETE, but has
    has not actually been soft-deleted yet. For the purposes of
    delete_stack in service.py, consider a DELETE_COMPLETE stack as
    equivalent to a soft-deleted one (it soon will be), thereby avoiding a
    race where we would have attempted to update the stack, running into a
    foreign-key constraint issue for a non-existing user_cred.

    Change-Id: Iec021e6a0df262d447fdf9ee1789603c7a1c55f8
    Closes-Bug: #1626173
    Closes-Bug: #1626107

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/374918

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/374918
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=fb26ec9da5ee6fd051d3d0b39588972ecc86f1d9
Submitter: Jenkins
Branch: stable/newton

commit fb26ec9da5ee6fd051d3d0b39588972ecc86f1d9
Author: Crag Wolfe <email address hidden>
Date: Wed Sep 21 19:06:36 2016 -0400

    Do not attempt deletion of a DELETE_COMPLETE stack in service api

    A stack may be in transient state where it is DELETE_COMPLETE, but has
    has not actually been soft-deleted yet. For the purposes of
    delete_stack in service.py, consider a DELETE_COMPLETE stack as
    equivalent to a soft-deleted one (it soon will be), thereby avoiding a
    race where we would have attempted to update the stack, running into a
    foreign-key constraint issue for a non-existing user_cred.

    Change-Id: Iec021e6a0df262d447fdf9ee1789603c7a1c55f8
    Closes-Bug: #1626173
    Closes-Bug: #1626107
    (cherry picked from commit e1f161a19a9eef3edb05612aad905e2e7ae7c674)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 7.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 7.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 8.0.0.0b1 development milestone.