Heat stack delete failing on invalidToken exception during _delete_credentials

[Drew Freiberger]

We have a site where we are having a failure to delete a heat stack due to an issue with the heat credentials store in the database.

Traceback of the failure looks like this:

2018-06-11 22:45:58.839 267852 INFO heat.engine.stack [-] Stack DELETE FAILED (CBAM-ec2d9576f0e34ed684f0581363f8aa91-990b6cb2-ff67-11e7-8871-fa163ef36821):
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service [-] Unhandled error in asynchronous task
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service Traceback (most recent call last):
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 132, in log_exceptions
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service gt.wait()
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service return self._exit_event.wait()
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service current.throw(*self._exc)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service result = function(*args, **kwargs)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 123, in _start_with_trace
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service return func(*args, **kwargs)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 117, in wrapper
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service return f(*args, **kwargs)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 86, in handle_exceptions
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service {'func': func.__name__, 'msg': errmsg})
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service self.force_reraise()
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service six.reraise(self.type_, self.value, self.tb)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 81, in handle_exceptions
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service return func(stack, *args, **kwargs)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 1675, in delete
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service abandon)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 1551, in _delete_credentials
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service user_creds = self._try_get_user_creds()
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 1539, in _try_get_user_creds
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service user_creds = ucreds_object.UserCreds.get_by_id(self.user_creds_id)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/objects/user_creds.py", line 71, in get_by_id
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service user_creds_db = db_api.user_creds_get(context_id)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/db/api.py", line 230, in user_creds_get
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service return IMPL.user_creds_get(context_id)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/db/sqlalchemy/api.py", line 685, in user_creds_get
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service db_result.decrypt_method, result['trust_id'])
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/common/crypt.py", line 50, in decrypt
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service value = decryptor(data, encryption_key)
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/heat/common/crypt.py", line 66, in cryptography_decrypt_v1
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service return sym.decrypt(encodeutils.safe_encode(value))
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service File "/usr/lib/python2.7/dist-packages/cryptography/fernet.py", line 101, in decrypt
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service raise InvalidToken
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service InvalidToken
2018-06-11 22:45:58.848 267852 ERROR heat.engine.service

If master branch is to be believed: https://github.com/openstack/heat/blob/master/heat/common/exception.py#L166 this is an error: "Can not decrypt data with the auth_encryption_key in heat config"

While I'm looking through the trace, I can see that there is an exception handler for exception.Error at _try_get_user_creds https://github.com/openstack/heat/blob/master/heat/engine/stack.py#L1760 it is not properly trapping for exception.InvalidToken, and hence, the stack fails to delete.

If this truly is to do with the auth_encryption_key changing/failing to decrypt the value in the database, is it possible this changed somehow due to charm changes?

The cloud is a xenial/Mitaka cloud running 17.02 charms.

[Drew Freiberger (afreiberger)]

It appears that this is not a charm bug, but needs to be run upstream to heat. The stack was created and attempted to be deleted under 48 hours later, and there were no changes to heat configs within that timeframe that would have changed the encryption key, hence, I believe this is bad exception handling within swift.heat.engine.stack._try_get_user_creds() function that should trap also InvalidToken and InvalidEncryptionKey exceptions as well as exception.Error to allow for deletion of the stack.

[Pen Gale (pengale)]

Agree that this is a bug against upstream, rather than the charm.

Marking as "invalid" as the heat project doens't use launchpad, and there is no way to directly point the bug at heat.

[Alex Kavanagh (ajkavanagh)]

Is this upstream bug in heat relevant: https://storyboard.openstack.org/#!/story/1711047

It indicates that it was fixed, but it's not clear which version of Heat is running in the submitter's system.