Delete stack can get stuck due to uncaught exception

[Ken Thomas]

Observed in Icehouse.

We use some an internal user validation infrastructure which returns cookies rather than password. When the base64 encoded cookie value is longer than heat user_creds.password column, the delete stack process will throw an uncaught/unlogged expection. (Note that the line numbers in heat/engine/parser.py may be off because I added some addition logging statement.

ERROR Incorrect padding
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/heat/engine/parser.py", line 670, in delete
    user_creds = db_api.user_creds_get(self.user_creds_id)
  File "/usr/lib/python2.6/site-packages/heat/db/api.py", line 170, in user_creds_get
    return IMPL.user_creds_get(context_id)
  File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py", line 468, in user_creds_get
    result['password'] = _decrypt(result['password'], db_result.decrypt_method)
  File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py", line 174, in _decrypt
    value = decryptor(enc_value)
  File "/usr/lib/python2.6/site-packages/heat/common/crypt.py", line 48, in oslo_decrypt_v1
    auth_info, b64decode=True)
  File "/usr/lib/python2.6/site-packages/heat/openstack/common/crypto/utils.py", line 158, in decrypt
    msg = base64.b64decode(msg)
  File "/usr/lib64/python2.6/base64.py", line 76, in b64decode
    raise TypeError(msg)
TypeError: Incorrect padding

The encoded value gets truncated when stored in the table and is invalid when it's pulled out again. This can cause a delete stack to get stuck in a DELETE_IN_PROGRESS state without any any indication that something has gone wrong. Increasing the size of the password column fixes the problem. I bumped it up to 1024 in my testing since my particular cookie value was over 800 characters after the b64 encoding.

I understand if our situation is a bit pathological since y'all probably don't expect 'passwords' to be that long. Since our internal cloud has to operate withing our company guidelines, there's not a lot we can do about that.

[Zane Bitter (zaneb)]

At a minimum we should:

- Handle the error properly, so the stack doesn't get stuck in DELETE_IN_PROGRESS
- Detect when the password will be truncated on storage and at least log an error

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/137771

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/139299

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/137771
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=0d9af5c6833e26532bf798ebd0756e93d674b7fc
Submitter: Jenkins
Branch: master

commit 0d9af5c6833e26532bf798ebd0756e93d674b7fc
Author: tengqm <email address hidden>
Date: Fri Nov 28 19:37:13 2014 +0800

    Check if encoded user credential may get truncated

    In some deployments, user passwords are generated by other
    authentication infrastructures where the length of the Base64 encoded
    credentials is longer than the DB limit.

    This patch checks if the encoded credential might be truncated when
    it is generated and fail early during stack creation, or else it will
    cause troubles later on when operating the stack.

    Change-Id: I901b9e39dd0d394991520fea1f527e250c2c9680
    Partial-Bug: 1386213

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/139299
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9698da68040b0e3ac63c23e72f31d9739c76e03e
Submitter: Jenkins
Branch: master

commit 9698da68040b0e3ac63c23e72f31d9739c76e03e
Author: tengqm <email address hidden>
Date: Fri Dec 5 09:39:16 2014 +0800

    Don't block stack deletion if user creds fail

    In some deployments, user passwords are generated by other
    authentication infrastructures where the length of the Base64 encoded
    credentials is longer than the DB limit. When this incorrect behavior
    happens silently, the user credential can not be retrieved.
    In this case, stack deletion will abort due to uncaught exception.

    This patch enables a stack deletion to handle this error so that stack
    can be properly deleted.

    Change-Id: Iaf80b18a83c7752465621c232335dcf32afbcbf3
    Closes-Bug: 1386213