Delete stack can get stuck due to uncaught exception
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Heat |
Fix Released
|
Medium
|
Qiming Teng |
Bug Description
Observed in Icehouse.
We use some an internal user validation infrastructure which returns cookies rather than password. When the base64 encoded cookie value is longer than heat user_creds.password column, the delete stack process will throw an uncaught/unlogged expection. (Note that the line numbers in heat/engine/
ERROR Incorrect padding
Traceback (most recent call last):
File "/usr/lib/
user_creds = db_api.
File "/usr/lib/
return IMPL.user_
File "/usr/lib/
result[
File "/usr/lib/
value = decryptor(
File "/usr/lib/
auth_info, b64decode=True)
File "/usr/lib/
msg = base64.
File "/usr/lib64/
raise TypeError(msg)
TypeError: Incorrect padding
The encoded value gets truncated when stored in the table and is invalid when it's pulled out again. This can cause a delete stack to get stuck in a DELETE_IN_PROGRESS state without any any indication that something has gone wrong. Increasing the size of the password column fixes the problem. I bumped it up to 1024 in my testing since my particular cookie value was over 800 characters after the b64 encoding.
I understand if our situation is a bit pathological since y'all probably don't expect 'passwords' to be that long. Since our internal cloud has to operate withing our company guidelines, there's not a lot we can do about that.
Changed in heat: | |
assignee: | nobody → Qiming Teng (tengqim) |
Changed in heat: | |
milestone: | none → kilo-2 |
status: | Fix Committed → Fix Released |
Changed in heat: | |
milestone: | kilo-2 → 2015.1.0 |
At a minimum we should:
- Handle the error properly, so the stack doesn't get stuck in DELETE_IN_PROGRESS
- Detect when the password will be truncated on storage and at least log an error