Fernet rotate doesn't prevent rotation when disk is full
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Identity (keystone) |
Fix Released
|
Low
|
John Lin |
Bug Description
When the root partition on any control node is full, the Fernet key on all control nodes will be empty. This will cause Keystone cannot auth anyone (500 Internal Server Error). Is that caused by Fernet key rotation?
When I check the files in /etc/keystone/
root@control1:
total 40
drwxr-s--- 2 keystone keystone 4096 Nov 17 00:00 ./
drwxr-xr-x 5 keystone keystone 4096 Nov 10 11:24 ../
-rw------- 1 keystone keystone 0 Nov 17 00:00 0
-rw------- 1 keystone keystone 44 Nov 16 13:57 10
-rw------- 1 keystone keystone 44 Nov 9 00:00 3
-rw------- 1 keystone keystone 44 Nov 10 00:00 4
-rw------- 1 keystone keystone 44 Nov 11 00:00 5
-rw------- 1 keystone keystone 44 Nov 12 00:00 6
-rw------- 1 keystone keystone 44 Nov 13 00:00 7
-rw------- 1 keystone keystone 44 Nov 14 00:00 8
-rw------- 1 keystone keystone 44 Nov 15 00:00 9
Here is some of the Keystone logs when the master Fernet token is empty.
[req-37cfe30f-
Traceback (most recent call last):
File "/openstack/
result = method(context, **params)
File "/openstack/
parent_
File "/openstack/
__ret_val = __f(*args, **kwargs)
File "/openstack/
parent_
File "/openstack/
*args, **kwargs)
File "/openstack/
token_id = self._get_
File "/openstack/
access_
File "/openstack/
token = self.pack(
File "/openstack/
return self.crypto.
File "/openstack/
fernet_
File "/openstack/
"Fernet key must be 32 url-safe base64-encoded bytes."
ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
summary: |
- Root partition full on any control node makes master Fernet keys empty - on all nodes + Fernet rotate doesn't prevent rotation when disk is full |
Changed in keystone: | |
assignee: | nobody → John Lin (johnlinp) |
Changed in keystone: | |
status: | Triaged → In Progress |
Changed in keystone: | |
milestone: | none → ocata-3 |
Changed in keystone: | |
importance: | Wishlist → Low |
Hi John,
Would you be able to check the contents of the key? It looks like the staged key, or the zero key (0), doesn't contain *anything*. As a result, the MultiFernet object is unable to initialize itself. The error being reported by keystone is actually coming from the fernet library [0].
How many max_active_keys are you using? More than 11? Are there any keys in your key repository that you know are stale and you can remove?
[0] https:/ /github. com/pyca/ cryptography/ blob/7ff4c8fe24 00244a2e6e2a86b c015b2025907b5a /src/cryptograp hy/fernet. py#L35- L38