Comment 0 for bug 1816927

The implementation for fernet tokens relies on symmetric encryption. This underpinning requires that each keystone API node "share" the same key repository, specifically in deployments where keystone servers need to validate tokens issued by one another (e.g., a cluster of keystone servers behind an HA proxy).

With getting into too much detail, each key repository consists of a set of files on disk. The naming of each file is crucial because it denotes the type of key it is (documented extensively [0]). Each file name corresponds to an integer. The file name with the highest index is used to encrypt new tokens, which is called the primary key. The file name with the lowest index, or 0, is known as a staged key and it is always promoted to be the primary key on the next rotation. Every other key in the repository is a secondary key and is only used to decrypt tokens. Each key on disk goes through a lifecycle, starting as a staged key, promoted to a primary key, eventually being demoted to a secondary key. Note that keystone does *not* handle token distribution between API servers. We recommend this be done using configuration management. The documentation suggests rsync as one possible utility to keep key repositories in sync.

I'm opening this bug because it was brought to our attention that keystone servers may respond with a 401 Invalid Fernet token, in deployments with high churn, or high token load, across a cluster of keystone nodes.

The issue is that in the process of key rotation, the staged key is promoted to be the primary key. As soon as this happens, any subsequent requests to create tokens will use the primary key to encrypt the token. It is assumed all other API servers have a copy of this key, because it's the staged key and also valid as a secondard key. A encrypted with the new primary key should be validatable on other API servers if they have a copy of the staged key, which has the same key contents as the new primary key on the API server that initiated the token rotation. The rsync implementation deletes the contents of the key repository and rebuilds it, alphanumerically. This results in the staged key always being written by rsync first, because its file name is 0. The primary key is always written last, because its filename is the highest index of the key repository.

A unique timing event where:

- a token is created after key rotation, but before key distribution
- key distribution is invoked using a mechanism like rsync
- token validation is performed on the API server getting its key repository built by rsync
- the token is validated before the primary key is written to the key repository by rsync, and fails validation because the key repository doesn't contain the key used to encrypt the token

A subsequent request to validate the token should succeed if rsync completes successfully.

pas-ha brought this to the #openstack-keystone channel as an issue that was affecting an internal CI/CD deployment that has a lot of churn [1].

[0] https://docs.openstack.org/keystone/latest/admin/fernet-token-faq.html#what-are-the-different-types-of-keys
[1] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2019-02-20.log.html#t2019-02-20T20:11:12