Credential Encryption breaks deployments without Fernet

[Adam Young]

A recent change to encrypt credetials broke RDO/Tripleo deployments:

2016-09-02 17:16:55.074 17619 ERROR keystone.common.fernet_utils [req-31d60075-7e0e-401e-a93f-58297cd5439b f2caffbaf10d4e3da294c6366fe19a36 fd71b607cfa84539bf0440915ea2d94b - default default] Either [fernet_tokens] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi [req-31d60075-7e0e-401e-a93f-58297cd5439b f2caffbaf10d4e3da294c6366fe19a36 fd71b607cfa84539bf0440915ea2d94b - default default] MultiFernet requires at least one Fernet instance
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi Traceback (most recent call last):
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 225, in __call__
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi result = method(req, **params)
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 164, in inner
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi return f(self, request, *args, **kwargs)
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/controllers.py", line 69, in create_credential
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi ref = self.credential_api.create_credential(ref['id'], ref)
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 124, in wrapped
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs)
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/core.py", line 106, in create_credential
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi credential_copy = self._encrypt_credential(credential)
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/core.py", line 72, in _encrypt_credential
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi json.dumps(credential['blob'])
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/providers/fernet/core.py", line 68, in encrypt
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi crypto, keys = get_multi_fernet_keys()
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/providers/fernet/core.py", line 49, in get_multi_fernet_keys
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi crypto = fernet.MultiFernet(fernet_keys)
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/cryptography/fernet.py", line 128, in __init__
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi "MultiFernet requires at least one Fernet instance"
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi ValueError: MultiFernet requires at least one Fernet instance
2016-09-02 17:16:55.074 17619 ERROR keystone.common.wsgi

[Emilien Macchi (emilienm)]

I'm adding TripleO because we need to automate the process of upgrade regarding:
http://docs.openstack.org/releasenotes/keystone/unreleased.html#upgrade-notes

"Keystone now supports encrypted credentials at rest. In order to upgrade successfully to Newton, deployers must encrypt all credentials currently stored before contracting the database. Deployers must run keystone-manage credential_setup in order to use the credential API within Newton, or finish the upgrade from Mitaka to Newton. This will result in a service outage for the credential API where credentials will be read-only for the duration of the upgrade process. Once the database is contracted credentials will be writeable again. Database contraction phases only apply to rolling upgrades."

So I'm going to try to make it transparent in puppet-keystone but for sure TripleO will have to run the command in the upgrade scripts.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/365087

[Emilien Macchi (emilienm)]

I'm currently working on a way to generate 2 keystone credentials, 0.txt and 1.txt (what would do keystone-manage credential_setup) and create the files with Puppet on all keystone nodes.

It would be a first iteration.
In future, we could implement keys export/collect mechanism with Mistral or whatever tool.
Also rotations.

[Lance Bragstad (lbragstad)]

FWIW

  ERROR keystone.common.wsgi ValueError: MultiFernet requires at least one Fernet instance

isn't a very helpful error message. I can improve it and tag it as a partial bug here (per stevemar's request in the keystone meeting today).

[Emilien Macchi (emilienm)]

This is my work for credentials: https://review.openstack.org/#/q/topic:keystone/credentials

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/366831

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/366832

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/366854

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/366831
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=e9b64378e64e0308a07242ddc38736cc3abd4c2a
Submitter: Jenkins
Branch: master

commit e9b64378e64e0308a07242ddc38736cc3abd4c2a
Author: Lance Bragstad <email address hidden>
Date: Wed Sep 7 14:53:44 2016 +0000

    Introduce null key for credential encryption

    To ease upgrades from Mitaka to Newton, we are introducing the
    concept of a null key to keystone's implementation of credential
    encryption. The null key can be assumed by keystone if no other
    keys exists in the configured `CONF [credential] key_repository`
    and it is a known value, so it doesn't need to be orchestrated
    across nodes in multi-node deployments.

    This allows an operator to upgrade from Mitaka to Newton without
    having to setup a credential key repository beforehand. It is
    strongly recommended that deployers configure their key_repository
    and migrate off of the null key as soon as possible. Since the null
    key is a known value, it is no more secure than storing secrets in
    plain text. It is only here to ease the upgrade process for
    deployers.

    Change-Id: I6cca7e40ce36a8a24dc73f92b22487998da6a1ae
    Related-Bug: 1619758

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/366832
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=59f117f6a87046391e1fc48e40d28a1da5b6cc1d
Submitter: Jenkins
Branch: master

commit 59f117f6a87046391e1fc48e40d28a1da5b6cc1d
Author: Lance Bragstad <email address hidden>
Date: Wed Sep 7 15:33:57 2016 +0000

    Log warning if null key is used for encryption

    The null key doesn't provide any real encryption protection. It only
    provides security through obscurity since the null key is a known
    thing. This commit makes it so we log a warning every time it is
    used for encryption.

    Change-Id: I10e8b6697c3b35c3ae6e8a1cec5e53f0913b42e6
    Related-Bug: 1619758

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/366854
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=407f08ec144fff2d443061ba3cfddf1b4a427799
Submitter: Jenkins
Branch: master

commit 407f08ec144fff2d443061ba3cfddf1b4a427799
Author: Lance Bragstad <email address hidden>
Date: Wed Sep 7 16:15:03 2016 +0000

    Add docs for the null key

    Add information regarding the null key to the encrypted credentials
    documentation.

    Change-Id: Idbf4b1b15c9777b81d2a92d9c2e20a87e3eb6c53
    Closes-Bug: 1619758

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lance Bragstad (<email address hidden>) on branch: master
Review: https://review.openstack.org/365087
Reason: Looks like we can abandon in favor of the null key

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 10.0.0.0rc1 release candidate.