Rebuilding keystone[0] container breaks credential keys

[Logan V]

In the process of rebuilding the keystone[0] container for my env I seem to have ended up in an unrecoverable situation where the credential keys are lost.

What I'm seeing is os-keystone-install failing with:
TASK [os_keystone : include] ***************************************************
included: /etc/ansible/roles/os_keystone/tasks/keystone_credential.yml for lsn-mc1008_keystone_container-a7b9e4c0, lsn-mc1007_keystone_container-d1108af0, lsn-mc1006_keystone_container-7b5f3a0e

TASK [os_keystone : include] ***************************************************
included: /etc/ansible/roles/os_keystone/tasks/keystone_credential_create.yml for lsn-mc1008_keystone_container-a7b9e4c0

TASK [os_keystone : Check if credential keys already exist] ********************
ok: [lsn-mc1008_keystone_container-a7b9e4c0]

TASK [os_keystone : Create credential keys for Keystone] ***********************

TASK [os_keystone : Ensure newest key is used for credential in Keystone] ******
fatal: [lsn-mc1008_keystone_container-a7b9e4c0]: FAILED! => {"changed": true, "cmd": ["/openstack/venvs/keystone-14.0.8/bin/keystone-manage", "credential_migrate", "--keystone-user", "keystone", "--keystone-group", "keystone"], "delta": "0:00:01.664529", "end": "2017-02-25 12:02:28.834966", "failed": true, "rc": 1, "start": "2017-02-25 12:02:27.170437", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}

Checking the container's keystone.log, I see:
2017-02-25 12:07:32.951 1866 ERROR keystone.credential.providers.fernet.core [-] Credential could not be decrypted. Please contact the administrator
2017-02-25 12:07:32.952 1866 CRITICAL keystone [-] CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator
2017-02-25 12:07:32.952 1866 ERROR keystone Traceback (most recent call last):
2017-02-25 12:07:32.952 1866 ERROR keystone File "/openstack/venvs/keystone-14.0.8/bin/keystone-manage", line 11, in <module>
2017-02-25 12:07:32.952 1866 ERROR keystone sys.exit(main())
2017-02-25 12:07:32.952 1866 ERROR keystone File "/openstack/venvs/keystone-14.0.8/lib/python2.7/site-packages/keystone/cmd/manage.py", line 44, in main
2017-02-25 12:07:32.952 1866 ERROR keystone cli.main(argv=sys.argv, config_files=config_files)
2017-02-25 12:07:32.952 1866 ERROR keystone File "/openstack/venvs/keystone-14.0.8/lib/python2.7/site-packages/keystone/cmd/cli.py", line 1270, in main
2017-02-25 12:07:32.952 1866 ERROR keystone CONF.command.cmd_class.main()
2017-02-25 12:07:32.952 1866 ERROR keystone File "/openstack/venvs/keystone-14.0.8/lib/python2.7/site-packages/keystone/cmd/cli.py", line 769, in main
2017-02-25 12:07:32.952 1866 ERROR keystone klass.migrate_credentials()
2017-02-25 12:07:32.952 1866 ERROR keystone File "/openstack/venvs/keystone-14.0.8/lib/python2.7/site-packages/keystone/cmd/cli.py", line 751, in migrate_credentials
2017-02-25 12:07:32.952 1866 ERROR keystone credential['encrypted_blob']
2017-02-25 12:07:32.952 1866 ERROR keystone File "/openstack/venvs/keystone-14.0.8/lib/python2.7/site-packages/keystone/credential/providers/fernet/core.py", line 107, in decrypt
2017-02-25 12:07:32.952 1866 ERROR keystone raise exception.CredentialEncryptionError(msg)
2017-02-25 12:07:32.952 1866 ERROR keystone CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator

All 3 containers in the env appear to have synced up /etc/keystone/credential-keys/ contents, and it seems that none of the keys in the directory work for decryption.

I'm not sure how this situation was reached other than I rebuilt the keystone[0] container, os-keystone-install ran successfully, and now it won't run at all in subsequent runs. Going to work on trying to reproduce this.

[Logan V (loganv)]

From a backup of the keystone[0] container taken immediately prior to the rebuild, I was able to confirm:

1) The keys in /etc/keystone/credential-keys on the old container were completely different than the ones on the rebuilt container and running env.
2) By dropping the backed up credential-keys from the old container into the new container I was able to run credential_migrate successfully.

[Logan V (loganv)]

https://github.com/logan2211/openstack-ansible-overlay/tree/bug-1667960 reliably reproduces this bug.

Basically, build AIO with keystone_container affinity 3, destroy keystone_all[0], rebuild, and then run the role twice. The first run will recreate the credential keys since they do not exist on the container, and then it will sync the new keys to the other containers (overwriting the old keys) without performing any migration from the old keys first. Then all subsequent runs of the keystone role will fail because the credentials keys have been permanently lost.

This file change outlines the workflow needed to test this:
https://github.com/Logan2211/openstack-ansible-overlay/commit/c023e27d15023becb5cc972cad40d9980e05b031#diff-93518fda10b0403d3c5c20b4df4740a6

[Logan V (loganv)]

Logs from a jenkins run of the above commit: http://jenkins.protiumit.com/jenkins-openstack-ansible-overlay-xenial-functional-bug-1667960-9/logs/

[Logan V (loganv)]

I have not checked the tasks yet, but I suspect this same behavior exists in our fernet key distribution tasks as well. However the impact there should be much lower because it will only result in invalidated tokens, not complete data loss.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/441264
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=b3425781ecb3dd5082955fa8bb199245d47ddf2e
Submitter: Jenkins
Branch: master

commit b3425781ecb3dd5082955fa8bb199245d47ddf2e
Author: Logan V <email address hidden>
Date: Fri Mar 3 12:03:05 2017 -0600

    Rebuild credential-key repo during keystone[0] rebuild

    When the first Keystone container is rebult in an existing environment,
    the credential key repository is overwritten with new keys and the
    existing keys are overwritten on the other infrastructure hosts without
    any migration taking place. This results in an irrevocable loss of
    the keys used to encrypt the credentials.

    Now we will collect keys from any existing credential keys on the other
    containers and use them to rebuild the credential-key repo on the primary
    container before performing a key migration and rotation.

    If no keys are found on the other containers, we will perform a
    credential_setup on the primary container and sync the keys, just
    as we would have before.

    Closes-Bug: #1667960
    Change-Id: Ic616d397574573629273838fbf68ea3f6bdb0468

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/444938

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/444939

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/444939
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=637b2e3526444815c795dbc3c8f2847559c13936
Submitter: Jenkins
Branch: stable/newton

commit 637b2e3526444815c795dbc3c8f2847559c13936
Author: Logan V <email address hidden>
Date: Fri Mar 3 12:03:05 2017 -0600

    Rebuild credential-key repo during keystone[0] rebuild

    When the first Keystone container is rebult in an existing environment,
    the credential key repository is overwritten with new keys and the
    existing keys are overwritten on the other infrastructure hosts without
    any migration taking place. This results in an irrevocable loss of
    the keys used to encrypt the credentials.

    Now we will collect keys from any existing credential keys on the other
    containers and use them to rebuild the credential-key repo on the primary
    container before performing a key migration and rotation.

    If no keys are found on the other containers, we will perform a
    credential_setup on the primary container and sync the keys, just
    as we would have before.

    Closes-Bug: #1667960
    Change-Id: Ic616d397574573629273838fbf68ea3f6bdb0468
    (cherry picked from commit b3425781ecb3dd5082955fa8bb199245d47ddf2e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/444938
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=b6234afb19cd7e629caccc088f6a67295b092f0a
Submitter: Jenkins
Branch: stable/ocata

commit b6234afb19cd7e629caccc088f6a67295b092f0a
Author: Logan V <email address hidden>
Date: Fri Mar 3 12:03:05 2017 -0600

    Rebuild credential-key repo during keystone[0] rebuild

    When the first Keystone container is rebult in an existing environment,
    the credential key repository is overwritten with new keys and the
    existing keys are overwritten on the other infrastructure hosts without
    any migration taking place. This results in an irrevocable loss of
    the keys used to encrypt the credentials.

    Now we will collect keys from any existing credential keys on the other
    containers and use them to rebuild the credential-key repo on the primary
    container before performing a key migration and rotation.

    If no keys are found on the other containers, we will perform a
    credential_setup on the primary container and sync the keys, just
    as we would have before.

    Closes-Bug: #1667960
    Change-Id: Ic616d397574573629273838fbf68ea3f6bdb0468
    (cherry picked from commit b3425781ecb3dd5082955fa8bb199245d47ddf2e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_keystone 16.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_keystone 14.2.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_keystone 15.1.2 release.