keystone-manage bootstrap isn't completely idempotent

[Lance Bragstad]

The keystone-manage bootstrap command was designed to be idempotent. Most everything in the bootstrap command is wrapped with a try/except to handle cases where specific entities already exist (i.e. there is already an admin project or an admin user from a previous bootstrap run). This is important because bootstrap handles the creation of administrator-like things in order to "bootstrap" a deployment. If bootstrap wasn't idempotent, the side-effect of running it multiple times would be catastrophic.

During an upgrade scenario, using OpenStack Ansible's rolling upgrade support [0], from stable/newton to master, I noticed a very specific case where bootstrap was not idempotent. Even if the admin user passed to bootstrap already exists, the command will still attempt to update it's password [1], even if the admin password hasn't changed. It does the same thing with the user's enabled property. This somehow creates a revocation event to be stored for that specific user [2]. As a result, all tokens for the user specified in the bootstrap command will be invalid once the upgrade happens, since OpenStack Ansible relies on `keystone-manage bootstrap` during the upgrade.

This only affects the bootstrap user, but it can be considered a service interruption since it is being done during an upgrade. We could look into only updating the user's password, or enabled field, if and only if they have changed. In that case, a revocation event *should* be persisted since the bootstrap command is changing something about the account. In the case where there is no change in password or enabled status, tokens should still be able to be validated across releases.

I have documented the upgrade procedure and process in a separate repository [3]

[0] https://review.openstack.org/#/c/384269/
[1] https://github.com/openstack/keystone/blob/1c60b1539cf63bba79711e237df496dfa094b2c5/keystone/cmd/cli.py#L226-L232
[2] http://cdn.pasteraw.com/9gz9964mwufyw3f98rv1mv1hqxezpis
[3] https://github.com/lbragstad/keystone-performance-upgrade

[Lance Bragstad (lbragstad)]

Adding a little more context for this since jmccrory and I discussed it quite a bit in IRC:

http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2016-12-05.log.html#t2016-12-05T22:19:12

[Dolph Mathews (dolph)]

Marking this as High because the consequence is perceivable downtime during a zero-downtime upgrade.

[Dolph Mathews (dolph)]

Marking this as Medium in mitaka since we didn't support zero-downtime upgrades then, but this is still an unexpected behavior of bootstrap that would potentially affect an upgrade process.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/408694

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/408719

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/408834

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/408835

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/408837

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/408838

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408694
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=2dae412940105c64c4ea1ed77e6a45793faa0efa
Submitter: Jenkins
Branch: master

commit 2dae412940105c64c4ea1ed77e6a45793faa0efa
Author: Lance Bragstad <email address hidden>
Date: Thu Dec 8 15:53:51 2016 +0000

    Expose idempotency issue with bootstrap

    During some upgrade testing I was doing locally, I noticed an issue
    where `keystone-manage bootstrap` isn't completely idempotent. This
    is because `bootstrap` has the ability to recover lost admin accounts
    by reseting the admin user's enabled status and updating their
    password, regardless of it being different. This creates a revocation
    event and causes admin tokens to be invalid after bootstrap is run
    for a second time, making it not as idempotent as we'd like.

    This commit introduces a test that exposes this behavior.

    Change-Id: I627255b2b5d6ec401af2c07c4018930fea206e4a
    Partial-Bug: 1647800

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408719
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=90f2f96e69b8bfd5058628b50c9f0083e3f293e9
Submitter: Jenkins
Branch: master

commit 90f2f96e69b8bfd5058628b50c9f0083e3f293e9
Author: Lance Bragstad <email address hidden>
Date: Thu Dec 8 17:01:22 2016 +0000

    Make bootstrap idempotent when it needs to be

    This commit makes `keystone-manage bootstrap` completely idempotent
    when configuration values or environment variables haven't changed
    between runs. If they have changed, then `bootstrap` shouldn't be
    as idempotent becuase it's changing the state of the deployment.

    This commit addresses these issues and adds tests to ensure the
    proper behavior is tested.

    Change-Id: I053b27e881f5bb67db1ace01e6d06aead10b1e47
    Closes-Bug: 1647800

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408834
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0d9a51ea7719afe95c754cbf4efba4621e6a58e6
Submitter: Jenkins
Branch: stable/newton

commit 0d9a51ea7719afe95c754cbf4efba4621e6a58e6
Author: Lance Bragstad <email address hidden>
Date: Thu Dec 8 15:53:51 2016 +0000

    Expose idempotency issue with bootstrap

    During some upgrade testing I was doing locally, I noticed an issue
    where `keystone-manage bootstrap` isn't completely idempotent. This
    is because `bootstrap` has the ability to recover lost admin accounts
    by reseting the admin user's enabled status and updating their
    password, regardless of it being different. This creates a revocation
    event and causes admin tokens to be invalid after bootstrap is run
    for a second time, making it not as idempotent as we'd like.

    This commit introduces a test that exposes this behavior.

    Change-Id: I627255b2b5d6ec401af2c07c4018930fea206e4a
    Partial-Bug: 1647800
    (cherry picked from commit 2dae412940105c64c4ea1ed77e6a45793faa0efa)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408835
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=16a7b074eff5597752a86a676ae8838be32b91fb
Submitter: Jenkins
Branch: stable/newton

commit 16a7b074eff5597752a86a676ae8838be32b91fb
Author: Lance Bragstad <email address hidden>
Date: Thu Dec 8 17:01:22 2016 +0000

    Make bootstrap idempotent when it needs to be

    This commit makes `keystone-manage bootstrap` completely idempotent
    when configuration values or environment variables haven't changed
    between runs. If they have changed, then `bootstrap` shouldn't be
    as idempotent becuase it's changing the state of the deployment.

    This commit addresses these issues and adds tests to ensure the
    proper behavior is tested.

    Change-Id: I053b27e881f5bb67db1ace01e6d06aead10b1e47
    Closes-Bug: 1647800
    (cherry picked from commit 90f2f96e69b8bfd5058628b50c9f0083e3f293e9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 11.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408837
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=c0f56aece02e77d7775ec823348dafc7776387ad
Submitter: Jenkins
Branch: stable/mitaka

commit c0f56aece02e77d7775ec823348dafc7776387ad
Author: Lance Bragstad <email address hidden>
Date: Thu Dec 8 15:53:51 2016 +0000

    Expose idempotency issue with bootstrap

    During some upgrade testing I was doing locally, I noticed an issue
    where `keystone-manage bootstrap` isn't completely idempotent. This
    is because `bootstrap` has the ability to recover lost admin accounts
    by reseting the admin user's enabled status and updating their
    password, regardless of it being different. This creates a revocation
    event and causes admin tokens to be invalid after bootstrap is run
    for a second time, making it not as idempotent as we'd like.

    This commit introduces a test that exposes this behavior.

    Conflicts:
      keystone/tests/unit/test_cli.py

      In Newton and newer releases the context is no longer passed in
      to the controller, but rather a request object.

    Change-Id: I627255b2b5d6ec401af2c07c4018930fea206e4a
    Partial-Bug: 1647800
    (cherry picked from commit 2dae412940105c64c4ea1ed77e6a45793faa0efa)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408838
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=a244e9bea6e4691a910b777598121c99384dd089
Submitter: Jenkins
Branch: stable/mitaka

commit a244e9bea6e4691a910b777598121c99384dd089
Author: Lance Bragstad <email address hidden>
Date: Thu Dec 8 17:01:22 2016 +0000

    Make bootstrap idempotent when it needs to be

    This commit makes `keystone-manage bootstrap` completely idempotent
    when configuration values or environment variables haven't changed
    between runs. If they have changed, then `bootstrap` shouldn't be
    as idempotent becuase it's changing the state of the deployment.

    This commit addresses these issues and adds tests to ensure the
    proper behavior is tested.

    Conflicts:
        keystone/tests/unit/test_cli.py

    As of Newton and newers releases, a context is no longer passed into
    the controller, but a request object.

    Change-Id: I053b27e881f5bb67db1ace01e6d06aead10b1e47
    Closes-Bug: 1647800
      (cherry picked from commit 90f2f96e69b8bfd5058628b50c9f0083e3f293e9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 10.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 9.3.0 release.