Prevent --expand, --migrate, --contract from being run out of order

[Dolph Mathews]

Currently, keystone does nothing to prevent an operator from running each step of the rolling migration process out of order. Theoretically, most migrations will fail if the table they're looking to drop does not exist, etc, but that might not always be the case.

The transition to rolling migrations introduces a few expectations between the migration repositories that we should be able to enforce rather easily, given that all 3 of the new migration repos should always contain the same number of migration steps).

1. All legacy migrations need to be run before any --expand migrations are allow to run.

2. The version number of the --expand repo must be greater than or equal to the version number of the --migrate repo.

3. The version number of the --migrate repo must be greater than or equal to the version number of the --contract repo.

I'd expect each command to abort with an error message if there are outstanding steps from the previous repository that had not been run. As a bit of a special case, db_sync --expand could (continue to) run the legacy repository automatically, but only if the legacy repository is guaranteed to be additive-only, as non-additive migrations should never be run by --expand (perhaps we should find the version number of the last non-additive migration and check the current state of the db will not result in that non-additive migration to be accidentally run.

[Dolph Mathews (dolph)]

Bug 1615020 and bug 1615024 also document new assertions regarding rolling upgrades.

[Steve Martinelli (stevemar)]

as with bug 1615020 and bug 1615024, this is not necessary for newton-3

[Lance Bragstad (lbragstad)]

This seems similar to what I was just talking to Jesse about:

https://bugs.launchpad.net/keystone/+bug/1642212

[Lance Bragstad (lbragstad)]

Here is a related patch that exposes this behavior in tests:

https://review.openstack.org/#/c/429113/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/429113
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0ebd557325ff16c2535dac0eb190792e6d511a79
Submitter: Jenkins
Branch: master

commit 0ebd557325ff16c2535dac0eb190792e6d511a79
Author: Richard Avelar <email address hidden>
Date: Fri Feb 3 23:53:21 2017 +0000

    Add unit test for db_sync run out of order

    This patch adds a unit test that shows that db_sync expand, migrate,
    and contract can be run out of order.

    Related-Bug: 1615014
    Change-Id: Ied6c0a48a50f882b604636d0288d954b29439e6c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/437441

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/437441
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=6bab551cd8a523332b7c387c36c701cb90fd96bd
Submitter: Jenkins
Branch: master

commit 6bab551cd8a523332b7c387c36c701cb90fd96bd
Author: Richard Avelar <email address hidden>
Date: Thu Feb 23 15:35:21 2017 +0000

    Validate rolling upgrade is run in order

    This patch addresses a bug that allows rolling upgrades to be run
    out of order and without first checking if the previous command
    has been run to a higher version before hand.

    Change-Id: I55fa4f600d89f3a2fb14868f6886b52fd1ef6c6b
    Closes-Bug: 1615014

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 12.0.0.0b3 development milestone.