Prevent --expand, --migrate, --contract from being run out of order

Bug #1615014 reported by Dolph Mathews on 2016-08-19
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Low
Richard

Bug Description

Currently, keystone does nothing to prevent an operator from running each step of the rolling migration process out of order. Theoretically, most migrations will fail if the table they're looking to drop does not exist, etc, but that might not always be the case.

The transition to rolling migrations introduces a few expectations between the migration repositories that we should be able to enforce rather easily, given that all 3 of the new migration repos should always contain the same number of migration steps).

1. All legacy migrations need to be run before any --expand migrations are allow to run.

2. The version number of the --expand repo must be greater than or equal to the version number of the --migrate repo.

3. The version number of the --migrate repo must be greater than or equal to the version number of the --contract repo.

I'd expect each command to abort with an error message if there are outstanding steps from the previous repository that had not been run. As a bit of a special case, db_sync --expand could (continue to) run the legacy repository automatically, but only if the legacy repository is guaranteed to be additive-only, as non-additive migrations should never be run by --expand (perhaps we should find the version number of the last non-additive migration and check the current state of the db will not result in that non-additive migration to be accidentally run.

Dolph Mathews (dolph) on 2016-08-19
Changed in keystone:
milestone: none → newton-3
Dolph Mathews (dolph) wrote :

Bug 1615020 and bug 1615024 also document new assertions regarding rolling upgrades.

Henry Nash (henry-nash) on 2016-08-19
Changed in keystone:
assignee: nobody → Henry Nash (henry-nash)
Steve Martinelli (stevemar) wrote :

as with bug 1615020 and bug 1615024, this is not necessary for newton-3

Changed in keystone:
milestone: newton-3 → none
Changed in keystone:
milestone: none → ocata-1
Changed in keystone:
milestone: ocata-1 → ocata-2
Lance Bragstad (lbragstad) wrote :

This seems similar to what I was just talking to Jesse about:

https://bugs.launchpad.net/keystone/+bug/1642212

Changed in keystone:
milestone: ocata-2 → none
Lance Bragstad (lbragstad) wrote :

Here is a related patch that exposes this behavior in tests:

https://review.openstack.org/#/c/429113/

Reviewed: https://review.openstack.org/429113
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0ebd557325ff16c2535dac0eb190792e6d511a79
Submitter: Jenkins
Branch: master

commit 0ebd557325ff16c2535dac0eb190792e6d511a79
Author: Richard Avelar <email address hidden>
Date: Fri Feb 3 23:53:21 2017 +0000

    Add unit test for db_sync run out of order

    This patch adds a unit test that shows that db_sync expand, migrate,
    and contract can be run out of order.

    Related-Bug: 1615014
    Change-Id: Ied6c0a48a50f882b604636d0288d954b29439e6c

Fix proposed to branch: master
Review: https://review.openstack.org/437441

Changed in keystone:
assignee: Henry Nash (henry-nash) → Richard (csravelar)
status: Triaged → In Progress
Changed in keystone:
assignee: Richard (csravelar) → Lance Bragstad (lbragstad)
Changed in keystone:
milestone: none → pike-3
tags: added: office-hours
Changed in keystone:
assignee: Lance Bragstad (lbragstad) → nobody
assignee: nobody → Richard (csravelar)

Reviewed: https://review.openstack.org/437441
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=6bab551cd8a523332b7c387c36c701cb90fd96bd
Submitter: Jenkins
Branch: master

commit 6bab551cd8a523332b7c387c36c701cb90fd96bd
Author: Richard Avelar <email address hidden>
Date: Thu Feb 23 15:35:21 2017 +0000

    Validate rolling upgrade is run in order

    This patch addresses a bug that allows rolling upgrades to be run
    out of order and without first checking if the previous command
    has been run to a higher version before hand.

    Change-Id: I55fa4f600d89f3a2fb14868f6886b52fd1ef6c6b
    Closes-Bug: 1615014

Changed in keystone:
status: In Progress → Fix Released

This issue was fixed in the openstack/keystone 12.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers