Prevent keystone from serving requests when schema or data migrations are not up to date

Bug #1623117 reported by Dolph Mathews on 2016-09-13
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Medium
Unassigned

Bug Description

There are three scenarios during a rolling upgrade process where we could prevent operators from doing the "wrong thing" (doing things out of order):

1) Operators running code from the next release before `keystone-manage db_sync --expand` has been run: If you run the next release before --expand is run, then you'll surely end up with fatal query errors as columns and tables won't exist that the app thinks should exist.

2) (the scary one) Operators running code from the next release before `keystone-manage db_sync --migrate` has been run: If you run the next release before --migrate is run, then any number of different types of failures are possible due to unpopulated columns & tables, including a risk of data loss as the new release tries to update records that it perceives to be unpopulated, which might propagate to the legacy schema during UPDATE operations, for example.

3) Operators running code from the previous release after `keystone-manage db_sync --contract` has been run: As in case (1), this may result in fatal query errors, but also presents a risk of introducing data inconsistency, as the legacy schema might not have a "full understanding" of the new schema, as would be the case with additive schema changes. The legacy application would no longer have triggers to rely on, so consequences would mostly be dependent on the default values of columns, constraints, etc.

The second case worries me, as it's the most likely scenario where operators might not realize what's going on until it's too late.

To prevent all of these scenarios, I think the application should check at startup to ensure that the expand and data migration repositories both match a minimum value (specifically, the most recent migration in the application's respective repositories).

Doing the same sort of check at startup for the contract repo would be more difficult, as it'd be entirely dependent on when you last upgraded (whether it be last stable/* release or master at any point), so I'd like to leave that out of scope here.

Dolph Mathews (dolph) on 2016-09-13
summary: - Prevent keystone from serving requests when schema is not up to date
+ Prevent keystone from serving requests when schema or data migrations
+ are not up to date
tags: added: rc-potential
Changed in keystone:
milestone: next → ocata-1
Steve Martinelli (stevemar) wrote :

No patch yet, we're running out of time for patches going into RC, so removing rc-potential

tags: removed: rc-potential
Changed in keystone:
milestone: ocata-1 → ocata-2
Lance Bragstad (lbragstad) wrote :

This seems related to a conversation I was just having with Jesse about:

https://bugs.launchpad.net/keystone/+bug/1642212

Changed in keystone:
milestone: ocata-2 → none
Changed in keystone:
status: New → Triaged
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers