nova-manage: db sync failure leaves DB in an inconsistent state
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Confirmed
|
Wishlist
|
MOS Nova |
Bug Description
Steps to reproduce:
1. Start deploying a cluster (1 controller node and a compute one is enough)
2. Manually interrupt nova-manage db sync
3. Try to re-run nova-manage db sync
The expected result: the 2nd run of nova-manage db sync completes successfully
The actual result: the 2nd run of nova-manage db sync fails (backtrace: [1]).
Thus a failed db sync leaves the cluster in a badly broken state.
Recovering from such a state requires manual inspection of tables and selectively running
DB migration scripts to add missing tables, columns, constraints, etc.
This is not a big deal for a new deployment (one can wipe out all tables and start over),
however this is a real disaster for upgrades.
Notes: nova-manage db sync expects DDL statements to be transactional (which is quite natural),
however they are not transactional with mysql (which is the default DB in MOS).
Therefore migration scripts can't trust the version recorded in the DB, and should explicitly
check if all tables/
This involves using 'create table if not exists' (as opposed to 'create table'), and so on.
[1]
root@node-1:~# nova-manage db sync
Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
Option "notification_
Option "notification_
error: (_mysql_
Changed in fuel: | |
assignee: | MOS Nova (mos-nova) → nobody |
Changed in fuel: | |
status: | Invalid → New |
Changed in fuel: | |
status: | New → Confirmed |
importance: | High → Wishlist |
assignee: | nobody → MOS Nova (mos-nova) |
milestone: | 9.2 → 11.0 |
Nova team, can you comment here?