Instances not visible when hidden=NULL
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Critical
|
Dan Smith |
Bug Description
During an upgrade of a cloud from Stein to Train, there is a migration which adds the `hidden` field to the database.
In that migration, it was assumed that it does not backfill all of the columns. However, upon verifying, it actually does backfill all columns and the order of operations *seems* to be:
1. Create new column for `hidden`
2. Update database migration version
3. Start backfilling all existing instances with hidden=0
In my case, the migration did create the column but failed to backfill all existing instances because of the large number of instances. However, running the migrations again seems to simply continue and not block on that migration, but leaving all columns with hidden=NULL.
=======
Feb 06 14:06:13 control02-
Feb 06 14:07:18 control02-
Feb 06 14:07:18 control02-
Feb 06 14:10:22 control02-
Feb 06 14:10:22 control02-
Feb 06 14:10:22 control02-
Feb 06 14:10:22 control02-
Feb 06 14:10:22 control02-
Feb 06 14:10:23 control02-
Feb 06 14:10:23 control02-
Feb 06 14:10:23 control02-
=======
This issue is two-part, because now it seems that Nova does not assume that hidden=NULL means that the instance is not hidden and no longer displays the instance via API or any other operations.
The "very silly" confirmation of this behaviour of backfilling was my attempt at patching things up resulted in the same error:
==================
MariaDB [nova]> update instances set hidden=0;
ERROR 1180 (HY000): Got error 90 "Message too long" during COMMIT
===================
Ideally, Nova shouldn't try and backfill values and it should treat hidden=NULL as 0.
Changed in nova: | |
importance: | Undecided → Critical |
status: | New → Confirmed |
Some more context:
(11:29:22 AM) mriedem: i sort of remember this discussion with jaypipes on https:/ /review. opendev. org/#/c/ 557958/ /docs.sqlalchem y.org/en/ 13/core/ metadata. html#sqlalchemy .schema. Column. params. default /docs.sqlalchem y.org/en/ 13/core/ metadata. html#sqlalchemy .schema. Column. params. server_ default - "A FetchedValue instance, str, Unicode or text() construct representing the DDL DEFAULT value for the column."
(11:29:26 AM) mriedem: which uses server_default
(11:30:09 AM) dansmith: I guess server_default would be one option, but it involves another schema change which I think is less good since we have to backport this
(11:34:30 AM) mriedem: https:/
(11:34:41 AM) mriedem: "which will be invoked upon insert if this column is otherwise not specified in the VALUES clause of the insert"
(11:34:45 AM) mriedem: didn't realize it would backfill though
(11:35:03 AM) dansmith: right, I always assume it fills on read too, but just insert
(11:35:07 AM) mriedem: https:/
(11:35:23 AM) mriedem: so yeah, i guess never use default in migrations