[RFE] Add new ironic-dbsync command online_data_migrations

[Tan Lin]

This is a command to do online data migration while the ironic services are running. This is the final step to be done, after the rolling upgrade work of https://review.openstack.org/#/c/299245/. (Or the first step to be done, before the next rolling upgrade is started...)

This command is very similar to nova's "nova-manage db online_data_migrations --max-count <number>" : https://github.com/openstack/nova/blob/81aacf5cbd444dd401367cb2218499ec41207c35/nova/cmd/manage.py#L902

The command: ironic-dbsync online_data_migrations [--max-count <positive number>']

Perform online data migrations for the release, in batches. By default (unless max-count is specified), the migrations are done in batches of 50 objects, until all the objects have been migrated.

If --max-count is specified, it is the maximum number of objects to migrate in this call. It must be a positive integer; otherwise 127 is returned.

Returns 0 if all the objects have been migrated and there is nothing more to do. Otherwise returns 1.

Developers will have to add their data migrations to this script, as well as avoid or postpone incompatible schema changes to the next release.

Documentation will be added with instructions on how to write the migration scripts as well as when and how to do incompatible changes, checking that all data was migrated to the newest format before removing unused columns and tables.

Example of a change which requires online data migration:

FromVer Node Object is 1.15, ToVer Node object is 1.16 with a new field 'fake' and deprecate 'extra' field.

The rolling upgrade process will involve:

1. Updating the schema with the new column

2. Running services with the new version, pinned, which still saves data in an old format:

db_node['version'] == '1.15', db_node['fake'] is None, and db_node['extra'] contain values.

3. When this new version is unpinned, it starts saving records in a new version format:

db_node['version'] == '1.16', db_node['fake'] contain values, and db_node['extra'] is None,

but some 'untouched' rows still contain data in the old format, these are rows which weren't updated at run-time.

4. To make sure everything is converted, online schema migrations are run

NEXT RELEASE

5. A contraction script is added, to remove the old, unused 'extra' column.

The contraction script must check if all data is in the new version before removing the old column. For example, it can make sure that SELECT count(*) node WHERE version = '1.15' returns 0.

If it returns something other then 0, it should return with an error with information about running the online_data_migrations before upgrading to this release.

[Ruby Loo (rloo)]

I don't think this requires a spec, but we're still working on the rolling-upgrades spec: https://review.openstack.org/299245.

Tan Lin is no longer working on this, I'm going to assign it to Grzegorz in the mean time.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/408556

[Jim Rollenhagen (jim-rollenhagen)]

Can we add some more info about this, e.g. command syntax, how we block the next upgrade until this has run (so columns aren't deleted without migrating data), that we'll document this as part of the upgrade process.

I'd also highly recommend using ironic-dbsync online_data_migrations --limit <number>, for some consistency with nova (nova-manage db online_data_migrations --limit <number>). I'd also prefer this to just run all data migrations, rather than letting the user choose (and eventually do it wrong, and force us to document which migrations to run for which release).

[Ruby Loo (rloo)]

This looks good to me. I am fine with approving it.

I have one question wrt the example in the description :

> 5. A contraction script is added, to remove the old, unused 'extra' column.

Where is this contraction script? Is it added to this dbsync online-data-migration command? If so, and the online-data-migration command from the previous release wasn't run (to do the migrations), that's what this/current command will indicate -- to run the older online-data-migration command?

[Jim Rollenhagen (jim-rollenhagen)]

@rloo that would just be a typical database schema migration, like we have lots of already. :)

[Jim Rollenhagen (jim-rollenhagen)]

And yes, I think this command should block itself from running if the online data migration had not been run.

[Jim Rollenhagen (jim-rollenhagen)]

I'm not sure why I thought nova used --limit in my last comment, but --max-count is indeed correct. This LGTM, I'm +1 on approving.

[Ruby Loo (rloo)]

Since nova's command is 'nova-manage db online_data_migrations...', I think we should be consistent and name this 'ironic-dbsync online_data_migrations' (instead of the singular 'ironic-dbsync online_data_migration'; I don't know grammatically which one might be more correct).

[Jim Rollenhagen (jim-rollenhagen)]

Changing the description to be online_data_migrations, and approving.

[Ruby Loo (rloo)]

Here's the patch that addresses this: https://review.openstack.org/#/c/408556/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408556
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=54efd312395a56cbeee5c556df34afd8153c8076
Submitter: Jenkins
Branch: master

commit 54efd312395a56cbeee5c556df34afd8153c8076
Author: Grzegorz Grasza <email address hidden>
Date: Thu Dec 8 11:34:27 2016 +0100

    Add new dbsync command with first online data migration

    This adds the new command 'ironic-dbsync online_data_migrations'.
    To limit downtime during upgrades, data migrations will be done online
    with migration scripts that could be run during normal operation of an
    ironic cluster, after the upgrade but before the next one.

    Each migration script should ensure that all related DB records are
    migrated to the new format. Scripts can detect the format based on
    the object version which is stored in the version column.

    The online data migration has one script; a function that backfills
    the new version column, using versions of objects from the release
    prior to this.

    This includes code to check the object versions for compatibility with
    an ironic release. However, the check is turned off (and will be turned on
    in Queens), since we need to boot-strap the new version column before
    we can turn the check on. To do this check, we need to keep a list of all
    supported versions for every object; release_mapping.RELEASE_MAPPING was
    modified so that the object versions is now a list instead of one value.

    Change-Id: I1a9fa829951ecf98cae6896d82ba20cf89062394
    Closes-Bug: #1585141
    Partial-bug: #1526283
    Co-Authored-By: Ruby Loo <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 9.0.0 release.