[RFE] Support rolling upgrades

[Vladyslav Drok]

This is going to take quite a bit of work. Things to consider here:

* getting all API->DB calls remoted to conductor
* Properly versioning objects and making them compatible between versions
* Ditto for RPC APIs
* How we do DB migrations
* all the docs.

This is going to be partially code, partially user docs, and partially developer/reviewer docs.

[Vladyslav Drok (vdrok)]

https://etherpad.openstack.org/p/ironic-online-upgrade-support

[Vladyslav Drok (vdrok)]

Copy of whiteboard contents:

Gerrit topic: https://review.openstack.org/#q,topic:bp/online-upgrade-support,n,z

Addressed by: https://review.openstack.org/235909
    Make object methods remotable again

Do we need a spec for that? I create a etherpad for this, but a spec should be better. https://etherpad.openstack.org/p/ironic-online-upgrade-support // lintan 2015-10-29

Addressed by: https://review.openstack.org/243497
    Isolate ir-api from DB

Addressed by: https://review.openstack.org/248647
    Add Code Review Guide for Ironic

Gerrit topic: https://review.openstack.org/#q,topic:bug/1502895,n,z

Addressed by: https://review.openstack.org/249624
    Add a test to enforce object version bump correctly

Gerrit topic: https://review.openstack.org/#q,topic:RPC_version,n,z

Addressed by: https://review.openstack.org/253355
    Add a config option [upgrade_level]/condcutor_rpcapi_cap

[Jim Rollenhagen (jim-rollenhagen)]

@lintan yeah, maybe we should create a spec for this one.

[Jim Rollenhagen (jim-rollenhagen)]

We should probably also talk about what grenade-partial means for us. Do we only upgrade ironic api or conductor? Should nova be upgraded? Should nova be partially upgraded?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/299245

[Tan Lin (tan-lin-good)]

@jroll, I create a spec and let's discuss https://review.openstack.org/299245

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/306357

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tan Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/243497
Reason: Not a correct way

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/332614

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299245
Committed: https://git.openstack.org/cgit/openstack/ironic-specs/commit/?id=11bb3ac9604065b932062d3bd71c0590481223ee
Submitter: Jenkins
Branch: master

commit 11bb3ac9604065b932062d3bd71c0590481223ee
Author: Lin Tan <email address hidden>
Date: Thu Mar 31 13:45:55 2016 +0800

    Support rolling upgrades

    This describes how to support rolling upgrades of ironic.

    Partial-Bug: #1526283
    Co-Authored-By: Ruby Loo <email address hidden>
    Co-Authored-By: Grzegorz Grasza <email address hidden>
    Change-Id: I75392ee17678b3087f47ada98d6fa18b4ad66a8e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/407491

[Ruby Loo (rloo)]

Spec was approved: https://review.openstack.org/299245

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/407997

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/408556

[OpenStack Infra (hudson-openstack)]

Change abandoned by Szymon Borkowski (<email address hidden>) on branch: master
Review: https://review.openstack.org/407997
Reason: Merging with the RPC version pinning patch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/412397

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/419439

[OpenStack Infra (hudson-openstack)]

Change abandoned by Grzegorz Grasza (xek) (<email address hidden>) on branch: master
Review: https://review.openstack.org/420728

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ruby Loo (<email address hidden>) on branch: master
Review: https://review.openstack.org/306357
Reason: This has been superceded by https://review.openstack.org/#/c/407491/ and https://review.openstack.org/#/c/412397/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407491
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=9bc06783ecaff5eef58535904abc12ea15982329
Submitter: Jenkins
Branch: master

commit 9bc06783ecaff5eef58535904abc12ea15982329
Author: Grzegorz Grasza <email address hidden>
Date: Tue Dec 6 13:40:45 2016 +0100

    Add RPC and object version pinning

    To support rolling upgrades, capping of RPC communication and Ironic
    objects is required. Old RPC services and objects may still be running
    while an upgrade is in progress. This makes sure that these old
    services are called and all objects are used in a supported RPC and
    objects version.

    This patch adds the configuration option "pin_release_version". Setting
    this option caps (downgrades) the internal RPC communication to the
    specified version to enable communication with older services. When
    doing a rolling upgrade from version X to Y, set this to X. It defaults
    to using the newest (current) possible RPC behavior and object versions.

    Change-Id: Ie2342d4051f85392a8b10d39ebffc287da57bf2b
    Partial-Bug: #1526283
    Co-Authored-By: Szymon Borkowski <email address hidden>
    Co-Authored-By: Ruby Loo <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/455536

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/455536
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=d8438df350c044ca2992822a888c20f21f1c2ea1
Submitter: Jenkins
Branch: master

commit d8438df350c044ca2992822a888c20f21f1c2ea1
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Tue Apr 11 06:32:27 2017 +0000

    Add comments re RPC versions being in sync

    place it to remind to keep it in sync there too.

    Change-Id: I745b4b14d6217030a463a9a343eee139d8e155b3
    Related-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/455811

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ruby Loo (<email address hidden>) on branch: master
Review: https://review.openstack.org/415462
Reason: This has been superceded by this POC: https://review.openstack.org/#/c/455811/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/469940

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/469940
Committed: https://git.openstack.org/cgit/openstack/ironic-specs/commit/?id=5370fd3878c86622a389c236c87c7a2ac6ea4903
Submitter: Jenkins
Branch: master

commit 5370fd3878c86622a389c236c87c7a2ac6ea4903
Author: Ruby Loo <email address hidden>
Date: Thu Jun 1 12:31:31 2017 -0400

    Rolling upgrades: different object versions

    This updates the rolling upgrades spec to change the design
    of how ironic services will handle the different IronicObject
    versions during a rolling upgrade.

    It also adds some more alternatives.

    Change-Id: I3c6caa639aa75ed87bddd59bf514e2d904db6e0a
    Partial-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/412397
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=ebfc4fe4c4c3910bf8b1229cb75259befa530877
Submitter: Jenkins
Branch: master

commit ebfc4fe4c4c3910bf8b1229cb75259befa530877
Author: Grzegorz Grasza <email address hidden>
Date: Mon Dec 19 11:32:34 2016 +0100

    Add version column

    This extends versioned objects and uses the object's
    convert_to_version() to perform conversions between different
    versions of the object. This conversion is only done at the
    "boundaries" of the api and conductor services. That is, when
    reading/writing to the database and when serializing/deserializing
    (for RPC). Internally, the services deal with the latest versions
    of the objects.

    Version column is introduced to make sure reading the DB object
    versions happens transparently for the developer. An exception
    is raised in case of a version compatibility error.

    The version column is null at first and will be filled with the
    appropriate versions by a data migration script in a following patch.

    Change-Id: I34629a5cbab7ff3f246ea19c0cb766badc7061db
    Partial-Bug: #1526283
    Co-Authored-By: Ruby Loo <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/474325

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/474325
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=6061585fc1e095c233a62d46f1e877496e7cebc8
Submitter: Jenkins
Branch: master

commit 6061585fc1e095c233a62d46f1e877496e7cebc8
Author: Ruby Loo <email address hidden>
Date: Wed Jun 14 15:10:37 2017 -0400

    Minor changes to object version-related code

    This includes some changes to make the code easier to understand,
    as well as updated docstrings and comments for more clarity.

    This is a follow up to ebfc4fe4c4c3910bf8b1229cb75259befa530877.

    Change-Id: I3620b6ada9915eb68d6bba47279d61bf51a6e608
    Partial-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/476779

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/477694

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/484800

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/484890

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/476779
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=8e7d795db5e614de6a3d8dd19ff26f869c8e9f9a
Submitter: Jenkins
Branch: master

commit 8e7d795db5e614de6a3d8dd19ff26f869c8e9f9a
Author: Ruby Loo <email address hidden>
Date: Fri Jun 23 01:00:42 2017 -0400

    Modifications for rolling upgrades

    This contains some changes to conversions of objects during a rolling
    upgrade.

    This changes objects.base.IronicObject.convert_to_version() to have a new
    parameter 'remove_unavail_fields': True (default) to remove fields that are
    unavailable in the target version -- for serialization/deserialization
    of objects. False to change unavailable fields as appropriate (for DB
    interactions). The reason for doing this is to make sure that during
    serialization (eg for RPC), that we don't include any object
    fields that are not supported.

    To make the code a bit more performant, we don't perform object
    conversions when the API service is serializing the objects for RPC to
    the conductor. This is because the conductor will always be running the
    same or a newer release as the API service.

    Change-Id: I6f77b24199412e3489dd6f3dcf0f51ed04c5c7c0
    Partial-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/484800
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=abf6d356b3162b63c239325761f949b3f87725e4
Submitter: Jenkins
Branch: master

commit abf6d356b3162b63c239325761f949b3f87725e4
Author: Ruby Loo <email address hidden>
Date: Tue Jul 18 09:45:24 2017 -0400

    Rename 'remove_unavail_fields' parameter

    Renames the parameter 'remove_unavail_fields' to
    'remove_unavailable_fields'.

    This is a follow up to 8e7d795db5e614de6a3d8dd19ff26f869c8e9f9a.

    Change-Id: I57f7bde85e959f3edeb878089057bfcd26672570
    Partial-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ruby Loo (<email address hidden>) on branch: master
Review: https://review.openstack.org/484890
Reason: This is in https://review.openstack.org/#/c/469933/, which merged

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/485527

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mark Goddard (<email address hidden>) on branch: master
Review: https://review.openstack.org/485527
Reason: Abandoning this in favour of a specific solution in [1] after discussion with Ruby.

[1] https://review.openstack.org/#/c/477694

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/485773

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/477694
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=7e4030600fe28f5f641e9743462700a28156c40e
Submitter: Jenkins
Branch: master

commit 7e4030600fe28f5f641e9743462700a28156c40e
Author: Ruby Loo <email address hidden>
Date: Mon Jun 26 18:11:48 2017 -0400

    Rolling upgrades support for port.physical_network

    The new field 'physical_network' was added to Port in version
    1.7, and port creation was moved from the API to the conductor
    service in commit 9e3f4121865807294dd3768b095b67eb01ef539e.

    This adds objects.port.Port._convert_to_version(). The method handles
    converting the new physical_network field between different versions of
    the Port.

    Change-Id: I8a5b1e3dd72884a511b43839cdb264354bc46ea0
    Co-Authored-By: Mark Goddard <email address hidden>
    Related-Bug: #1666009
    Related-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/485773
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=fc1e18e3e4a28b1163e7a597ee8b07d465d7a174
Submitter: Jenkins
Branch: master

commit fc1e18e3e4a28b1163e7a597ee8b07d465d7a174
Author: Mark Goddard <email address hidden>
Date: Thu Jul 20 10:41:54 2017 +0100

    Rolling upgrades support for create_port RPCAPI

    The new field 'physical_network' was added to Port in version
    1.7, and port creation was moved from the API to the conductor
    service in commit 9e3f4121865807294dd3768b095b67eb01ef539e.

    This change adds a method can_send_create_port() to the conductor
    RPCAPI that allows the caller to determine whether the conductor is able
    to create ports. During a rolling upgrade this may return False, and the
    API will need to determine whether it is able to create the port locally
    as was done previously.

    The create_port RPC method was added to support validation of the
    physical_network field of ports in portgroups. A port may therefore be
    safely created in the API service if it is not a member of a portgroup.
    If the port being created is a member of a portgroup, then it cannot be
    safely validated by the API service and the request must be rejected.

    Change-Id: I8c417cba085f61c3d2ffe1f7e97c64fa85a014cb
    Related-Bug: #1666009
    Related-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408556
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=54efd312395a56cbeee5c556df34afd8153c8076
Submitter: Jenkins
Branch: master

commit 54efd312395a56cbeee5c556df34afd8153c8076
Author: Grzegorz Grasza <email address hidden>
Date: Thu Dec 8 11:34:27 2016 +0100

    Add new dbsync command with first online data migration

    This adds the new command 'ironic-dbsync online_data_migrations'.
    To limit downtime during upgrades, data migrations will be done online
    with migration scripts that could be run during normal operation of an
    ironic cluster, after the upgrade but before the next one.

    Each migration script should ensure that all related DB records are
    migrated to the new format. Scripts can detect the format based on
    the object version which is stored in the version column.

    The online data migration has one script; a function that backfills
    the new version column, using versions of objects from the release
    prior to this.

    This includes code to check the object versions for compatibility with
    an ironic release. However, the check is turned off (and will be turned on
    in Queens), since we need to boot-strap the new version column before
    we can turn the check on. To do this check, we need to keep a list of all
    supported versions for every object; release_mapping.RELEASE_MAPPING was
    modified so that the object versions is now a list instead of one value.

    Change-Id: I1a9fa829951ecf98cae6896d82ba20cf89062394
    Closes-Bug: #1585141
    Partial-bug: #1526283
    Co-Authored-By: Ruby Loo <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416565
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=519e379c7809e4e7903c2648e13da3587ee0f1ce
Submitter: Jenkins
Branch: master

commit 519e379c7809e4e7903c2648e13da3587ee0f1ce
Author: Grzegorz Grasza <email address hidden>
Date: Wed Jan 4 15:08:14 2017 +0100

    Rolling upgrade procedure documentation

    This extends the upgrade guide with information on how to perform
    a rolling upgrade with minimal downtime. It also adds a new section
    for offline (cold) upgrades.

    Partial-bug: #1526283
    Co-Authored-By: Ruby Loo <email address hidden>
    Change-Id: If7f41ccbfd8d3128a142cd6e4f090c5c9825cf60

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/497593

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/497666

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/497593
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=8a982282513bf496944124e685e5bc7938cd11e8
Submitter: Jenkins
Branch: master

commit 8a982282513bf496944124e685e5bc7938cd11e8
Author: Ruby Loo <email address hidden>
Date: Thu Aug 24 15:34:13 2017 -0400

    conductor saves version in db

    For rolling upgrades, we added a 'version' column to all the
    DB object tables. However, we forgot to save the version value
    for conductor objects. This patch addresses that.

    Change-Id: Ic366f771491774f9708c9a81ab76dc13757d852b
    Partial-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/498579

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/497949
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=e5b7aa203324b5d390e21ae725c910ea7e0e9502
Submitter: Jenkins
Branch: master

commit e5b7aa203324b5d390e21ae725c910ea7e0e9502
Author: Vasyl Saienko <email address hidden>
Date: Fri Aug 25 17:39:54 2017 +0300

    Test ironic-dbsync online_data_migrations

    Run ironic-dbsync online_data_migrations on single node grenade
    right after usual dbsync.
    Do not set pin_relase_version for single node grenade as we
    do cold upgrade and it is not required.

    Related-Bug: 1526283
    Related-Bug: 1502887
    Change-Id: I2e331252aeb66376c2d6cc3c16b6de0aefca5434

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ruby Loo (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/498579
Reason: After talking to Julia Kreger about it, we decided it wasn't worth backporting. If there was a cluster out there with 20-40 (max) conductors, that would be very quick to update via the ironic-dbsync online_data_migrations command. This backport isn't needed, I did it thinking it would save a bit of time/avoid the online migration to set the version in the DB, but it doesn't seem worth it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/498579
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=46f9192234bd8e924c3fd2f506da423c3f2dc92f
Submitter: Jenkins
Branch: stable/pike

commit 46f9192234bd8e924c3fd2f506da423c3f2dc92f
Author: Ruby Loo <email address hidden>
Date: Thu Aug 24 15:34:13 2017 -0400

    conductor saves version in db

    For rolling upgrades, we added a 'version' column to all the
    DB object tables. However, we forgot to save the version value
    for conductor objects. This patch addresses that.

    Change-Id: Ic366f771491774f9708c9a81ab76dc13757d852b
    Partial-Bug: #1526283
    (cherry picked from commit 8a982282513bf496944124e685e5bc7938cd11e8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/497666
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=f1d442b72286ec54ffc19fc6884114c695f8da5f
Submitter: Zuul
Branch: master

commit f1d442b72286ec54ffc19fc6884114c695f8da5f
Author: Ruby Loo <email address hidden>
Date: Thu Aug 24 18:04:13 2017 -0400

    Add online data migrations for conductor version

    This modifies the backfill_version_column method that was used in Pike,
    to backfill the version column for the Conductor object, since we missed
    setting the Conductor's version column in Pike. This method is invoked
    when the operator runs ``ironic-dbsync online_data_migrations``.

    Change-Id: I42fec0ac9c8b684219f14822648e627fce529d36
    Partial-Bug: 1526283

[Ruby Loo (rloo)]

This is basically done, ironic has the tag: supports-rolling upgrade (https://review.openstack.org/#/c/514448/)

The only thing left is to get this merged: https://review.openstack.org/#/c/419439/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419439
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=4fb3808316791999efcd76dc9932c2a479477d56
Submitter: Zuul
Branch: master

commit 4fb3808316791999efcd76dc9932c2a479477d56
Author: Grzegorz Grasza <email address hidden>
Date: Thu Jan 12 13:12:02 2017 +0100

    Rolling upgrades related dev documentation

    Adds a new 'Rolling Upgrades' developer document that describes the
    design (from the associated spec) and includes notes on what to do
    when developing a new feature or modifying an IronicObject.

    Replaces the rolling upgrades information in the code contribution
    guide, with a link to this new Rolling Upgrades page.

    Change-Id: I5cc0b42da6e9e66f5c607bddbb3a2232cda97c6f
    Closes-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 9.2.0 release.