Mitaka to Newton undercloud upgrade needs "nova db online_data_migrations" after the undercloud upgrade

[Marios Andreou]

As first discussed at https://bugzilla.redhat.com/show_bug.cgi?id=1411856 when upgrading Mitaka->Newton->Ocata, you need to have run the nova db online_data_migrations after the Newton undercloud upgrade, otherwise the subsequent Ocata undercloud upgrade will fail:

Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: ValidationError: Migration cannot continue until all these have been migrated to the api database. Please run `nova-manage db online_migrations' on Newton code before continuing.There are still 8 unmigrated flavors. ESC[0m
Error: /usr/bin/nova-manage db sync returned 1 instead of one of [0]ESC[0m
Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: /usr/bin/nova-manage db sync returned 1 instead of one of [0]

The db syncs usually happen in https://github.com/openstack/instack-undercloud/blob/554977801aad85f3ac08e16762fcdc8550c95754/elements/puppet-stack-config/puppet-stack-config.pp#L26 but since this is an upgrade only action, and a newton only one at that I think we should just fix this in tripleoclient (incoming review) and only for stable/newton.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/420637

[Marios Andreou (marios-b)]

in fact going to move to puppet-stack-config as per https://bugzilla.redhat.com/show_bug.cgi?id=1411856#c4

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/421262

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/420637
Reason: abandoning in favor of adding to puppet-nova and then puppet-stack-config as discussed on the bug... see

https://review.openstack.org/#/c/421262/

[Marios Andreou (marios-b)]

adding puppet-nova because of the need to add the online-data-migrations there as well

[Marios Andreou (marios-b)]

sorry for noise, i assumed I could add more than one project. assigning back to tripleo then

[Marios Andreou (marios-b)]

So, relevant reviews are

puppet-nova@ https://review.openstack.org/#/c/421249/ "Add a class to run the db online_data_migrations" Ibb119c280b5b6d07db06a63dd9efcc2526355f5f

instack-undercloud@ https://review.openstack.org/#/c/421262/ "Adds the nova db online data migrations to the list of db sync" Ib7a495491e8fa16ab62a1c415cef30ee18664b35

As discussed in the BZ mentioned above, I think we need these backported to stable/newton too

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421249
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=50c1733e799025bc0c9c07745fc0fde7cf15b378
Submitter: Jenkins
Branch: master

commit 50c1733e799025bc0c9c07745fc0fde7cf15b378
Author: marios <email address hidden>
Date: Tue Jan 17 14:38:36 2017 +0200

    Add a class to run the db online_data_migrations

    This adds support for running the online_data_migrations
    which needs to happen after a dbsync when you upgrade to
    Newton.

    Change-Id: Ibb119c280b5b6d07db06a63dd9efcc2526355f5f
    Related-Bug: 1656791

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421262
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=bbbb9081018b8d5de6772ad55175ef444c2e855b
Submitter: Jenkins
Branch: master

commit bbbb9081018b8d5de6772ad55175ef444c2e855b
Author: marios <email address hidden>
Date: Tue Jan 17 15:01:06 2017 +0200

    Adds the nova db online data migrations to the list of db sync

    We aren't currently running the online_data_migrations nova db
    sync in puppet-stack-config

    Change-Id: Ib7a495491e8fa16ab62a1c415cef30ee18664b35
    Related-Bug: 1656791
    Depends-On: Ibb119c280b5b6d07db06a63dd9efcc2526355f5f

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/427623

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/427752

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/425323
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=f4db8c8012a0336b1a365f5c84509c6077c22a88
Submitter: Jenkins
Branch: master

commit f4db8c8012a0336b1a365f5c84509c6077c22a88
Author: Sven Anderson <email address hidden>
Date: Wed Jan 25 18:57:02 2017 +0100

    Add a pre-upgrade online_data_migration step for undercloud

    A complete upgrade from M->N would require a db sync followed by a
    db online_data_migration step after the package upgrades. Because of
    the referred bug this however cannot be guaranteed to have happened,
    which would result in an upgrade failure when later upgrading from
    N->O. Therefore we need to do an additional online_data_migration
    before the package upgrades happen, to make sure the subsequent db
    sync will not fail.

    Related-Bug: #1656791
    Change-Id: I87dad6166e335577711ddb02156d7c5146e7ea96

[Sven Anderson (ansiwen)]

I created an etherpad to track all the different tasks that are required to solve this issue completely: https://etherpad.openstack.org/p/tripleo-nova-online-data-migration

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/431590

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/427623
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=0e8534cb57f032a1bccf2bf1c9293d5748aaaaf8
Submitter: Jenkins
Branch: stable/newton

commit 0e8534cb57f032a1bccf2bf1c9293d5748aaaaf8
Author: marios <email address hidden>
Date: Tue Jan 17 14:38:36 2017 +0200

    Add a class to run the db online_data_migrations

    This adds support for running the online_data_migrations
    which needs to happen after a dbsync when you upgrade to
    Newton.

    Change-Id: Ibb119c280b5b6d07db06a63dd9efcc2526355f5f
    Related-Bug: 1656791
    (cherry picked from commit 50c1733e799025bc0c9c07745fc0fde7cf15b378)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/431590
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=0e530b489d841af0bb83f25af5a8e8f0ccbd8f51
Submitter: Jenkins
Branch: stable/newton

commit 0e530b489d841af0bb83f25af5a8e8f0ccbd8f51
Author: marios <email address hidden>
Date: Tue Jan 17 15:01:06 2017 +0200

    Adds the nova db online data migrations to the list of db sync

    We aren't currently running the online_data_migrations nova db
    sync in puppet-stack-config

    Change-Id: Ib7a495491e8fa16ab62a1c415cef30ee18664b35
    Related-Bug: 1656791
    Depends-On: Ibb119c280b5b6d07db06a63dd9efcc2526355f5f
    (cherry picked from commit bbbb9081018b8d5de6772ad55175ef444c2e855b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/427752
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=47857c48098b5d7c9a778102702def3b467e85e2
Submitter: Jenkins
Branch: stable/newton

commit 47857c48098b5d7c9a778102702def3b467e85e2
Author: Sven Anderson <email address hidden>
Date: Wed Feb 1 15:53:46 2017 +0100

    Run online_data_migrations during minor updates.

    We didn't complete the M->N upgrades correctly with a `nova-manage db
    online_data_migrations` command before, which might result in a
    performance impairment. So, as a stop-gap-solution we run it here,
    which is a noop if there is nothing to do.

    Change-Id: I7c8b04c1a0640ce3222d22920f37492c6387759e
    Related-Bug: #1656791