Undercloud upgrade fails with "Cell mappings are not created, but required for Ocata"

[Steven Hardy]

Trying to upgrade with recent trunk nova and puppet-nova gives this error:

Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: error: Cell mappings are not created, but required for Ocata. Please run nova-manage db simple_cell_setup before continuing.
Error: /usr/bin/nova-manage api_db sync returned 1 instead of one of [0]
Error: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: change from notrun to 0 failed: /usr/bin/nova-manage api_db sync returned 1 instead of one of [0]

Debugging manually gives:

$ sudo /usr/bin/nova-manage api_db sync
error: Cell mappings are not created, but required for Ocata. Please run nova-manage db simple_cell_setup before continuing.

but...

$ sudo nova-manage db simple_cell_setup
usage: nova-manage db [-h]

                      {archive_deleted_rows,null_instance_uuid_scan,online_data_migrations,sync,version}
                      ...
nova-manage db: error: argument action: invalid choice: 'simple_cell_setup' (choose from 'archive_deleted_rows', 'null_instance_uuid_scan', 'online_data_migrations', 'sync', 'version')

I tried adding openstack-nova* to the delorean-current whitelist, but with the latest nova packages there still appears to be this mismatch.

[stack@instack /]$ rpm -qa | grep nova
openstack-nova-conductor-15.0.0-0.20161212155146.909410c.el7.centos.noarch
python-nova-15.0.0-0.20161212155146.909410c.el7.centos.noarch
openstack-nova-scheduler-15.0.0-0.20161212155146.909410c.el7.centos.noarch
puppet-nova-10.0.0-0.20161211003757.09b9f7b.el7.centos.noarch
python2-novaclient-6.0.0-0.20161003181629.25117fa.el7.centos.noarch
openstack-nova-api-15.0.0-0.20161212155146.909410c.el7.centos.noarch
openstack-nova-cert-15.0.0-0.20161212155146.909410c.el7.centos.noarch
openstack-nova-common-15.0.0-0.20161212155146.909410c.el7.centos.noarch
openstack-nova-compute-15.0.0-0.20161212155146.909410c.el7.centos.noarch

[Steven Hardy (shardy)]

Hmm, so it looks like the command may actually be "nova-manage cell_v2 simple_cell_setup, but that requires a transport-url that's evidently different to that set in the nova.conf :\

[Steven Hardy (shardy)]

[stack@instack ~]$ sudo grep transport_url /etc/nova/nova.conf
transport_url=rabbit://55f7b1c2b4ee0e8a4f8311de334c6b71d13c1b45:1cf85a15b3fb0d86ec3bda2dedd3b8952ad6d72a@192.0.2.1//

[stack@instack ~]$ sudo nova-manage cell_v2 simple_cell_setup --transport-url "rabbit://55f7b1c2b4ee0e8a4f8311de334c6b71d13c1b45:1cf85a15b3fb0d86ec3bda2dedd3b8952ad6d72a@192.0.2.1//"
Traceback (most recent call last):
  File "/bin/nova-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1561, in main
    config.parse_args(sys.argv)
  File "/usr/lib/python2.7/site-packages/nova/config.py", line 50, in parse_args
    rpc.init(CONF)
  File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 74, in init
    TRANSPORT = create_transport(get_transport_url())
  File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 154, in get_transport_url
    return messaging.TransportURL.parse(CONF, url_str, TRANSPORT_ALIASES)
  File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 398, in parse
    url = url or conf.transport_url
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2320, in __getattr__
    raise NoSuchOptError(name)
oslo_config.cfg.NoSuchOptError: no such option transport_url in group [DEFAULT]

It may be this is partly a nova and partly a puppet-nova bug? As it seems the nova releasenotes and api_db sync help text is wrong, and it seems like puppet-nova isn't driving the cells_v2 simple_cell_setup at the appropriate time with the URL it expects (which isn't all that clear as evidently it doesn't match the nova.conf setting with the same name).

[Steven Hardy (shardy)]

[stack@instack ~]$ sudo nova-manage --config-file /etc/nova/nova.conf cell_v2 simple_cell_setup
usage: nova-manage cell_v2 simple_cell_setup [-h] --transport-url <transport
                                             url>
nova-manage cell_v2 simple_cell_setup: error: argument --transport-url is required

So it seems you're forced to provide the --transport-url even when you point to the nova.conf :(

[Steven Hardy (shardy)]

Ok, so lets try to enumerate the issues:

1. sudo /usr/bin/nova-manage api_db sync should handle this internally, but assuming that's not possible...

2. The "nova-manage db simple_cell_setup" help and releasenote text appears to be wrong

3. The (presumably correct?) nova-manage cell_v2 simple_cell_setup command should handle getting the transport URL from the -c <config> file, and that should default to /etc/nova/nova.conf. Also, the transport_url should work when you cut/paste it from nova.conf?

4. puppet-nova should probably handle this dance (assuming 1 above is not possible), but I'm not clear how we detect when to run this command - according to the git history it should be idempotent, but are there reasons to not run it every update?

[Alex Schultz (alex-schultz)]

https://review.openstack.org/#/c/392227 is the change that added this requirement

[Matt Riedemann (mriedem)]

The command to run should be "nova-manage cell_v2 simple_cell_setup" as seen in grenade:

http://git.openstack.org/cgit/openstack-dev/grenade/tree/projects/60_nova/from-newton/upgrade-nova#n9

That's a bug in nova.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/409876

[Matt Riedemann (mriedem)]

Here is the simple_cell_setup command being run in the grenade job for master (ocata):

http://logs.openstack.org/27/392227/6/check/gate-grenade-dsvm-neutron-ubuntu-xenial/a4db7a1/logs/grenade.sh.txt.gz#_2016-12-06_01_14_37_580

nova-manage cell_v2 simple_cell_setup --transport-url rabbit://stackrabbit:secretrabbit@localhost:5672/

[Matt Riedemann (mriedem)]

The transport_url is required to be passed in because of limitations in oslo.messaging, as seen in the code comment here:

https://github.com/openstack/nova/blob/d9d9071af250409c0102e7a25e35d7521812172d/nova/cmd/manage.py#L1211

melwitt would be the person to ask about that, since I think she was working an oslo blueprint for this in newton.

[Steven Hardy (shardy)]

Ok so it appears we didn't have the very latest oslo.messaging, which contains a fix related to the comment #2 error - now we get this:

python2-oslo-messaging.noarch 0:5.14.0-0.20161208143704.044e1bb.el7.centos

Complete!
[stack@instack ~]$ sudo nova-manage cell_v2 simple_cell_setup --transport-url "rabbit://55f7b1c2b4ee0e8a4f8311de334c6b71d13c1b45:1cf85a15b3fb0d86ec3bda2dedd3b8952ad6d72a@192.0.2.1//"
error: (pymysql.err.OperationalError) (1044, u"Access denied for user 'nova_api'@'192.0.2.1' to database 'nova_api_cell0'")

So I guess we've got some DB wrangling to do in puppet too :(

[Matt Riedemann (mriedem)]

The nova team needs to cleanup this docs change for the cells v2 upgrade information:

https://review.openstack.org/#/c/267153/

[Matt Riedemann (mriedem)]

The map_cell0 command is not idempotent, so if you run it more than once it's going to fail the 2nd time with a duplicate entry here:

https://github.com/openstack/nova/blob/d9d9071af250409c0102e7a25e35d7521812172d/nova/cmd/manage.py#L1283

The simple_cell_setup command uses that information here to tell if we've already done the simple setup:

https://github.com/openstack/nova/blob/d9d9071af250409c0102e7a25e35d7521812172d/nova/cmd/manage.py#L1230

And then exits cleanly so simple_cell_setup is idempotent. But you can get into trouble if you run map_cell0 and then simple_cell_setup, because simple_cell_setup will exit early and then not actually perform the host mapping stuff here:

https://github.com/openstack/nova/blob/d9d9071af250409c0102e7a25e35d7521812172d/nova/cmd/manage.py#L1237

And then later when you run the nova-manage api_db sync it will fail because of the 030 API DB migration added in https://review.openstack.org/#/c/392227. The nova team needs to work on that. A manual workaround is delete the cell0 entry from the cell_mappings table in the api_db and then run simple_cell_setup again.

[Steven Hardy (shardy)]

It seems that simple_cell_setup is not idempotent if there's permissions issues with the DB:

[stack@instack ~]$ sudo nova-manage cell_v2 simple_cell_setup --transport-url "rabbit://55f7b1c2b4ee0e8a4f8311de334c6b71d13c1b45:1cf85a15b3fb0d86ec3bda2dedd3b8952ad6d72a@192.0.2.1//"
error: (pymysql.err.OperationalError) (1044, u"Access denied for user 'nova_api'@'192.0.2.1' to database 'nova_api_cell0'")
[stack@instack ~]$ sudo nova-manage cell_v2 simple_cell_setup --transport-url "rabbit://55f7b1c2b4ee0e8a4f8311de334c6b71d13c1b45:1cf85a15b3fb0d86ec3bda2dedd3b8952ad6d72a@192.0.2.1//"
Already setup, nothing to do.

In this case I forgot to grant the nova_api user access to the new nova_api_cell0 DB and got the above confusing result.

The fix is:

1. Create a new database nova_api_cell0
2. Grant the nova_api user all privileges on the new DB
3. Run sudo nova-manage cell_v2 simple_cell_setup --transport-url "rabbit://55f7b1c2b4ee0e8a4f8311de334c6b71d13c1b45:1cf85a15b3fb0d86ec3bda2dedd3b8952ad6d72a@192.0.2.1//"
4. sudo /usr/bin/nova-manage api_db sync

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/409890

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409876
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6764ff0db2aaa4dda0d804c7db886d3b64226674
Submitter: Jenkins
Branch: master

commit 6764ff0db2aaa4dda0d804c7db886d3b64226674
Author: Matt Riedemann <email address hidden>
Date: Mon Dec 12 12:52:27 2016 -0500

    Fix instructions for running simple_cell_setup

    Change ff6b9998bb977421a5cbc94878ced8542d910c9e enforces in
    a database migration that you've run the simple_cell_setup
    command for cells v2 but the instructions in the error and
    in the release note said to use 'nova-manage db' when it should
    be 'nova-manage cell_v2'.

    Change-Id: I8e71d1c7022d1000f26b7c16ed1c56f6e87ab8ac
    Closes-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/409948

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/410324

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/412006

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/412007

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/412520

[Ben Nemec (bnemec)]

Are any of the proposed tripleo patches marked closes-bug? If not it would be nice if we could mark the last one we expect to merge that way so the bug state gets handled automatically by the release tools.

[Alex Schultz (alex-schultz)]

I'll make sure the last one has a closes-bug

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409948
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=4234ce3df430f16d3e9278e587238148db8e3a47
Submitter: Jenkins
Branch: master

commit 4234ce3df430f16d3e9278e587238148db8e3a47
Author: Alex Schultz <email address hidden>
Date: Mon Dec 12 14:52:57 2016 -0700

    Add cell_v2 simple_cell_setup

    As part of Ocata, nova has made the cell_v2 setup manditory for the
    nova-api db sync process. This change adds a simple cell_v2 setup with a
    cell0 and an execution of the 'nova-manage cell_v2 simple_cell_setup' as
    part of the nova-api db setup and sync process.

    Change-Id: Idfc369e9e17f7d5a30ce4ff52beb604dd4a6ac23
    Closes-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/412006
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=4f1727990019e9bfbc1e1f265b163952257ada22
Submitter: Jenkins
Branch: master

commit 4f1727990019e9bfbc1e1f265b163952257ada22
Author: Alex Schultz <email address hidden>
Date: Fri Dec 16 16:05:43 2016 -0700

    Add cell_v2 simple_cell_setup

    The cell_v2 setup for Nova is becoming required in Ocata and is required
    for upgrades. This change enables the cell0 setup as part of the
    undercloud setup.

    Change-Id: I9462ef16fd64a577c3f950bd121f0bd28670fabc
    Depends-On: Idfc369e9e17f7d5a30ce4ff52beb604dd4a6ac23
    Related-Bug: #1649341

[Steven Hardy (shardy)]

Can anyone clarify how we promoted current-tripleo yesterday given that this still isn't fixed on the TripleO side?

I just upgraded my undercloud and it still breaks in exactly the same way. I'm also not clear why our upgrades CI job doesn't fail?

[Alex Schultz (alex-schultz)]

So this hasn't been fixed for the upgrade yet. The issue with the upgrade is that the database needs to have been configured prior to the upgrade occurring and we just landed support for this on the undercloud in Ocata. The api db sync has to happen before the cell setup on new installs and the upgrade needs to have the cell setup before the api db sync. Because of this distinct ordering issue, we may need to raise this as something that needs to be backported to Newton to get the cell setup prior to the upgrade. Alternatively if there's a way to handle specific N->O upgrade actions, that might also be a solution.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/413717

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/413717
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=53e6f7a64e59e5af03b219f5cf54f118dc8b867a
Submitter: Jenkins
Branch: stable/newton

commit 53e6f7a64e59e5af03b219f5cf54f118dc8b867a
Author: Alex Schultz <email address hidden>
Date: Mon Dec 12 14:52:57 2016 -0700

    Add cell_v2 simple_cell_setup

    As part of Ocata, nova has made the cell_v2 setup manditory for the
    nova-api db sync process. This change adds a simple cell_v2 setup with a
    cell0 and an execution of the 'nova-manage cell_v2 simple_cell_setup' as
    part of the nova-api db setup and sync process.

    Change-Id: Idfc369e9e17f7d5a30ce4ff52beb604dd4a6ac23
    Closes-Bug: #1649341
    (cherry picked from commit 4234ce3df430f16d3e9278e587238148db8e3a47)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/412520
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7f58104f5dc9ebf02ba8bfbc5864b59482b87fed
Submitter: Jenkins
Branch: master

commit 7f58104f5dc9ebf02ba8bfbc5864b59482b87fed
Author: Alex Schultz <email address hidden>
Date: Mon Dec 19 09:23:33 2016 -0700

    Pass nova rabbit information to mysql

    The cell v2 setup requires the transport url for nova. We need to
    provide mysql with the rabbit connection information so that it can it
    when setting up the cell information.

    Change-Id: I43ba77cd4c8da7c6dc117ab0bd53e5cd330dc3de
    Related-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/416342

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/412007
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=b6f79567e23709b87e538ac75f7be20a306f7438
Submitter: Jenkins
Branch: master

commit b6f79567e23709b87e538ac75f7be20a306f7438
Author: Alex Schultz <email address hidden>
Date: Fri Dec 16 16:22:54 2016 -0700

    Add cell_v2 setup for nova

    We need to run the basic cell v2 setup for nova as it is required for
    Ocata.

    Change-Id: I693239ff5026f58a65eb6278b1a8fcb97af4f561
    Depends-On: I43ba77cd4c8da7c6dc117ab0bd53e5cd330dc3de
    Depends-On: I9462ef16fd64a577c3f950bd121f0bd28670fabc
    Closes-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/416353

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/410324
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=055f91446fa2e32eb2ee04f0db232bb7fc8cdba3
Submitter: Jenkins
Branch: master

commit 055f91446fa2e32eb2ee04f0db232bb7fc8cdba3
Author: Alex Schultz <email address hidden>
Date: Tue Dec 13 10:08:58 2016 -0700

    Enable cell_v2 setup by default

    In Ocata, nova has made the cell_v2 setup mandatory as part of the
    upgrade process in I72fb724dc13e1a5f4e97c58915b538ba761c582d. This
    change makes the cell_v2 setup part of the default install for Ocata as
    this feature will be required for some parts of nova in Ocata.

    Depends-On: I693239ff5026f58a65eb6278b1a8fcb97af4f561
    Depends-On: I08efa36f5f62ae406ab20e8da77c01fbf3d9b11f
    Change-Id: I7839e3c8bfdec96b0d372e7a9d3511289e8004b9
    Related-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416342
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4e3b085a59e7af49d1025986fd80796be338f5fd
Submitter: Jenkins
Branch: master

commit 4e3b085a59e7af49d1025986fd80796be338f5fd
Author: Alex Schultz <email address hidden>
Date: Tue Jan 3 13:14:03 2017 -0700

    Specify cell0 db creation

    This change pulls the hard coded value out of puppet-tripleo to later
    allow people to skip the cell0 creation if they want a more complex cell
    v2 setup for nova.

    Change-Id: I08119d781ef60750cc19753bc03190e413159925
    Related-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416353
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=77cd102c9a96c1be4bef7f8afb95f300c06d2134
Submitter: Jenkins
Branch: master

commit 77cd102c9a96c1be4bef7f8afb95f300c06d2134
Author: Alex Schultz <email address hidden>
Date: Tue Jan 3 14:19:45 2017 -0700

    Use THT to define cell0 creation

    As part of the initial implementation, we hard coded the cell0 setup in
    puppet. This change switches it to leverage the defined value in the
    tripleo-heat-templates

    Change-Id: I896a124d91d06ca85b77c9fbe24fd252815a2d28
    Depends-On: I08119d781ef60750cc19753bc03190e413159925
    Related-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/419112

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/419118

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/419121

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409890
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=34761e678f286cce2479d320356c9196d67cc4fb
Submitter: Jenkins
Branch: master

commit 34761e678f286cce2479d320356c9196d67cc4fb
Author: Dan Smith <email address hidden>
Date: Mon Dec 12 11:05:28 2016 -0800

    Make simple_cell_setup fully idempotent

    Previously this command would treat the presence of a cell0 mapping
    as indication that cellsv2 is fully setup. This could be a false
    indication for a variety of situations, including having set up a
    cell0 mapping manually, as well as failing to sync the database and
    thus not continuing on to host/instance mapping.

    This patch makes us continue on, even if the mapping is present,
    and also even if the sync of the cell0 database fails.

    Change-Id: Ic05e1a998d7807d58fc35637742059d61f7669e9
    Closes-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/420051

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/420351

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420351
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0e7a38a3d930ae717da624efea99ae57ff7025da
Submitter: Jenkins
Branch: master

commit 0e7a38a3d930ae717da624efea99ae57ff7025da
Author: Emilien Macchi <email address hidden>
Date: Sat Jan 14 10:21:19 2017 -0500

    nova: disable ::nova::db::sync_cell_v2

    This feature is broken for us now and there is work in progress in Nova
    to improve nova cell deployment.
    Until it's fixed upstream, we need to disable cells deployment for now,
    so we can promote our CI.

    Change-Id: I379ba9e94a92ed225a03a67fc975b542447a9c8b
    Related-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419112
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=a661b6d5e17fbe3652f77726a0b4c7a87f7fcbbc
Submitter: Jenkins
Branch: master

commit a661b6d5e17fbe3652f77726a0b4c7a87f7fcbbc
Author: Alex Schultz <email address hidden>
Date: Wed Jan 11 12:43:01 2017 -0700

    Improve upgrade process to include upgrade flag

    Previously, the upgrade process for the undercloud has been to just
    rerun the install process. The problem is that this does not include
    anwyay to indicate that it's already been run if we need to trigger some
    different actions on subsequent runs. As part of the Newton to Ocata
    upgrade, we need to be able to run the cell v2 setup in a different
    order than on a traditional install. We need a way for the puppet
    scripts that ultimately get run to know if it's an upgrade or install
    action. This change adds the creation of an undercloud_upgrade fact that
    will be used when the upgrade process is run. Additionally a new script
    called instack-upgrade-undercloud has been created to be used when
    upgrade. The use of instack-install-undercloud will cause the install
    flow to be execuated, while the instack-upgrade-undercloud will set the
    fact to true to allow for the puppet scripts to use the upgrade flow.

    Change-Id: Ie3cb21e30334fe8ffc0a9d6e707b42269b64c9ec
    Related-Bug: #1649341
    Related-Blueprint: undercloud-upgrade

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419118
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=78d468884200e862d81faceb4629ff0ea1c94b67
Submitter: Jenkins
Branch: master

commit 78d468884200e862d81faceb4629ff0ea1c94b67
Author: Alex Schultz <email address hidden>
Date: Wed Jan 11 13:04:06 2017 -0700

    Switch undercloud upgrade to use upgrade script

    In order to better handle upgrades, we need to be able to know when
    we are performing an upgrade instead of an installation. This change
    switches the undercloud upgrade process to use the
    instack-upgrade-undercloud script which will expose the fact that we are
    running an upgrade process to the underlying puppet scripts.

    Change-Id: I493204117ca158e99f706770a02fbb7f6392673b
    Depends-On: Ie3cb21e30334fe8ffc0a9d6e707b42269b64c9ec
    Related-Bug: #1649341
    Related-Blueprint: undercloud-upgrade

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/425273

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-nova 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419121
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=ad47a3c40fbf13427457e4b0b9fb7a684de6377a
Submitter: Jenkins
Branch: master

commit ad47a3c40fbf13427457e4b0b9fb7a684de6377a
Author: Alex Schultz <email address hidden>
Date: Wed Jan 11 13:18:55 2017 -0700

    Run cell v2 setup differently for upgrades

    For the upgrades from Newton, we need to run the cell v2 simple setup
    before running the api db sync since we did not deploy cell v2 in
    Newton. Starting with Ocata, api db sync errors if the cell v2 simple
    setup has not been done.

    Change-Id: I80ecc0943e08952fbc29c55da9d4ba8f51b0eb7e
    Closes-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/425273
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=185546cfa5cd7da03733091dc3621e28ecc3ad4f
Submitter: Jenkins
Branch: master

commit 185546cfa5cd7da03733091dc3621e28ecc3ad4f
Author: Oliver Walsh <email address hidden>
Date: Wed Jan 25 15:28:40 2017 +0000

    Run nova manage cell_v2 discover_host after ironic nodes are registered

    Once cell_v2 is enabled by default in Ocata we will need to run this command
    to update the host mappings whenever a new host is added.

    Change-Id: I0bb98bfcf0c271ae3b577ca126dd3dfdaece179e
    Depends-On: I591b451197dc3bd0783978f5e3d2b1c830afe54e
    Related-Bug: #1649341

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-nova 9.5.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420051
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=afa458f1bdb3bfa36ce3512abae0b20db8547411
Submitter: Jenkins
Branch: stable/newton

commit afa458f1bdb3bfa36ce3512abae0b20db8547411
Author: Dan Smith <email address hidden>
Date: Mon Dec 12 11:05:28 2016 -0800

    Make simple_cell_setup fully idempotent

    Previously this command would treat the presence of a cell0 mapping
    as indication that cellsv2 is fully setup. This could be a false
    indication for a variety of situations, including having set up a
    cell0 mapping manually, as well as failing to sync the database and
    thus not continuing on to host/instance mapping.

    This patch makes us continue on, even if the mapping is present,
    and also even if the sync of the cell0 database fails.

    Change-Id: Ic05e1a998d7807d58fc35637742059d61f7669e9
    Closes-Bug: #1649341
    (cherry picked from commit 34761e678f286cce2479d320356c9196d67cc4fb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 6.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 6.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.4 release.