failed openstack-nova-scheduler after updating undercloud from mitaka to newton packages

[Marios Andreou]

The first part of upgrading your undercloud is to setup the new repos and then run a sudo yum -y update. The upgrade is completed by running 'openstack undercloud upgrade'. This last step is failing for me because of https://bugs.launchpad.net/tripleo/+bug/1593179 - however, I noticed that after the yum update nova-scheduler is failed:

[stack@instack ~]$ openstack-service status | grep -v active
MainPID=0 Id=neutron-server.service ActiveState=failed
MainPID=31347 Id=openstack-nova-scheduler.service ActiveState=activating

Things aren't generally expected to work OK until after you've completed the openstack undercloud upgrade, however the reason that nova-scheduler is failing is like:

[root@instack yum.repos.d]# tail /var/log/nova/nova-scheduler.log
2016-06-16 09:29:48.546 12209 ERROR nova File "/usr/lib/python2.7/site-packages/nova/cmd/scheduler.py", line 43, in main
2016-06-16 09:29:48.546 12209 ERROR nova topic=CONF.scheduler_topic)
2016-06-16 09:29:48.546 12209 ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 218, in create
2016-06-16 09:29:48.546 12209 ERROR nova db_allowed=db_allowed)
2016-06-16 09:29:48.546 12209 ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 92, in __init__
2016-06-16 09:29:48.546 12209 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2016-06-16 09:29:48.546 12209 ERROR nova File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 75, in __init__
2016-06-16 09:29:48.546 12209 ERROR nova {'conf': scheduler_driver})
2016-06-16 09:29:48.546 12209 ERROR nova RuntimeError: (u'Cannot load scheduler driver from configuration %(conf)s.', {'conf': 'nova.scheduler.filter_scheduler.FilterScheduler'})
2016-06-16 09:29:48.546 12209 ERROR nova

i.e. in nova.conf we have

scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler

If I manually change this to just 'filter_scheduler' then it passes that but hits:

[root@instack ~]# tail /var/log/nova/nova-scheduler.log
2016-06-16 09:44:21.482 19438 ERROR nova File "/usr/lib/python2.7/site-packages/nova/scheduler/driver.py", line 44, in __init__
2016-06-16 09:44:21.482 19438 ERROR nova CONF.scheduler_host_manager,
2016-06-16 09:44:21.482 19438 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2193, in __getattr__
2016-06-16 09:44:21.482 19438 ERROR nova return self._get(name)
2016-06-16 09:44:21.482 19438 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2598, in _get
2016-06-16 09:44:21.482 19438 ERROR nova value = self._do_get(name, group, namespace)
2016-06-16 09:44:21.482 19438 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2641, in _do_get
2016-06-16 09:44:21.482 19438 ERROR nova % (opt.name, str(ve)))
2016-06-16 09:44:21.482 19438 ERROR nova ConfigFileValueError: Value for option scheduler_host_manager is not valid: Valid values are [host_manager, ironic_host_manager], but found 'nova.scheduler.ironic_host_manager.IronicHostManager'

and similarly here, we have in nova.conf:

scheduler_host_manager=nova.scheduler.ironic_host_manager.IronicHostManager

So do we need to change these two values for the upgrade? Are the previously used string no longer valid for newton in nova.conf?

Package info:
[root@instack ~]# rpm -qa | grep "openstack\|tripleo\|heat"
openstack-nova-network-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
openstack-nova-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
openstack-zaqar-3.0.0-0.20160616085522.b8a70e4.el7.centos.noarch
openstack-neutron-ml2-9.0.0-0.20160616085845.1f12cb4.el7.centos.noarch
openstack-heat-api-7.0.0-0.20160616043631.2b0ae3c.el7.centos.noarch
openstack-aodh-listener-2.0.2-0.20160607162328.7212765.el7.centos.noarch
openstack-tripleo-image-elements-5.0.0-0.20160613170926.5feb901.el7.centos.noarch
openstack-ceilometer-polling-6.0.1-0.20160613225348.f99cedf.el7.centos.noarch
openstack-selinux-0.7.2-1.el7.noarch
openstack-nova-compute-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
openstack-cinder-9.0.0-0.20160616074803.93aa664.el7.centos.noarch
openstack-sahara-common-5.0.0-0.20160614222831.5fefb63.el7.centos.noarch
python-openstack-mistral-3.0.0-0.20160616041356.b03f1f1.el7.centos.noarch
openstack-aodh-common-2.0.2-0.20160607162328.7212765.el7.centos.noarch
openstack-utils-2015.2-1.el7.noarch
openstack-ceilometer-central-6.0.1-0.20160613225348.f99cedf.el7.centos.noarch
openstack-swift-object-2.8.1-0.20160615215434.3944d82.el7.centos.noarch
openstack-aodh-notifier-2.0.2-0.20160607162328.7212765.el7.centos.noarch
python-heatclient-1.2.1-0.20160609061334.1e86edb.el7.centos.noarch
openstack-swift-container-2.8.1-0.20160615215434.3944d82.el7.centos.noarch
python-openstackclient-2.6.1-0.20160615194619.114eeeb.el7.centos.noarch
openstack-heat-common-7.0.0-0.20160616043631.2b0ae3c.el7.centos.noarch
openstack-ceilometer-notification-6.0.1-0.20160613225348.f99cedf.el7.centos.noarch
openstack-nova-common-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
openstack-nova-novncproxy-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
openstack-nova-conductor-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
openstack-keystone-10.0.0-0.20160615220717.3a62069.el7.centos.noarch
openstack-sahara-5.0.0-0.20160614222831.5fefb63.el7.centos.noarch
openstack-ironic-conductor-5.1.1-0.20160615200732.823582c.el7.centos.noarch
openstack-neutron-9.0.0-0.20160616085845.1f12cb4.el7.centos.noarch
openstack-heat-api-cfn-7.0.0-0.20160616043631.2b0ae3c.el7.centos.noarch
openstack-swift-proxy-2.8.1-0.20160615215434.3944d82.el7.centos.noarch
openstack-tripleo-0.0.1-0.20160427125528.b076a5a.el7.centos.noarch
openstack-swift-2.8.1-0.20160615215434.3944d82.el7.centos.noarch
python2-openstacksdk-0.8.7-0.20160614203452.8a56981.el7.centos.noarch
openstack-ceilometer-common-6.0.1-0.20160613225348.f99cedf.el7.centos.noarch
openstack-nova-api-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
openstack-nova-scheduler-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
openstack-sahara-engine-5.0.0-0.20160614222831.5fefb63.el7.centos.noarch
openstack-glance-13.0.0-0.20160615102423.5f48788.el7.centos.noarch
openstack-neutron-common-9.0.0-0.20160616085845.1f12cb4.el7.centos.noarch
openstack-ironic-common-5.1.1-0.20160615200732.823582c.el7.centos.noarch
openstack-tripleo-puppet-elements-5.0.0-0.20160613150956.bee7e8b.el7.centos.noarch
openstack-heat-templates-0.0.1-0.20160614215755.b9f87c5.el7.centos.noarch
python-tripleoclient-5.0.0-0.20160608084412.edcbcf3.el7.centos.noarch
openstack-neutron-openvswitch-9.0.0-0.20160616085845.1f12cb4.el7.centos.noarch
openstack-ceilometer-api-6.0.1-0.20160613225348.f99cedf.el7.centos.noarch
openstack-ironic-inspector-3.3.1-0.20160614104057.8550b8b.el7.centos.noarch
openstack-nova-cert-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
python-heat-tests-7.0.0-0.20160616043631.2b0ae3c.el7.centos.noarch
openstack-sahara-api-5.0.0-0.20160614222831.5fefb63.el7.centos.noarch
openstack-mistral-common-3.0.0-0.20160616041356.b03f1f1.el7.centos.noarch
openstack-aodh-evaluator-2.0.2-0.20160607162328.7212765.el7.centos.noarch
openstack-tripleo-heat-templates-5.0.0-0.20160615053553.4bd42ea.el7.centos.noarch
openstack-tempest-12.0.1-0.20160615103516.9fd0de9.el7.centos.noarch
openstack-ceilometer-collector-6.0.1-0.20160613225348.f99cedf.el7.centos.noarch
openstack-swift-account-2.8.1-0.20160615215434.3944d82.el7.centos.noarch
openstack-aodh-api-2.0.2-0.20160607162328.7212765.el7.centos.noarch
openstack-swift-plugin-swift3-1.10-1.el7.noarch
openstack-nova-console-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
openstack-nova-cells-14.0.0-0.20160616021801.2b0557e.el7.centos.noarch
openstack-tripleo-common-5.0.0-0.20160615184428.4b0830e.el7.centos.noarch
openstack-ironic-api-5.1.1-0.20160615200732.823582c.el7.centos.noarch
openstack-heat-engine-7.0.0-0.20160616043631.2b0ae3c.el7.centos.noarch
openstack-puppet-modules-8.1.1-0.20160609150428.ab63b38.el7.centos.noarch

[Marios Andreou (marios-b)]

so I'm not seeing things :) from @ http://docs.openstack.org/developer/nova/sample_config.html
"
# For example, "nova.scheduler.filter_scheduler.FilterScheduler". But note: this
# support will be dropped in the N Release.
"

[Marios Andreou (marios-b)]

So it seems this has already been fixed on master for ironic_host_manager@instack-undercloud and filter_scheduler@puppet-nova as below:

https://review.openstack.org/#/c/309129/ "Use entrypoint for ironic_host_manager scheduler" I39348fa4247f3dc391208609bef5d920d3005a27 instack-undercloud
and
https://review.openstack.org/#/c/308452/ "Change scheduler configurations to use entrypoints" Idbbae5281d429edb95783cdde3d45804ddaeace1 puppet-nova

Does that then mean we just need a new release of instack-undercloud and puppet-nova for newton/latest ?

[Marios Andreou (marios-b)]

oops forgot to include what i've got here as the description ommitted the instack package info:

[stack@instack ~]$ yum info instack-undercloud
Installed Packages
Name : instack-undercloud
Arch : noarch
Version : 5.0.0
Release : 0.20160615175348.baf0a96.el7.centos
Size : 246 k
Repo : installed
From repo : delorean

[stack@instack releasenotes]$ yum info openstack-puppet-modules
Installed Packages
Name : openstack-puppet-modules
Arch : noarch
Epoch : 1
Version : 8.1.1
Release : 0.20160609150428.ab63b38.el7.centos
Size : 14 M
Repo : installed
From repo : delorean

[Ben Nemec (bnemec)]

Actually, I think the fix for this is to not run yum update with the services running: https://review.openstack.org/#/c/315683/

The package update restarts services automatically, and since at that point we haven't run new puppet to configure the newer versions correctly this sort of breakage is unavoidable. If we instead change the flow to be 1) stop all services 2) yum update 3) openstack undercloud install then that problem should go away (eventually openstack undercloud upgrade will just do those steps for you).

Note that this is the flow I was testing in my local undercloud upgrade test job because otherwise the undercloud upgrade was taking upwards of an hour (sometimes two!) due to services breaking. With stopping the services it takes more in the range of 20 minutes start to finish.

[Marios Andreou (marios-b)]

thanks for that bnemec; I applied that but still continued to hit errors for lots of missing things - things that landed in master but for some reason I didn't get after yum update with latest delorean/delorean-deps repos. I filed bugs for each of those individually for tracking and have listed them (as well as this.bug) at the blueprint @ https://blueprints.launchpad.net/tripleo/+spec/overcloud-upgrades-workflow-mitaka-to-newton

also thanks for the mail info on your undercloud upgrade; I will sanity run that today at some point too

[Marios Andreou (marios-b)]

this should be addressed by:

"Fix mitaka..newton upgrade for openstack-puppet-modules package" https://review.openstack.org/#/c/332889/ I2b37d2103156b7801f78738f22741183d77e8e2a python-tripleoclient

[Emilien Macchi (emilienm)]

This bug was last updated over 180 days ago, as tripleo is a fast moving project and we'd like to get the tracker down to currently actionable bugs, this is getting marked as Invalid. If the issue still exists, please feel free to reopen it.