Nova APi failed after upgrade mitaka->newton

[Jack Ivanov]

Hello everybody!

I have 2 controllers and 2 compute nodes. I disabled all the services on the first controller and compute (nova service-disable), then I upgraded those services to newton, executed `db sync` and started all the services on both nodes.
After start the FIRST controller and compute nodes, I noticed, that nova-api on my SECOND controller is down and the services don't work anymore.

nova-api log on the second controller:
ServiceTooOld: This service is older (v9) than the minimum (v15) version of the rest of the deployment. Unable to continue.

And now i have a question:

How that upgrading process of the first half of my cluster affected on the second half of my cluster?

(my goals: upgrade from mitaka to newton with minimal downtime, because I have 2 controllers, I want to upgrade the first one and only then the second one)

both server:
3.10.0-327.36.3.el7.x86_64

The first one (already newton):
openstack-nova-console-14.0.0-1.el7.noarch
openstack-nova-api-14.0.0-1.el7.noarch
openstack-nova-scheduler-14.0.0-1.el7.noarch
openstack-nova-common-14.0.0-1.el7.noarch
openstack-nova-novncproxy-14.0.0-1.el7.noarch
openstack-nova-conductor-14.0.0-1.el7.noarch

The second one (still mitaka):
openstack-nova-common-13.1.2-1.el7.noarch
openstack-nova-scheduler-13.1.2-1.el7.noarch
openstack-nova-conductor-13.1.2-1.el7.noarch
openstack-nova-api-13.1.2-1.el7.noarch
openstack-nova-console-13.1.2-1.el7.noarch
openstack-nova-novncproxy-13.1.2-1.el7.noarch

Thanks!

[Jack Ivanov (gunph1ld)]

How to reproduce:

Install nova 13 on 2 nodes
Disable the first node
Upgrade the first node to nova 14
Restart nova-api on the second node (13 ver.)
Result on the second node:
ERROR oslo_service.service ServiceTooOld: This service is older (v9) than the minimum (v15) version of the rest of the deployment. Unable to continue.

[Matt Riedemann (mriedem)]

What is the value of the 'upgrade_levels' config option in nova.conf on your controllers and compute nodes?

[Matt Riedemann (mriedem)]

Also, do you have a stacktrace of that ServiceTooOld failure?

[Jack Ivanov (gunph1ld)]

the value is mitaka

2016-12-14 08:16:57.080 10778 INFO nova.wsgi [-] osapi_compute listening on 0.0.0.0:8774
2016-12-14 08:16:57.080 10778 INFO oslo_service.service [-] Starting 8 workers
2016-12-14 08:16:57.137 10778 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
2016-12-14 08:16:57.756 10928 INFO nova.service [req-75b031fa-5bd1-4f04-80b8-3dbde522098a - - - - -] Updating service version for nova-osapi_compute on 0.0.0.0 from 15 to 9
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service [req-75b031fa-5bd1-4f04-80b8-3dbde522098a - - - - -] Error starting thread.
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service Traceback (most recent call last):
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 680, in run_service
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service service.start()
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 430, in start
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service _update_service_ref(service_ref)
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 151, in _update_service_ref
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service service.save()
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 223, in wrapper
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service return fn(self, *args, **kwargs)
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 291, in save
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service self._check_minimum_version()
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 275, in _check_minimum_version
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service minver=minver)
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service ServiceTooOld: This service is older (v9) than the minimum (v15) version of the rest of the deployment. Unable to continue.
2016-12-14 08:16:57.780 10928 ERROR oslo_service.service

[Anusha Unnam (anusha-unnam)]

We have faced the same issue when upgrading from mitaka to newton with multiapi setup.
We filed a bug https://bugs.launchpad.net/nova/+bug/1631430 for this and there is a patch proposed https://review.openstack.org/#/c/384724/ to fix this issue.

[Anusha Unnam (anusha-unnam)]

As a temporary fix, you can change the osapi-compute-listen and metadata-listen conf options in nova.conf to <ip of that node> on both nodes and restart the services and then start your upgrade process. This approach worked for us.

[Anusha Unnam (anusha-unnam)]

As this is a duplicate of this bug https://bugs.launchpad.net/nova/+bug/1631430, I am marking this bug as duplicate. please update it if you feel this is of a different bug.