nova service performs wrong update from version 15 to 9

[levon]

We are running openstack mitaka out of clean installation (no upgrade).
All was functioning correctly until recently nova started to perform updates like this

2016-11-23 10:48:09.065 17498 INFO nova.service [req-c91ec40f-2dc2 - - - - -] Updating service version for nova-conductor on nova-tlc01.specs.de from 15 to 9

The full error log from /var/log/nova/nova-conductor.log looks like

2016-11-23 10:48:08.967 17499 INFO nova.service [-] Starting conductor node (version 13.1.2)
2016-11-23 10:48:09.065 17498 INFO nova.service [req-c91ec40f-2dc2- - - - - -] Updating service version for nova-conductor on nova-tlc01.specs.de from 15 to 9
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service [req-c91ec40f-2dc2-4ab0-ad61-1ab1f9d7bbc9 - - - - -] Error starting thread.
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service Traceback (most recent call last):
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 680, in run_service
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service service.start()
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 195, in start
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service _update_service_ref(self.service_ref)
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 151, in _update_service_ref
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service service.save()
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 223, in wrapper
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service return fn(self, *args, **kwargs)
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 291, in save
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service self._check_minimum_version()
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 275, in _check_minimum_version
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service minver=minver)
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service ServiceTooOld: This service is older (v9) than the minimum (v15) version of the rest of the deployment. Unable to continue.
2016-11-23 10:48:09.068 17498 ERROR oslo_service.service

I checked the changes in recent versions of nova/service.py
https://fossies.org/diffs/nova/13.1.1_vs_13.1.2/nova/service.py-diff.html

[Dan Smith (danms)]

What this means is that you are not running mitaka code everywhere. At least one service was upgraded to newton (where version 15 was introduced), it updated its service record, and then may or may not have been downgraded back. At that point, if you stop all the mitaka services, you end up with the newest service in the deployment at 15, which may then start doing things assuming no version 9 services will ever come back, which is why the version 9 services refuse to start.

So, if I were you, I would try tweaking the service records (in the services table) back to version 9 and then start all your services again (ensuring they are all on mitaka) and hope for the best. That, or upgrade to newton.

[levon (levong)]

Thanks Dan for the detailed reply.
Will check out all the version installed and let you know if this was the case.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

[alf (dmlp)]

Not a bug. Had some after mistakenly installed neutron on the second node instead of mitaka and it changed the version in the db on the first node. And did not roll back when downgraded. Solved by rolling down to mitaka and setting in nova db "update services set version='9' where version='15';"