mitaka is incompatible with newton - IncompatibleObjectVersion Version 2.1 of InstanceList is not supported

[Jack Ivanov]

Description
===========

I get an error after upgrade half of my cluster. Can't place any VMs.
"RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported"

Steps to reproduce
==================

1) Install 4 nodes with mitaka
2) Disable 2 nodes (1 api controller and 1 compute): nova service-disable
3) Upgrade to newton on the disable nodes
4) compute=mitaka to [upgrade_levels]
5) db sync
6) Start newton
7) Try to place any VMs, it will fail
8) See nova-compute.log on the mitaka nodes

Expected result
===============

Successful upgrade one half of cluster, then another half

Actual result
=============

Nova can't place any VMs.

Compute logs:

2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [req-41e6df10-b33b-47f5-be0c-86793cbcae6e - - - - -] Error during ComputeManager._sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task task(self, context)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1637, in _sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task use_slave=True)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task args, kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task args=args, kwargs=kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task retry=self.retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task timeout=timeout, retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task raise result
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n :param incoming: incoming message\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n objname, object_versions[objname])\n', u' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n'].
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task

nova-conductor:

2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Exception during message handling: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher :param incoming: incoming message
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher objname, object_versions[objname])
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher supported=latest_ver)
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.491 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Returning exception Version 2.1 of InstanceList is not supported to caller
2016-12-23 08:01:00.492 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n :param incoming: incoming message\n', ' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n objname, object_versions[objname])\n', ' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n']

Environment
===========

CentOS Linux 7 (Core) 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

mitaka nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-compute-13.1.2-1.el7.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-common-13.1.2-1.el7.noarch

contoller:
openstack-nova-scheduler-13.1.2-1.el7.noarch
openstack-nova-conductor-13.1.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-api-13.1.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-common-13.1.2-1.el7.noarch
openstack-nova-console-13.1.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-novncproxy-13.1.2-1.el7.noarch

netonw nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
python-nova-14.0.2-1.el7.noarch
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-compute-14.0.2-1.el7.noarch

controller:
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-console-14.0.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-conductor-14.0.2-1.el7.noarch
openstack-nova-novncproxy-14.0.2-1.el7.noarch
openstack-nova-api-14.0.2-1.el7.noarch
openstack-nova-scheduler-14.0.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-14.0.2-1.el7.noarch

[Matt Riedemann (mriedem)]

Your nova-conductor node/service is still running mitaka code, correct? And do you have upgrade_levels set to anything in the nova.conf on the host that's running the nova-conductor service?

[Matt Riedemann (mriedem)]

Oh I guess you have mitaka and newton controllers, so nova-conductor services are running both versions, I'm not sure if that's going to work unless you've set [upgrade_levels]compute=mitaka in both places and restarted nova-conductor.

Dan Smith is really the person to ask about this but he's out until next week.

[Jack Ivanov (gunph1ld)]

Matt, thanks for the answer. This is my 4th step in the "Reproduce list":

"4) compute=mitaka to [upgrade_levels]"

[Dan Smith (danms)]

Yeah, mixed-version controllers isn't supported. We've made some progress towards being able to support it in master, but it's definitely not going to work in mitaka/newton.

You have to upgrade your controllers simultaneously (well, most critically, your conductor services), and then you can have any mix of versions among your computes that you want.