Kilo nova-compute unable to register with Juno nova-conductor

[Nick Jones]

When deploying a compute node running Kilo against a control node running Juno, nova-compute fails to start with the following exceptions thrown by nova-conductor:

2015-08-10 16:56:02.236 977 ERROR oslo.messaging.rpc.dispatcher [req-1d9be6ed-7b53-4dc8-bb3a-995a7ca7e359 ] Exception during message handling: Version 1.12 of Service is not supported
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 408, in object_class_action
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher objver)
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 288, in obj_class_from_name
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher supported=latest_ver)
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher IncompatibleObjectVersion: Version 1.12 of Service is not supported
2015-08-10 16:56:02.236 977 TRACE oslo.messaging.rpc.dispatcher
2015-08-10 16:56:02.237 977 ERROR oslo.messaging._drivers.common [req-1d9be6ed-7b53-4dc8-bb3a-995a7ca7e359 ] Returning exception Version 1.12 of Service is not supported to caller
2015-08-10 16:56:02.237 977 ERROR oslo.messaging._drivers.common [req-1d9be6ed-7b53-4dc8-bb3a-995a7ca7e359 ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 408, in object_class_action\n objver)\n', ' File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 288, in obj_class_from_name\n supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 1.12 of Service is not supported\n']
2015-08-10 16:56:04.212 976 WARNING nova.context [-] Arguments dropped when creating context: {u'read_only': False, u'domain': None, u'show_deleted': False, u'user_identity': u'- - - - -', u'project_domain': None, u'resource_uuid': None, u'user_domain': None}

2015-08-10 16:56:04.244 972 ERROR oslo.messaging.rpc.dispatcher [req-eb961fea-3b0a-4c34-b774-7c8f4312467f ] Exception during message handling: Version 1.16 of InstanceList is not supported
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 408, in object_class_action
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher objver)
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 288, in obj_class_from_name
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher supported=latest_ver)
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher IncompatibleObjectVersion: Version 1.16 of InstanceList is not supported
2015-08-10 16:56:04.244 972 TRACE oslo.messaging.rpc.dispatcher
2015-08-10 16:56:04.246 972 ERROR oslo.messaging._drivers.common [req-eb961fea-3b0a-4c34-b774-7c8f4312467f ] Returning exception Version 1.16 of InstanceList is not supported to caller
2015-08-10 16:56:04.246 972 ERROR oslo.messaging._drivers.common [req-eb961fea-3b0a-4c34-b774-7c8f4312467f ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 408, in object_class_action\n objver)\n', ' File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 288, in obj_class_from_name\n supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 1.16 of InstanceList is not supported\n']
2015-08-10 16:56:08.205 976 WARNING nova.context [-] Arguments dropped when creating context: {u'read_only': False, u'domain': None, u'show_deleted': False, u'user_identity': u'- - - - -', u'project_domain': None, u'resource_uuid': None, u'user_domain': None}

upgrade_levels is set to Juno on both the compute node and the control node:

[upgrade_levels]
compute = juno
conductor=juno
scheduler=juno
cert=juno
network=juno
consoleauth=juno
console=juno

The situation is further complicated by the fact that the new compute node is running CentOS 7.1, while the control node is running Ubuntu 14.04. However, I believe I should be able to make this combination work especially with the compatibility / upgrade_levels options set.

On the controller:

# dpkg -l | grep -i nova
ii nova-api 1:2014.2.2-0ubuntu1~cloud0 all OpenStack Compute - API frontend
ii nova-cert 1:2014.2.2-0ubuntu1~cloud0 all OpenStack Compute - certificate management
ii nova-common 1:2014.2.2-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-conductor 1:2014.2.2-0ubuntu1~cloud0 all OpenStack Compute - conductor service
ii nova-consoleauth 1:2014.2.2-0ubuntu1~cloud0 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 1:2014.2.2-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 1:2014.2.2-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
ii python-nova 1:2014.2.2-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 1:2.19.0-0ubuntu1~cloud0 all client library for OpenStack Compute API

On the compute node:

# rpm -qa | grep -i nova
openstack-nova-common-2015.1.0-3.el7.noarch
openstack-nova-compute-2015.1.0-3.el7.noarch
python-novaclient-2.23.0-1.el7.noarch
python-nova-2015.1.0-3.el7.noarch

Expected result is for nova-compute to start and register itself via nova-conductor.

Actual result is that nova-compute fails to start.

[Markus Zoeller (markus_z) (mzoeller)]

The exceptions are:

IncompatibleObjectVersion: Version 1.16 of InstanceList is not supported
IncompatibleObjectVersion: Version 1.12 of Service is not supported

This shows similarities to bug 1393320. Did you run the database
migration scripts (``nova-manage db sync``)? IIUC the database needs to
be on the most recent release level, even in "mixed releases" deployments.
Otherwise it wouldn't be possible to store the information from Kilo
objects in the database.

[jichenjc (jichenjc)]

As far as I can remember Juno compute nodes work with Kilo conductor but
Kilo compute nodes _not_ work with Juno conductor, maybe some one can comment it or
post it in the ML for confirm?

[Markus Zoeller (markus_z) (mzoeller)]

@jichenjc in reply to comment #2

If I understand [1] correctly, then Kilo's nova-compute shouldn't be
able to talk to Juno's nova-conductor. I specifically mean this sentence:

    "RPC pinning ensures new services can talk to the
     older service’s method signatures".

As nova-compute is the service which talks to nova-conductor, it is here
in the role of the "new service" and nova-conductor is in the role of
the "old service".

But another statement of [1] can be interpreted in the opposite way:

    "Through careful RPC versioning, newer nodes are able to talk
     to older nova-compute nodes."

As nova-conductor is usually on the controller node, the statement
above makes me believe that it should be possible that Juno's nova-
conductor should be able to talk to Kilo's nova-compute service.

The "operators guide" [2] reinforces the first assumption, that
nova-compute has to be newer than nova-conductor. It uses the example
of Havanna nova-compute services with a Grizzly nova-conductor service.

Long story short, I'm confused and will ask on the ML for clarification
and put the link to it in here.

[1] http://docs.openstack.org/developer/nova/upgrade.html?highlight=upgrade#concepts
[2] http://docs.openstack.org/openstack-ops/content/ops_upgrades_upgrade_levels.html

[Markus Zoeller (markus_z) (mzoeller)]

ML post which asks for clarification: http://lists.openstack.org/pipermail/openstack-dev/2015-August/072233.html

[jichenjc (jichenjc)]

@Markus
I remembered I saw somewhere about how to upgrade, (step by step in updating from J to K or others)
I think it's best and suitable way to use this kind of 'different version' talking to make compute host alive
during system upgrade.

so I believe controller should be upgrade first the compute node to be update set by set
that means Kilo conductor should works with Juno compute
but on the contrary I didn't see a value for supporting it ...

[Chris Friesen (cbf123)]

For Icehouse to Juno at least, the documented upgrade procedure was to upgrade the controller (including DB and nova-conductor) first, then do nova-compute later. (http://docs.openstack.org/openstack-ops/content/upgrade-icehouse-juno.html)

[Markus Zoeller (markus_z) (mzoeller)]

@Nick Jones:

It seems that this setup is not a supported model [1]:

    "No, that's not valid behaviour. You need to upgrade the controller
    infrastructure (conductor, API nodes, etc) before any compute nodes."

I'll close this bug as "Invalid" and remove the assignee because of this.

[1] http://lists.openstack.org/pipermail/openstack-dev/2015-August/072237.html

[Nick Jones (yankcrime)]

Thanks for the clarification all - duly noted. The various guides I've read suggested - to me at least - that it was possible to run mixed versions this way around. I'll go back through my various references and propose changes to documentation where applicable to remove any ambiguity.