RPCVersionCapError: Requested message version, 3.1 is incompatible. It needs to be equal in major version and less than or equal in minor version as the specified version cap 3.0. in c-vol in grenade jobs

[Matt Riedemann]

Seeing this in grenade jobs on master (ocata) changes since 12/2:

http://logs.openstack.org/40/402140/7/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/ad0cf41/logs/new/screen-c-vol.txt.gz?level=TRACE

2016-12-05 09:05:33.071 ERROR oslo_service.service [req-24b2e43c-5d5e-4d60-823c-7c7f73060e7c None] Error starting thread.
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service Traceback (most recent call last):
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service service.start()
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service File "/opt/stack/new/cinder/cinder/service.py", line 239, in start
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service service_id=Service.service_id)
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service File "/opt/stack/new/cinder/cinder/volume/manager.py", line 461, in init_host
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service self.publish_service_capabilities(ctxt)
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service File "/opt/stack/new/cinder/cinder/volume/manager.py", line 2018, in publish_service_capabilities
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service self._publish_service_capabilities(context)
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service File "/opt/stack/new/cinder/cinder/manager.py", line 181, in _publish_service_capabilities
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service self.last_capabilities)
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service File "/opt/stack/new/cinder/cinder/scheduler/rpcapi.py", line 149, in notify_service_capabilities
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service capabilities=capabilities)
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 144, in cast
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service self._check_version_cap(msg.get('version'))
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 121, in _check_version_cap
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service version_cap=self.version_cap)
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service RPCVersionCapError: Requested message version, 3.1 is incompatible. It needs to be equal in major version and less than or equal in minor version as the specified version cap 3.0.
2016-12-05 09:05:33.071 10423 ERROR oslo_service.service

That's due to this change merging in cinder on 12/2 which adds the 3.1 RPC version:

https://github.com/openstack/cinder/commit/3c835298a47d20a07dbcd5195903e60a4d5f6117

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22RPCVersionCapError%3A%20Requested%20message%20version%2C%203.1%20is%20incompatible.%20%20It%20needs%20to%20be%20equal%20in%20major%20version%20and%20less%20than%20or%20equal%20in%20minor%20version%20as%20the%20specified%20version%20cap%203.0.%5C%22%20AND%20tags%3A%5C%22screen-c-vol.txt%5C%22&from=7d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/407612

[Michal Dulko (michal-dulko-f)]

Back to high, as it's actually the root cause of the problems seen in Neutron gate.

[Matt Riedemann (mriedem)]

FYI, detailed debug can be found in bug 1647431.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407612
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=850a19722640557ec8c5ad2992f2427c237a2129
Submitter: Jenkins
Branch: master

commit 850a19722640557ec8c5ad2992f2427c237a2129
Author: Michał Dulko <email address hidden>
Date: Tue Dec 6 17:58:50 2016 +0100

    Block 3.1 message with c-sch RPCAPI pinned to 3.0

    notify_service_capabilities was sent unconditionally, even when we
    shouldn't allow it due to RPC version pinning. This commit blocks that
    message when RPC API is pinned to anything lower than 3.1.

    Change-Id: Ib2e4f65f626cad79773f46077a75813f434cb806
    Closes-Bug: 1647789

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 10.0.0.0b2 development milestone.