l3 agent is failing with unsupported version endpoint does not support rpc version 1.3

[kedar kulkarni]

my devstack script is failing with the above mentioned error. It is causing my CI to fail.

the stack trace is

2014-08-06 12:31:44.935 TRACE neutron Traceback (most recent call last):
2014-08-06 12:31:44.935 TRACE neutron File "/usr/local/bin/neutron-l3-agent", line 10, in <module>
2014-08-06 12:31:44.935 TRACE neutron sys.exit(main())
2014-08-06 12:31:44.935 TRACE neutron File "/opt/stack/neutron/neutron/agent/l3_agent.py", line 1787, in main
2014-08-06 12:31:44.935 TRACE neutron manager=manager)
2014-08-06 12:31:44.935 TRACE neutron File "/opt/stack/neutron/neutron/service.py", line 264, in create
2014-08-06 12:31:44.935 TRACE neutron periodic_fuzzy_delay=periodic_fuzzy_delay)
2014-08-06 12:31:44.935 TRACE neutron File "/opt/stack/neutron/neutron/service.py", line 197, in __init__
2014-08-06 12:31:44.935 TRACE neutron self.manager = manager_class(host=host, *args, **kwargs)
2014-08-06 12:31:44.935 TRACE neutron File "/opt/stack/neutron/neutron/agent/l3_agent.py", line 1706, in __init__
2014-08-06 12:31:44.935 TRACE neutron super(L3NATAgentWithStateReport, self).__init__(host=host, conf=conf)
2014-08-06 12:31:44.935 TRACE neutron File "/opt/stack/neutron/neutron/agent/l3_agent.py", line 430, in __init__
2014-08-06 12:31:44.935 TRACE neutron self.plugin_rpc.get_service_plugin_list(self.context))
2014-08-06 12:31:44.935 TRACE neutron File "/opt/stack/neutron/neutron/agent/l3_agent.py", line 142, in get_service_plugin_list
2014-08-06 12:31:44.935 TRACE neutron version='1.3')
2014-08-06 12:31:44.935 TRACE neutron File "/opt/stack/neutron/neutron/common/log.py", line 36, in wrapper
2014-08-06 12:31:44.935 TRACE neutron return method(*args, **kwargs)
2014-08-06 12:31:44.935 TRACE neutron File "/opt/stack/neutron/neutron/common/rpc.py", line 170, in call
2014-08-06 12:31:44.935 TRACE neutron context, msg, rpc_method='call', **kwargs)
2014-08-06 12:31:44.935 TRACE neutron File "/opt/stack/neutron/neutron/common/rpc.py", line 196, in __call_rpc_method
2014-08-06 12:31:44.935 TRACE neutron return func(context, msg['method'], **msg['args'])
2014-08-06 12:31:44.935 TRACE neutron File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-08-06 12:31:44.935 TRACE neutron retry=self.retry)
2014-08-06 12:31:44.935 TRACE neutron File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-08-06 12:31:44.935 TRACE neutron timeout=timeout, retry=retry)
2014-08-06 12:31:44.935 TRACE neutron File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 404, in send
2014-08-06 12:31:44.935 TRACE neutron retry=retry)
2014-08-06 12:31:44.935 TRACE neutron File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 395, in _send
2014-08-06 12:31:44.935 TRACE neutron raise result
2014-08-06 12:31:44.935 TRACE neutron RemoteError: Remote error: UnsupportedVersion Endpoint does not support RPC version 1.3

Please help.

[kedar kulkarni (kedar-kulkarni)]

It started after https://github.com/openstack/neutron/commit/ebecca030af3541dc2c379f8f51183619d9a6655 is merged.

Are there any changes required from my side?

[Eugene Nikanorov (enikanorov)]

Please make sure you have updated whole neutron code. It seems that there is some discrepancy between l3 agent code and server-side code.

[kedar kulkarni (kedar-kulkarni)]

I have checked out latest neutron code, even though it shows the same error.

[Nurit Vilosny (nuritv)]

Same issue in our CI setup.

[Akihiro Motoki (amotoki)]

The following commit bumped the RPC version of l3-agent.
L3-agent tried to get service plugin list when it starts.

commit d6f014d0922e03864fd72efbcde04322711c2510
Author: sridhargaddam <email address hidden>
Date: Sun Apr 27 06:25:28 2014 +0530

    Exit Firewall Agent if config is invalid

The commit only bumped the RPC version of L3 router service plugin
and this bug affects existing plugins using L3 router mixin.
The version should be bumped for such plugins, but in most plugins
all callback mixin are mixin-ed into a single class and it is not easy to bump
the RPC version. Ideally RPC callback class should be split into a callback per feature.

As a fix for this bug, it seems better to enable FWaaS config check (which checks if configurations in FWaaS plugin and FWaaS agent matches) ONLY if version 1.3 L3 RPC is supported and otherwise skip the check.

[Salvatore Orlando (salvatore-orlando)]

Or what if we revert commit d6f014d0922e03864fd72efbcde04322711c2510, go back to the drawing board and look at the solution you're proposing?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/114146

[Sumit Naiksatam (snaiksat)]

Solution proposed by Akihiro in his patch seems like a reasonable compromise between short and long term approaches.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114146
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ca00b1739431fb9fe8f81919dc4644f4f467fa9d
Submitter: Jenkins
Branch: master

commit ca00b1739431fb9fe8f81919dc4644f4f467fa9d
Author: Akihiro Motoki <email address hidden>
Date: Thu Aug 14 15:53:47 2014 +0900

    Skip FWaaS config mismatch check if RPC method is unsupported

    In this commit FWaaS config check skip is skipped
    if neutron server does not support this API.

    Commit d6f014d introduced FWaaS config mismatch check between
    server and agent. It added a new RPC method get_service_plugin_list
    and bumped l3-agent RPC version to 1.3, but this version RPC is
    only supported by L3 router service plugin and
    it breaks existing plugins using L3 router mixin.

    Bumping l3-agent RPC version requires detailed investigation on all
    affected plugins and it can be done by plugin maintainer later.

    Change-Id: I388a24b0c6a507203674ef108bb14cea0534f98c
    Closes-Bug: #1353309