Environment is neutron 14.0.2 with DVR and HA (OVS).
Upgraded a single compute or deployed new with 14.0.3.
Expected outcome:
Minor versions should be fully compatible and neutron should work with the same major version.
Actual outcome:
Can't schedule instances on computes holding this version and neutron services spew out errors.
neutron-server on controller/network node:
Exception during message handling: InvalidTargetVersion: Invalid target version 1.5
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 148, in bulk_pull
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server **filter_kwargs)]
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 551, in obj_to_primitive
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server raise exception.InvalidTargetVersion(version=target_version)
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server InvalidTargetVersion: Invalid target version 1.5
2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server
neutron-openvswitch-agent on compute node:
Error while processing VIF ports: RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
[u'Traceback (most recent call last):\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner\n return func(*args, **kwargs)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 148, in bulk_pull\n **filter_kwargs)]\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 551, in obj_to_primitive\n raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2278, in rpc_loop
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1845, in process_network_ports
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 143, in _apply_port_filter
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.context, list(device_ids)[i:i + step])
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py", line 232, in security_group_info_for_devices
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ports = self._get_devices_info(context, devices)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py", line 300, in _get_devices_info
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ovo = self.rcache.get_resource_by_id('Port', device)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 61, in get_resource_by_id
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent agent_restarted=agent_restarted)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 79, in _flood_cache_for_query
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(*args, **kwargs)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 114, in bulk_pull
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron_lib/rpc.py", line 157, in call
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return self._original_context.call(ctxt, method, **kwargs)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry)
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise result
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [u'Traceback (most recent call last):\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner\n return func(*args, **kwargs)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 148, in bulk_pull\n **filter_kwargs)]\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 551, in obj_to_primitive\n raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
AND
Failed to get details for device 227d33c9-a8c5-45aa-bd98-8bc3ff2f6029: RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
[u'Traceback (most recent call last):\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner\n return func(*args, **kwargs)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 148, in bulk_pull\n **filter_kwargs)]\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 551, in obj_to_primitive\n raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc Traceback (most recent call last):
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 303, in get_devices_details_list_and_failed_devices
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc agent_restarted))
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 312, in get_device_details
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc resources.PORT, device, agent_restarted)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 61, in get_resource_by_id
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc agent_restarted=agent_restarted)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 79, in _flood_cache_for_query
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc return method(*args, **kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 114, in bulk_pull
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron_lib/rpc.py", line 157, in call
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc return self._original_context.call(ctxt, method, **kwargs)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc retry=self.retry)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc retry=retry)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc call_monitor_timeout, retry=retry)
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc raise result
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc [u'Traceback (most recent call last):\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner\n return func(*args, **kwargs)\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 148, in bulk_pull\n **filter_kwargs)]\n', u' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 551, in obj_to_primitive\n raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.5\n'].
2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc
AND
Clearing cache of registered ports, retries to resync were > 5: RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
I have switched back to 14.0.2 (5ish weeks old) and things are back on track in the environment.
Forgot to add, this comes from stein/stable branch.