oslo.messaging

  1. Bug #1301723
  2. Comment #0

Comment 0 for bug 1301723

Revision history for this message
zhangjialong (zhangjl) wrote : Message was sended to wrong node with use zmq as rpc_backend

In Icehouse b3 and rc1, i use zmq as nova rpc_backend. When i boot vm instances, error occored like follows:

2014-04-03 11:58:35.897 6583 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: computer_4 is not a valid node managed by this compute host.
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher payload)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 244, in decorated_function
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher pass
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 230, in decorated_function
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 295, in decorated_function
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 272, in decorated_function
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 259, in decorated_function
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1849, in run_instance
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher do_run_instance()
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1848, in do_run_instance
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher legacy_bdm_in_spec)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 997, in _run_instance
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher notify("error", fault=e) # notify that build failed
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 981, in _run_instance
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher instance, image_meta, legacy_bdm_in_spec)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1035, in _build_instance
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher rt = self._get_resource_tracker(node)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 454, in _get_resource_tracker
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher "compute host.") % nodename)
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher NovaException: computer_4 is not a valid node managed by this compute host.
2014-04-03 11:58:35.897 6583 TRACE oslo.messaging.rpc.dispatcher

For more details I found, the message should be send to computer_4, while computer_1 received.

In the method named _multi_send in the file impl_zmq.py , there is a problem when get and handle the queues .