First time,instance "vmtest" evacuated failed on B with libvirt error. Key word from log is: libvirtError: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. Log ==================== 2018-07-31 18:20:01.464 30831 INFO nova.compute.manager [req-0f721cbe-f8d4-43fb-a4c5-944fe13baff4 d8aeb1794cbb47259ad126a7310c65ef 9f86737686af478487424ebb31ea2be6 - - -] [instanc e: 6470fe19-f58e-4bca-a8a8-0048165f2dc4] Successfully reverted task state from rebuild_spawning on failure for instance. 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher [req-0f721cbe-f8d4-43fb-a4c5-944fe13baff4 d8aeb1794cbb47259ad126a7310c65ef 9f86737686af478487424ebb31ea2be6 - - - ] Exception during message handling: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the rep ly, the reply timeout expired, or the network connection was broken. 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher incoming.message)) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher payload) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409, in decorated_function 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2809, in rebuild_instance 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher bdms, recreate, on_shared_storage, preserve_ephemeral) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2853, in _do_rebuild_instance_with_claim 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher self._do_rebuild_instance(*args, **kwargs) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2969, in _do_rebuild_instance 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher self._rebuild_default_impl(**kwargs) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2734, in _rebuild_default_impl 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher block_device_info=new_block_device_info) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2779, in spawn 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher block_device_info=block_device_info) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4923, in _create_domain_and_network 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher xml, pause=pause, power_on=power_on) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4854, in _create_domain 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher guest.launch(pause=pause) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 142, in launch 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher self._encoded_xml, errors='ignore') 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 137, in launch 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher return self._domain.createWithFlags(flags) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher six.reraise(c, e, tb) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2018-07-31 18:20:01.469 30831 ERROR oslo_messaging.rpc.dispatcher libvirtError: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.