Paste-bins expire, so copy/pasting content from the paste-bin (http://paste.openstack.org/show/798201/) in the description below: -------------------------------------------- 2020-09-21 11:12:27.538 3887 INFO nova.compute.manager [req-1363655f-fa54-4674-a5ae-d1d26d2ea86d 473593595cf04e41a6382d261eea3576 19151bb269ae4214bdf83e18083e8b54 - default default] [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] Swapping volume ed93fa66-13d0-447e-8084-4fd523cf3015 for 1bde1669-657e-4608-a8c6-b654ef58121f 2020-09-21 11:12:28.979 3887 INFO os_brick.initiator.connectors.iscsi [req-1363655f-fa54-4674-a5ae-d1d26d2ea86d 473593595cf04e41a6382d261eea3576 19151bb269ae4214bdf83e18083e8b54 - default default] Trying to connect to iSCSI portal 172.20.213.204:3260 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [req-1363655f-fa54-4674-a5ae-d1d26d2ea86d 473593595cf04e41a6382d261eea3576 19151bb269ae4214bdf83e18083e8b54 - default default] [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] Failed to swap volume ed93fa66-13d0-447e-8084-4fd523cf3015 for 1bde1669-657e-4608-a8c6-b654ef58121f: libvirtError: block copy still active: domain has active block job 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] Traceback (most recent call last): 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/compute/manager.py", line 5726, in _swap_volume 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] mountpoint, resize_to) 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1568, in swap_volume 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] self._swap_volume(guest, disk_dev, conf, resize_to) 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1530, in _swap_volume 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] self._host.write_instance_config(xml) 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 864, in write_instance_config 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] domain = self.get_connection().defineXML(xml) 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] result = proxy_call(self._autowrap, f, *args, **kwargs) 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] rv = execute(f, *args, **kwargs) 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] six.reraise(c, e, tb) 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] rv = meth(*args, **kwargs) 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/libvirt.py", line 3621, in defineXML 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self) 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] libvirtError: block copy still active: domain has active block job 2020-09-21 11:13:56.292 3887 ERROR nova.compute.manager [instance: cd879458-57d9-4f4c-8214-1f99456b6f42] 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server [req-1363655f-fa54-4674-a5ae-d1d26d2ea86d 473593595cf04e41a6382d261eea3576 19151bb269ae4214bdf83e18083e8b54 - default default] Exception during message handling: libvirtError: block copy still active: domain has active block job 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server self.force_reraise() 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in decorated_function 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server self.force_reraise() 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/compute/manager.py", line 5871, in swap_volume 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server is_cinder_migration) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/compute/manager.py", line 5769, in _swap_volume 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server context, new_attachment_id) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server self.force_reraise() 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/compute/manager.py", line 5726, in _swap_volume 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server mountpoint, resize_to) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1568, in swap_volume 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server self._swap_volume(guest, disk_dev, conf, resize_to) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1530, in _swap_volume 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server self._host.write_instance_config(xml) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 864, in write_instance_config 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server domain = self.get_connection().defineXML(xml) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/libvirt.py", line 3621, in defineXML 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self) 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server libvirtError: block copy still active: domain has active block job 2020-09-21 11:13:59.930 3887 ERROR oslo_messaging.rpc.server -------------------------------------------- (That said, as discussed on IRC, for debugging libvirt/QEMU interactions, you might want to refer to: https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt) But I see that Lee has got a handle on the 'nova-compute' side of things.