When doing a nova live migration os-brick fails to disconnect from encrypted volume on source node after instance has switched
2016-10-07 10:37:43.079 122207 DEBUG nova.virt.libvirt.guest [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Domain has shutdown/gone away: Requested operation is not valid: domain is not running get_job_info /home/pcarlton/openstack/nova/nova/virt/libvirt/guest.py:610
2016-10-07 10:37:43.082 122207 INFO nova.virt.libvirt.driver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Migration operation has completed
2016-10-07 10:37:43.083 122207 INFO nova.compute.manager [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] _post_live_migration() is started..
2016-10-07 10:37:43.086 122207 DEBUG oslo_messaging._drivers.amqpdriver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] CALL msg_id: 7764cbb2fb3145ecb19182484d80b020 exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-10-07 10:37:43.088 122207 DEBUG nova.virt.libvirt.driver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Migration operation thread has finished _live_migration_operation /home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py:5954
2016-10-07 10:37:43.089 122207 DEBUG nova.virt.libvirt.driver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Migration operation thread notification thread_finished /home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py:6310
2016-10-07 10:37:43.113 122207 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 7764cbb2fb3145ecb19182484d80b020 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-10-07 10:37:43.115 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] ==> get_connector_properties: call {'execute': None, 'my_ip': '192.168.16.21', 'enforce_multipath': True, 'host': 'devstack-hlinux-c1', 'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'multipath': False} trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:141
2016-10-07 10:37:43.116 122207 INFO oslo.privsep.daemon [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpFk8QjN/privsep.sock']
2016-10-07 10:37:43.767 122207 INFO oslo.privsep.daemon [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Spawned new privsep daemon via rootwrap
2016-10-07 10:37:43.768 122207 DEBUG oslo.privsep.daemon [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Accepted privsep connection to /tmp/tmpFk8QjN/privsep.sock __init__ /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:313
2016-10-07 10:37:43.769 122207 INFO oslo.privsep.daemon [-] privsep daemon starting
2016-10-07 10:37:43.769 122207 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2016-10-07 10:37:43.769 122207 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
2016-10-07 10:37:43.769 122207 INFO oslo.privsep.daemon [-] privsep daemon running as pid 122351
2016-10-07 10:37:43.770 122207 DEBUG oslo.privsep.daemon [-] privsep: request[140188873531216]: (1,) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.770 122207 DEBUG oslo.privsep.daemon [-] privsep: reply[140188873531216]: (2,) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.771 122207 DEBUG oslo.privsep.daemon [-] privsep: request[140188873531216]: (3, 'os_brick.privileged.rootwrap.execute_root', ('cat', '/etc/iscsi/initiatorname.iscsi'), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.800 122207 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.811 122207 DEBUG oslo.privsep.daemon [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.010s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.811 122207 DEBUG oslo.privsep.daemon [-] privsep: reply[140188873531216]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:d6a93bd1597f\n', '')) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.813 122207 DEBUG oslo.privsep.daemon [-] privsep: request[140188873531216]: (3, 'os_brick.privileged.rootwrap.execute_root', ('systool', '-c', 'fc_host', '-v'), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.814 122207 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): systool -c fc_host -v out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.826 122207 DEBUG oslo.privsep.daemon [-] CMD "systool -c fc_host -v" returned: 1 in 0.012s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.827 122207 DEBUG oslo.privsep.daemon [-] u'systool -c fc_host -v' failed. Not Retrying. out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.827 122207 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140188873531216]: Unexpected error while running command.
Command: systool -c fc_host -v
Exit code: 1
Stdout: u''
Stderr: u'Error opening class fc_host\n' out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.828 122207 DEBUG oslo.privsep.daemon [-] privsep: reply[140188873531216]: (5, 'oslo_concurrency.processutils.ProcessExecutionError', (u'', u'Error opening class fc_host\n', 1, u'systool -c fc_host -v', None)) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.829 122207 DEBUG oslo.privsep.daemon [-] privsep: request[140188873531216]: (3, 'os_brick.privileged.rootwrap.execute_root', ('systool', '-c', 'fc_host', '-v'), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.829 122207 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): systool -c fc_host -v out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.838 122207 DEBUG oslo.privsep.daemon [-] CMD "systool -c fc_host -v" returned: 1 in 0.009s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.839 122207 DEBUG oslo.privsep.daemon [-] u'systool -c fc_host -v' failed. Not Retrying. out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.839 122207 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140188873531216]: Unexpected error while running command.
Command: systool -c fc_host -v
Exit code: 1
Stdout: u''
Stderr: u'Error opening class fc_host\n' out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.839 122207 DEBUG oslo.privsep.daemon [-] privsep: reply[140188873531216]: (5, 'oslo_concurrency.processutils.ProcessExecutionError', (u'', u'Error opening class fc_host\n', 1, u'systool -c fc_host -v', None)) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.840 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] <== get_connector_properties: return (723ms) {'initiator': u'iqn.1993-08.org.debian:01:d6a93bd1597f', 'ip': u'192.168.16.21', 'platform': u'x86_64', 'host': u'devstack-hlinux-c1', 'do_local_attach': False, 'os_type': u'linux2', 'multipath': False} trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:165
2016-10-07 10:37:43.854 122207 DEBUG cinderclient.v2.client [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] REQ: curl -g -i -X POST http://192.168.16.20:8776/v2/97539967182441828f42ecfd1a178ead/volumes/ccab3050-b9ed-4ae8-82d7-c823da6d4e9c/action -H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}07f4c14552ff964e6d004b61bb3ea0c9855838bb" -d '{"os-initialize_connection": {"connector": {"platform": "x86_64", "host": "devstack-hlinux-c1", "do_local_attach": false, "ip": "192.168.16.21", "os_type": "linux2", "multipath": false, "initiator": "iqn.1993-08.org.debian:01:d6a93bd1597f"}}}' _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:337
2016-10-07 10:37:46.337 122207 DEBUG cinderclient.v2.client [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] RESP: [200] X-Compute-Request-Id: req-e9dd066a-a75b-47fd-960d-72a18b030d15 Content-Type: application/json Content-Length: 446 X-Openstack-Request-Id: req-e9dd066a-a75b-47fd-960d-72a18b030d15 Date: Fri, 07 Oct 2016 09:37:46 GMT Connection: keep-alive
RESP BODY: {"connection_info": {"driver_volume_type": "iscsi", "data": {"auth_password": "7VgCEnU5dvDRfvPu", "target_discovered": false, "encrypted": true, "qos_specs": null, "target_iqn": "iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c", "target_portal": "192.168.16.20:3260", "volume_id": "ccab3050-b9ed-4ae8-82d7-c823da6d4e9c", "target_lun": 1, "access_mode": "rw", "auth_username": "d47tzjsH778ifRpoSgvi", "auth_method": "CHAP"}}}
_http_log_response /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:366
2016-10-07 10:37:46.338 122207 DEBUG cinderclient.client [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] POST call to cinderv2 for http://192.168.16.20:8776/v2/97539967182441828f42ecfd1a178ead/volumes/ccab3050-b9ed-4ae8-82d7-c823da6d4e9c/action used request id req-e9dd066a-a75b-47fd-960d-72a18b030d15 _log_request_id /usr/local/lib/python2.7/dist-packages/cinderclient/client.py:95
2016-10-07 10:37:46.338 122207 DEBUG nova.virt.libvirt.volume.iscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] calling os-brick to detach iSCSI Volume disconnect_volume /home/pcarlton/openstack/nova/nova/virt/libvirt/volume/iscsi.py:72
2016-10-07 10:37:46.339 122207 DEBUG os_brick.initiator.connectors.iscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] ==> disconnect_volume: call {'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f804c35b410>, {u'access_mode': u'rw', u'target_discovered': False, u'encrypted': True, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c', u'target_portal': u'192.168.16.20:3260', u'volume_id': u'ccab3050-b9ed-4ae8-82d7-c823da6d4e9c', u'target_lun': 1, u'auth_password': u'7VgCEnU5dvDRfvPu', u'auth_username': u'd47tzjsH778ifRpoSgvi', u'auth_method': u'CHAP'}, None), 'kwargs': {}} trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:141
2016-10-07 10:37:46.340 122207 DEBUG oslo_concurrency.lockutils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: waited 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:270
2016-10-07 10:37:46.340 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Checking to see if SCSI volume /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 has been removed. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:79
2016-10-07 10:37:46.341 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 still exists. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:81
2016-10-07 10:37:46.342 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Failed attempt 1 _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:45
2016-10-07 10:37:46.343 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Have been at this for 0.002 seconds _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:47
2016-10-07 10:37:46.345 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Sleeping for 2 seconds _backoff_sleep /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:40
2016-10-07 10:37:48.345 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Checking to see if SCSI volume /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 has been removed. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:79
2016-10-07 10:37:48.346 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 still exists. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:81
2016-10-07 10:37:48.346 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Failed attempt 2 _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:45
2016-10-07 10:37:48.346 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Have been at this for 2.005 seconds _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:47
2016-10-07 10:37:48.346 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Sleeping for 4 seconds _backoff_sleep /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:40
2016-10-07 10:37:52.347 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Checking to see if SCSI volume /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 has been removed. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:79
2016-10-07 10:37:52.347 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 still exists. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:81
2016-10-07 10:37:52.348 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Failed attempt 3 _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:45
2016-10-07 10:37:52.348 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Have been at this for 6.007 seconds _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:47
2016-10-07 10:37:52.349 122207 DEBUG oslo_concurrency.lockutils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: held 6.009s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282
2016-10-07 10:37:52.349 122207 DEBUG os_brick.initiator.connectors.iscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] <== disconnect_volume: exception (6009ms) VolumePathNotRemoved(u'Volume path /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 was not removed in time.',) trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:151
2016-10-07 10:37:52.353 122207 DEBUG oslo_messaging._drivers.amqpdriver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] CALL msg_id: 8adc245b29544db19ff9ed6695d94d20 exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-10-07 10:37:52.380 122207 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 8adc245b29544db19ff9ed6695d94d20 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-10-07 10:37:52.405 122207 DEBUG oslo_messaging._drivers.amqpdriver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] CAST unique_id: 4bd048c59e5948808c6cfa29fdede479 NOTIFY exchange 'nova' topic 'notifications.error' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:432
2016-10-07 10:37:52.446 122207 DEBUG oslo_messaging._drivers.amqpdriver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] CAST unique_id: 921b79201de74b798a2302e96289e130 NOTIFY exchange 'nova' topic 'versioned_notifications.error' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:432
2016-10-07 10:37:52.457 122207 WARNING nova.virt.libvirt.driver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Error monitoring migration: Volume path /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 was not removed in time.
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Traceback (most recent call last):
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6323, in _live_migration
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] finish_event, disk_paths)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6233, in _live_migration_monitor
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] migrate_data)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 75, in wrapped
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] function_name, call_dict, binary)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.force_reraise()
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.type_, self.value, self.tb)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 66, in wrapped
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return f(self, context, *args, **kw)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 216, in decorated_function
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] kwargs['instance'], e, sys.exc_info())
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.force_reraise()
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.type_, self.value, self.tb)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 204, in decorated_function
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return function(self, context, *args, **kwargs)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5402, in _post_live_migration
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] migrate_data)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6668, in post_live_migration
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._disconnect_volume(connection_info, disk_dev)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 1079, in _disconnect_volume
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] vol_driver.disconnect_volume(connection_info, disk_dev)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.connector.disconnect_volume(connection_info['data'], None)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 145, in trace_logging_wrapper
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] result = f(*args, **kwargs)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return f(*args, **kwargs)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 497, in disconnect_volume
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._disconnect_volume_iscsi(props)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 518, in _disconnect_volume_iscsi
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._linuxscsi.wait_for_volume_removal(host_device)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 61, in _wrapper
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return r.call(f, *args, **kwargs)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] raise attempt.get()
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.value[0], self.value[1], self.value[2])
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py", line 83, in wait_for_volume_removal
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] volume_path=volume_path)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] VolumePathNotRemoved: Volume path /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 was not removed in time.
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0]
2016-10-07 10:37:52.459 122207 DEBUG nova.virt.libvirt.driver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Live migration monitoring is all done _live_migration /home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py:6330
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Live migration failed.
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Traceback (most recent call last):
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5253, in _do_live_migration
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] block_migration, migrate_data)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 5761, in live_migration
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] migrate_data)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6323, in _live_migration
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] finish_event, disk_paths)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6233, in _live_migration_monitor
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] migrate_data)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 75, in wrapped
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] function_name, call_dict, binary)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.force_reraise()
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.type_, self.value, self.tb)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 66, in wrapped
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return f(self, context, *args, **kw)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 216, in decorated_function
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] kwargs['instance'], e, sys.exc_info())
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.force_reraise()
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.type_, self.value, self.tb)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 204, in decorated_function
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return function(self, context, *args, **kwargs)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5402, in _post_live_migration
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] migrate_data)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6668, in post_live_migration
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._disconnect_volume(connection_info, disk_dev)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 1079, in _disconnect_volume
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] vol_driver.disconnect_volume(connection_info, disk_dev)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.connector.disconnect_volume(connection_info['data'], None)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 145, in trace_logging_wrapper
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] result = f(*args, **kwargs)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return f(*args, **kwargs)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 497, in disconnect_volume
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._disconnect_volume_iscsi(props)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 518, in _disconnect_volume_iscsi
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._linuxscsi.wait_for_volume_removal(host_device)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 61, in _wrapper
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return r.call(f, *args, **kwargs)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] raise attempt.get()
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.value[0], self.value[1], self.value[2])
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py", line 83, in wait_for_volume_removal
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] volume_path=volume_path)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] VolumePathNotRemoved: Volume path /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 was not removed in time.
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0]
2016-10-07 10:37:52.462 122207 DEBUG oslo_messaging._drivers.amqpdriver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] CALL msg_id: f5f06b55fba245a1856f8c052b784a26 exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-10-07 10:37:52.499 122207 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: f5f06b55fba245a1856f8c052b784a26 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/queue.py", line 118, in switch
self.greenlet.switch(value)
File "/home/pcarlton/openstack/nova/nova/utils.py", line 1089, in context_wrapper
func(*args, **kwargs)
File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5281, in dispatch_live_migration
self._do_live_migration(*args, **kwargs)
File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5260, in _do_live_migration
self._set_migration_status(migration, 'error')
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
self.force_reraise()
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5253, in _do_live_migration
block_migration, migrate_data)
File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 5761, in live_migration
migrate_data)
File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6323, in _live_migration
finish_event, disk_paths)
File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6233, in _live_migration_monitor
migrate_data)
File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 75, in wrapped
function_name, call_dict, binary)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
self.force_reraise()
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 66, in wrapped
return f(self, context, *args, **kw)
File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 216, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
self.force_reraise()
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 204, in decorated_function
return function(self, context, *args, **kwargs)
File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5402, in _post_live_migration
migrate_data)
File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6668, in post_live_migration
self._disconnect_volume(connection_info, disk_dev)
File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 1079, in _disconnect_volume
vol_driver.disconnect_volume(connection_info, disk_dev)
File "/home/pcarlton/openstack/nova/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
self.connector.disconnect_volume(connection_info['data'], None)
File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 145, in trace_logging_wrapper
result = f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 497, in disconnect_volume
self._disconnect_volume_iscsi(props)
File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 518, in _disconnect_volume_iscsi
self._linuxscsi.wait_for_volume_removal(host_device)
File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 61, in _wrapper
return r.call(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py", line 83, in wait_for_volume_removal
volume_path=volume_path)
VolumePathNotRemoved: Volume path /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 was not removed in time.
this seems to be a nova issue, see https:/