unable to disconnect encrypted volume

Bug #1631318 reported by Paul Carlton
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
os-brick
Invalid
Undecided
Unassigned

Bug Description

When doing a nova live migration os-brick fails to disconnect from encrypted volume on source node after instance has switched

2016-10-07 10:37:43.079 122207 DEBUG nova.virt.libvirt.guest [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Domain has shutdown/gone away: Requested operation is not valid: domain is not running get_job_info /home/pcarlton/openstack/nova/nova/virt/libvirt/guest.py:610
2016-10-07 10:37:43.082 122207 INFO nova.virt.libvirt.driver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Migration operation has completed
2016-10-07 10:37:43.083 122207 INFO nova.compute.manager [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] _post_live_migration() is started..
2016-10-07 10:37:43.086 122207 DEBUG oslo_messaging._drivers.amqpdriver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] CALL msg_id: 7764cbb2fb3145ecb19182484d80b020 exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-10-07 10:37:43.088 122207 DEBUG nova.virt.libvirt.driver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Migration operation thread has finished _live_migration_operation /home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py:5954
2016-10-07 10:37:43.089 122207 DEBUG nova.virt.libvirt.driver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Migration operation thread notification thread_finished /home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py:6310
2016-10-07 10:37:43.113 122207 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 7764cbb2fb3145ecb19182484d80b020 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-10-07 10:37:43.115 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] ==> get_connector_properties: call {'execute': None, 'my_ip': '192.168.16.21', 'enforce_multipath': True, 'host': 'devstack-hlinux-c1', 'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'multipath': False} trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:141
2016-10-07 10:37:43.116 122207 INFO oslo.privsep.daemon [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpFk8QjN/privsep.sock']
2016-10-07 10:37:43.767 122207 INFO oslo.privsep.daemon [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Spawned new privsep daemon via rootwrap
2016-10-07 10:37:43.768 122207 DEBUG oslo.privsep.daemon [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Accepted privsep connection to /tmp/tmpFk8QjN/privsep.sock __init__ /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:313
2016-10-07 10:37:43.769 122207 INFO oslo.privsep.daemon [-] privsep daemon starting
2016-10-07 10:37:43.769 122207 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2016-10-07 10:37:43.769 122207 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
2016-10-07 10:37:43.769 122207 INFO oslo.privsep.daemon [-] privsep daemon running as pid 122351
2016-10-07 10:37:43.770 122207 DEBUG oslo.privsep.daemon [-] privsep: request[140188873531216]: (1,) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.770 122207 DEBUG oslo.privsep.daemon [-] privsep: reply[140188873531216]: (2,) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.771 122207 DEBUG oslo.privsep.daemon [-] privsep: request[140188873531216]: (3, 'os_brick.privileged.rootwrap.execute_root', ('cat', '/etc/iscsi/initiatorname.iscsi'), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.800 122207 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.811 122207 DEBUG oslo.privsep.daemon [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.010s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.811 122207 DEBUG oslo.privsep.daemon [-] privsep: reply[140188873531216]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:d6a93bd1597f\n', '')) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.813 122207 DEBUG oslo.privsep.daemon [-] privsep: request[140188873531216]: (3, 'os_brick.privileged.rootwrap.execute_root', ('systool', '-c', 'fc_host', '-v'), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.814 122207 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): systool -c fc_host -v out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.826 122207 DEBUG oslo.privsep.daemon [-] CMD "systool -c fc_host -v" returned: 1 in 0.012s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.827 122207 DEBUG oslo.privsep.daemon [-] u'systool -c fc_host -v' failed. Not Retrying. out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.827 122207 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140188873531216]: Unexpected error while running command.
Command: systool -c fc_host -v
Exit code: 1
Stdout: u''
Stderr: u'Error opening class fc_host\n' out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.828 122207 DEBUG oslo.privsep.daemon [-] privsep: reply[140188873531216]: (5, 'oslo_concurrency.processutils.ProcessExecutionError', (u'', u'Error opening class fc_host\n', 1, u'systool -c fc_host -v', None)) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.829 122207 DEBUG oslo.privsep.daemon [-] privsep: request[140188873531216]: (3, 'os_brick.privileged.rootwrap.execute_root', ('systool', '-c', 'fc_host', '-v'), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.829 122207 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): systool -c fc_host -v out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.838 122207 DEBUG oslo.privsep.daemon [-] CMD "systool -c fc_host -v" returned: 1 in 0.009s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.839 122207 DEBUG oslo.privsep.daemon [-] u'systool -c fc_host -v' failed. Not Retrying. out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.839 122207 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140188873531216]: Unexpected error while running command.
Command: systool -c fc_host -v
Exit code: 1
Stdout: u''
Stderr: u'Error opening class fc_host\n' out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.839 122207 DEBUG oslo.privsep.daemon [-] privsep: reply[140188873531216]: (5, 'oslo_concurrency.processutils.ProcessExecutionError', (u'', u'Error opening class fc_host\n', 1, u'systool -c fc_host -v', None)) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-10-07 10:37:43.840 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] <== get_connector_properties: return (723ms) {'initiator': u'iqn.1993-08.org.debian:01:d6a93bd1597f', 'ip': u'192.168.16.21', 'platform': u'x86_64', 'host': u'devstack-hlinux-c1', 'do_local_attach': False, 'os_type': u'linux2', 'multipath': False} trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:165
2016-10-07 10:37:43.854 122207 DEBUG cinderclient.v2.client [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] REQ: curl -g -i -X POST http://192.168.16.20:8776/v2/97539967182441828f42ecfd1a178ead/volumes/ccab3050-b9ed-4ae8-82d7-c823da6d4e9c/action -H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}07f4c14552ff964e6d004b61bb3ea0c9855838bb" -d '{"os-initialize_connection": {"connector": {"platform": "x86_64", "host": "devstack-hlinux-c1", "do_local_attach": false, "ip": "192.168.16.21", "os_type": "linux2", "multipath": false, "initiator": "iqn.1993-08.org.debian:01:d6a93bd1597f"}}}' _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:337
2016-10-07 10:37:46.337 122207 DEBUG cinderclient.v2.client [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] RESP: [200] X-Compute-Request-Id: req-e9dd066a-a75b-47fd-960d-72a18b030d15 Content-Type: application/json Content-Length: 446 X-Openstack-Request-Id: req-e9dd066a-a75b-47fd-960d-72a18b030d15 Date: Fri, 07 Oct 2016 09:37:46 GMT Connection: keep-alive
RESP BODY: {"connection_info": {"driver_volume_type": "iscsi", "data": {"auth_password": "7VgCEnU5dvDRfvPu", "target_discovered": false, "encrypted": true, "qos_specs": null, "target_iqn": "iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c", "target_portal": "192.168.16.20:3260", "volume_id": "ccab3050-b9ed-4ae8-82d7-c823da6d4e9c", "target_lun": 1, "access_mode": "rw", "auth_username": "d47tzjsH778ifRpoSgvi", "auth_method": "CHAP"}}}
 _http_log_response /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:366
2016-10-07 10:37:46.338 122207 DEBUG cinderclient.client [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] POST call to cinderv2 for http://192.168.16.20:8776/v2/97539967182441828f42ecfd1a178ead/volumes/ccab3050-b9ed-4ae8-82d7-c823da6d4e9c/action used request id req-e9dd066a-a75b-47fd-960d-72a18b030d15 _log_request_id /usr/local/lib/python2.7/dist-packages/cinderclient/client.py:95
2016-10-07 10:37:46.338 122207 DEBUG nova.virt.libvirt.volume.iscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] calling os-brick to detach iSCSI Volume disconnect_volume /home/pcarlton/openstack/nova/nova/virt/libvirt/volume/iscsi.py:72
2016-10-07 10:37:46.339 122207 DEBUG os_brick.initiator.connectors.iscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] ==> disconnect_volume: call {'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f804c35b410>, {u'access_mode': u'rw', u'target_discovered': False, u'encrypted': True, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c', u'target_portal': u'192.168.16.20:3260', u'volume_id': u'ccab3050-b9ed-4ae8-82d7-c823da6d4e9c', u'target_lun': 1, u'auth_password': u'7VgCEnU5dvDRfvPu', u'auth_username': u'd47tzjsH778ifRpoSgvi', u'auth_method': u'CHAP'}, None), 'kwargs': {}} trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:141
2016-10-07 10:37:46.340 122207 DEBUG oslo_concurrency.lockutils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: waited 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:270
2016-10-07 10:37:46.340 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Checking to see if SCSI volume /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 has been removed. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:79
2016-10-07 10:37:46.341 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 still exists. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:81
2016-10-07 10:37:46.342 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Failed attempt 1 _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:45
2016-10-07 10:37:46.343 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Have been at this for 0.002 seconds _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:47
2016-10-07 10:37:46.345 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Sleeping for 2 seconds _backoff_sleep /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:40
2016-10-07 10:37:48.345 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Checking to see if SCSI volume /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 has been removed. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:79
2016-10-07 10:37:48.346 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 still exists. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:81
2016-10-07 10:37:48.346 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Failed attempt 2 _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:45
2016-10-07 10:37:48.346 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Have been at this for 2.005 seconds _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:47
2016-10-07 10:37:48.346 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Sleeping for 4 seconds _backoff_sleep /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:40
2016-10-07 10:37:52.347 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Checking to see if SCSI volume /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 has been removed. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:79
2016-10-07 10:37:52.347 122207 DEBUG os_brick.initiator.linuxscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 still exists. wait_for_volume_removal /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:81
2016-10-07 10:37:52.348 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Failed attempt 3 _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:45
2016-10-07 10:37:52.348 122207 DEBUG os_brick.utils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Have been at this for 6.007 seconds _print_stop /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:47
2016-10-07 10:37:52.349 122207 DEBUG oslo_concurrency.lockutils [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: held 6.009s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282
2016-10-07 10:37:52.349 122207 DEBUG os_brick.initiator.connectors.iscsi [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] <== disconnect_volume: exception (6009ms) VolumePathNotRemoved(u'Volume path /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 was not removed in time.',) trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:151
2016-10-07 10:37:52.353 122207 DEBUG oslo_messaging._drivers.amqpdriver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] CALL msg_id: 8adc245b29544db19ff9ed6695d94d20 exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-10-07 10:37:52.380 122207 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 8adc245b29544db19ff9ed6695d94d20 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-10-07 10:37:52.405 122207 DEBUG oslo_messaging._drivers.amqpdriver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] CAST unique_id: 4bd048c59e5948808c6cfa29fdede479 NOTIFY exchange 'nova' topic 'notifications.error' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:432
2016-10-07 10:37:52.446 122207 DEBUG oslo_messaging._drivers.amqpdriver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] CAST unique_id: 921b79201de74b798a2302e96289e130 NOTIFY exchange 'nova' topic 'versioned_notifications.error' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:432
2016-10-07 10:37:52.457 122207 WARNING nova.virt.libvirt.driver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Error monitoring migration: Volume path /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 was not removed in time.
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Traceback (most recent call last):
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6323, in _live_migration
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] finish_event, disk_paths)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6233, in _live_migration_monitor
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] migrate_data)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 75, in wrapped
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] function_name, call_dict, binary)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.force_reraise()
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.type_, self.value, self.tb)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 66, in wrapped
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return f(self, context, *args, **kw)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 216, in decorated_function
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] kwargs['instance'], e, sys.exc_info())
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.force_reraise()
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.type_, self.value, self.tb)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 204, in decorated_function
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return function(self, context, *args, **kwargs)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5402, in _post_live_migration
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] migrate_data)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6668, in post_live_migration
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._disconnect_volume(connection_info, disk_dev)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 1079, in _disconnect_volume
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] vol_driver.disconnect_volume(connection_info, disk_dev)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.connector.disconnect_volume(connection_info['data'], None)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 145, in trace_logging_wrapper
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] result = f(*args, **kwargs)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return f(*args, **kwargs)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 497, in disconnect_volume
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._disconnect_volume_iscsi(props)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 518, in _disconnect_volume_iscsi
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._linuxscsi.wait_for_volume_removal(host_device)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 61, in _wrapper
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return r.call(f, *args, **kwargs)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] raise attempt.get()
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.value[0], self.value[1], self.value[2])
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py", line 83, in wait_for_volume_removal
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] volume_path=volume_path)
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] VolumePathNotRemoved: Volume path /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 was not removed in time.
2016-10-07 10:37:52.457 122207 ERROR nova.virt.libvirt.driver [instance: 9baf6405-46f6-4907-82ee-e892be6415e0]
2016-10-07 10:37:52.459 122207 DEBUG nova.virt.libvirt.driver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Live migration monitoring is all done _live_migration /home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py:6330
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Live migration failed.
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] Traceback (most recent call last):
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5253, in _do_live_migration
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] block_migration, migrate_data)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 5761, in live_migration
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] migrate_data)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6323, in _live_migration
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] finish_event, disk_paths)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6233, in _live_migration_monitor
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] migrate_data)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 75, in wrapped
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] function_name, call_dict, binary)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.force_reraise()
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.type_, self.value, self.tb)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 66, in wrapped
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return f(self, context, *args, **kw)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 216, in decorated_function
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] kwargs['instance'], e, sys.exc_info())
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.force_reraise()
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.type_, self.value, self.tb)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 204, in decorated_function
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return function(self, context, *args, **kwargs)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5402, in _post_live_migration
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] migrate_data)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6668, in post_live_migration
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._disconnect_volume(connection_info, disk_dev)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 1079, in _disconnect_volume
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] vol_driver.disconnect_volume(connection_info, disk_dev)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/home/pcarlton/openstack/nova/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self.connector.disconnect_volume(connection_info['data'], None)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 145, in trace_logging_wrapper
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] result = f(*args, **kwargs)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return f(*args, **kwargs)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 497, in disconnect_volume
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._disconnect_volume_iscsi(props)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 518, in _disconnect_volume_iscsi
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] self._linuxscsi.wait_for_volume_removal(host_device)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 61, in _wrapper
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] return r.call(f, *args, **kwargs)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] raise attempt.get()
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] six.reraise(self.value[0], self.value[1], self.value[2])
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py", line 83, in wait_for_volume_removal
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] volume_path=volume_path)
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0] VolumePathNotRemoved: Volume path /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 was not removed in time.
2016-10-07 10:37:52.460 122207 ERROR nova.compute.manager [instance: 9baf6405-46f6-4907-82ee-e892be6415e0]
2016-10-07 10:37:52.462 122207 DEBUG oslo_messaging._drivers.amqpdriver [req-6fc68e2b-3a5b-40f9-b031-16e1c5a3b37b admin admin] CALL msg_id: f5f06b55fba245a1856f8c052b784a26 exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-10-07 10:37:52.499 122207 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: f5f06b55fba245a1856f8c052b784a26 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/queue.py", line 118, in switch
    self.greenlet.switch(value)
  File "/home/pcarlton/openstack/nova/nova/utils.py", line 1089, in context_wrapper
    func(*args, **kwargs)
  File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5281, in dispatch_live_migration
    self._do_live_migration(*args, **kwargs)
  File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5260, in _do_live_migration
    self._set_migration_status(migration, 'error')
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5253, in _do_live_migration
    block_migration, migrate_data)
  File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 5761, in live_migration
    migrate_data)
  File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6323, in _live_migration
    finish_event, disk_paths)
  File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6233, in _live_migration_monitor
    migrate_data)
  File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 75, in wrapped
    function_name, call_dict, binary)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/home/pcarlton/openstack/nova/nova/exception_wrapper.py", line 66, in wrapped
    return f(self, context, *args, **kw)
  File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 216, in decorated_function
    kwargs['instance'], e, sys.exc_info())
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 204, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/home/pcarlton/openstack/nova/nova/compute/manager.py", line 5402, in _post_live_migration
    migrate_data)
  File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 6668, in post_live_migration
    self._disconnect_volume(connection_info, disk_dev)
  File "/home/pcarlton/openstack/nova/nova/virt/libvirt/driver.py", line 1079, in _disconnect_volume
    vol_driver.disconnect_volume(connection_info, disk_dev)
  File "/home/pcarlton/openstack/nova/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
    self.connector.disconnect_volume(connection_info['data'], None)
  File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 145, in trace_logging_wrapper
    result = f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 497, in disconnect_volume
    self._disconnect_volume_iscsi(props)
  File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 518, in _disconnect_volume_iscsi
    self._linuxscsi.wait_for_volume_removal(host_device)
  File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 61, in _wrapper
    return r.call(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py", line 83, in wait_for_volume_removal
    volume_path=volume_path)
VolumePathNotRemoved: Volume path /dev/disk/by-path/ip-192.168.16.20:3260-iscsi-iqn.2010-10.org.openstack:volume-ccab3050-b9ed-4ae8-82d7-c823da6d4e9c-lun-1 was not removed in time.

Revision history for this message
Paul Carlton (paul-carlton2) wrote :

this seems to be a nova issue, see https://bugs.launchpad.net/nova/+bug/1633033

Changed in os-brick:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.