Failed to migrate encrypted volume

Bug #1522233 reported by Lisa Li on 2015-12-03
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Lisa Li

Bug Description

Openstack version:
From git.
stack@devstack1-hp:~/nova$ git log -n 3
commit 996c2f6f0543fa40fe34ca940bd56d8fd3f2e0f7
Merge: 4cf44ea e55d335
Author: Jenkins <email address hidden>
Date: Fri Nov 27 19:34:45 2015 +0000

    Merge "NFS setup for live-migration job"

Test steps:
1. Create an encrypted volume in Cinder.
2. Attach it to an VM.
3. Migrate the volume to another backend storage.

Expected result:
Migration succeeds.

Actual result:
Failed to rebase the device.

^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m self.volume_api.unreserve_volume(context, new_volume_id)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m six.reraise(self.type_, self.value, self.tb)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/nova/nova/compute/manager.py", line 4797, in _swap_volume
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m resize_to)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1209, in swap_volume
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m self._disconnect_volume(old_connection_info, disk_dev)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1068, in _disconnect_volume
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m driver.disconnect_volume(connection_info, disk_dev)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 93, in disconnect_volume
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m self.connector.disconnect_volume(connection_info['data'], None)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m return f(*args, **kwargs)

^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m return f(*args, **kwargs)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 743, in disconnect_volume
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m self._disconnect_volume_iscsi(props)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 764, in _disconnect_volume_iscsi
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m self._linuxscsi.wait_for_volume_removal(host_device)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/os-brick/os_brick/utils.py", line 54, in _wrapper
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m return r.call(f, *args, **kwargs)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m raise attempt.get()
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m six.reraise(self.value[0], self.value[1], self.value[2])
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/os-brick/os_brick/initiator/linuxscsi.py", line 82, in wait_for_volume_removal
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m volume_path=volume_path)
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mVolumePathNotRemoved: Volume path /dev/disk/by-path/ip-10.0.2.15:3260-iscsi-iqn.2010-10.org.openstack:volume-f6dcb40c-0477-43a9-8a58-4dd27deb0580-lun-1 was not removed in time.
^[[01;31m2015-12-03 02:30:30.155 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m

Analysis:
As this is a encrypted volume, it needs to detach encryptors.

Lisa Li (lisali) on 2015-12-03
Changed in nova:
assignee: nobody → Lisa Li (lisali)

Fix proposed to branch: master
Review: https://review.openstack.org/252809

Changed in nova:
status: New → In Progress
Lisa Li (lisali) on 2015-12-21
tags: added: migration
removed: encryption
Lisa Li (lisali) on 2016-01-19
tags: added: live-migration
removed: migration

Volume migration to another backend is not related to nova's live migration functionality

tags: removed: live-migration
Lisa Li (lisali) on 2016-01-25
tags: added: volumes
Lisa Li (lisali) on 2016-02-03
Changed in nova:
importance: Undecided → High
Lisa Li (lisali) on 2016-07-15
tags: added: encryption migration
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers