Cinder

Lun linked to an unexist file when detach an encrypted volume

Bug #1493622 reported by YuZhang on 2015-09-09

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Cinder	New	Undecided	Unassigned

Bug Description

Lun linked to an unexist file when detach an encrypted volume, and then it would lead all volume attach failed with error unable to find a Fibre Channel volume device. Seems like the lun should be removed instead of linking to an unexist file
nova version: 2.28.1
cinder version :1.3.1
os-brick version: 0.4.0

Reproduce steps:
1. create a type and mark the volume type as encrypted
cinder type-create LUKS
cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
2. create an encrypted volume and attach to an instance
cinder create --display-name 'encrypted volume' --volume-type LUKS 1
nova volume-attach instance-id volume-id auto
3. Check the lun created:
under /dev/disk/by-path:
pci-0000:03:00.0-fc-0x500507680140a743-lun-0 -> /dev/mapper/pci-0000:03:00.0-fc-0x500507680140a743-lun-0
under /dev/mapper:
lrwxrwxrwx 1 root root 7 Sep 8 21:47 pci-0000:03:00.0-fc-0x500507680140a743-lun-0 -> ../dm-2
4. detach the volume: nova volume-detach instance-id volume-id
5. check the lun under /dev/disk/by-path, lun is still linked to another under /dev/mapper which already been removed. so it is marked as read when ls -al
6. create volume again and do attach, it would be failed and report error:
DEBUG os_brick.initiator.connector [^[[00;36m-^[[00;32m] ^[[00;32mLooking for Fibre Channel dev /dev/disk/by-path/pci-0000:03:00.0-fc-0x500507680140a743-lun-0 ^[[00;33mfrom (pid=19105) _wait_for_device_discovery /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py:981
ERROR os_brick.initiator.connector [^[[00;36m-] Fibre Channel volume device not found.
ERROR oslo.service.loopingcall [^[[00;36m-] Fixed interval looping call 'os_brick.initiator.connector._wait_for_device_discovery' failed
TRACE oslo.service.loopingcall Traceback (most recent call last):
TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
TRACE oslo.service.loopingcall result = func(*self.args, **self.kw)
TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 991, in _wait_for_device_discovery
TRACE oslo.service.loopingcall raise exception.NoFibreChannelVolumeDeviceFound()
TRACE oslo.service.loopingcall NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume device.
TRACE oslo.service.loopingcall
DEBUG oslo_concurrency.lockutils [^[[01;36mreq-37f0f319-b53e-4da7-a0f8-cf031a1fe727 ^[[00;36madmin admin^[[00;32m] ^[[00;32mLock "connect_volume" released by "os_brick.initiator.connector.connect_volume" :: held 10.638s ^[[00;33mfrom (pid=19105) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:265
ERROR nova.virt.block_device [^[[01;36mreq-37f0f319-b53e-4da7-a0f8-cf031a1fe727 ^[[00;36madmin admin] Driver failed to attach volume 7a7e1c42-1ca5-473d-a098-e5d307b55e1e at /dev/vdb
TRACE nova.virt.block_device Traceback (most recent call last):
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/block_device.py", line 256, in attach
TRACE nova.virt.block_device device_type=self['device_type'], encryption=encryption)
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1100, in attach_volume
TRACE nova.virt.block_device self._connect_volume(connection_info, disk_info)
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1050, in _connect_volume
TRACE nova.virt.block_device driver.connect_volume(connection_info, disk_info)
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/libvirt/volume/fibrechannel.py", line 54, in connect_volume
TRACE nova.virt.block_device device_info = self.connector.connect_volume(connection_info['data'])
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
TRACE nova.virt.block_device return f(*args, **kwargs)
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 1005, in connect_volume
TRACE nova.virt.block_device timer.start(interval=2).wait()
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
TRACE nova.virt.block_device return hubs.get_hub().switch()
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
TRACE nova.virt.block_device return self.greenlet.switch()
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
TRACE nova.virt.block_device result = func(*self.args, **self.kw)
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 991, in _wait_for_device_discovery
TRACE nova.virt.block_device raise exception.NoFibreChannelVolumeDeviceFound()
TRACE nova.virt.block_device NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume device.

See original description

Revision history for this message

YuZhang (ivysdu) wrote on 2015-09-10:

Download full text (3.9 KiB)

It also can reproduced with iSCSI protocol. ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device ERROR nova.virt.block_device
manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager ERROR nova.compute.manager Following is the error logs:
Traceback (most recent call last):
File "/opt/stack/new/nova/nova/virt/block_device.py", line 256, in attach
device_type=self['device_type'], encryption=encryption)
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1099, in attach_volume
self._connect_volume(connection_info, disk_info)
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1049, in _connect_volume
driver.connect_volume(connection_info, disk_info)
File "/opt/stack/new/nova/nova/virt/libvirt/volume/iscsi.py", line 84, in connect_volume
device_info = self.connector.connect_volume(connection_info['data'])
File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 499, in connect_volume
raise exception.VolumeDeviceNotFound(device=host_devices)
VolumeDeviceNotFound: Volume device not found at [u'/dev/disk/by-path/ip-9.11.101.17:3260-iscsi-iqn.1986-03.com.ibm:2145.arctbd3.arc64g00gf-2-lun-4'].
/> />manager [req-02370405-6fa4-414d-a588-39b84cf2caa0 tempest-TestEncryptedCinderVolumes-586258091 tempest-TestEncryptedCinderVolumes-19401] Failed to attach b70c823c-7992-445a-8d24-5668fe762818 at /dev/vdb
Traceback (most recent call last):
File "/opt/stack/new/nova/nova/compute/manager.py", line 4536, in _attach_volume
do_check_attach=False, do_driver_attach=True)
File "/opt/stack/new/nova/nova/virt/block_device.py", line 52, in wrapped
ret_val = method(obj, context, *args, **kwargs)
File "/opt/stack/new/nova/nova/virt/block_device.py", line 265, in attach
connector)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/new/nova/nova/virt/block_device.py", line 256, in attach
device_type=self['device_type'], encryption=encryption)
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1099, in attach_volume
self._connect_volume(connection_info, disk_info)
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1049, in _connect_volume
driver.connect_volume(connection_info, d...

It also can reproduced with iSCSI protocol.  Following is the error logs:
ERROR nova.virt.block_device  Traceback (most recent call last):
 ERROR nova.virt.block_device    File "/opt/stack/new/nova/nova/virt/block_device.py", line 256, in attach
 ERROR nova.virt.block_device      device_type=self['device_type'], encryption=encryption)
 ERROR nova.virt.block_device    File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1099, in attach_volume
 ERROR nova.virt.block_device      self._connect_volume(connection_info, disk_info)
 ERROR nova.virt.block_device    File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1049, in _connect_volume
 ERROR nova.virt.block_device      driver.connect_volume(connection_info, disk_info)
 ERROR nova.virt.block_device    File "/opt/stack/new/nova/nova/virt/libvirt/volume/iscsi.py", line 84, in connect_volume
 ERROR nova.virt.block_device      device_info = self.connector.connect_volume(connection_info['data'])
 ERROR nova.virt.block_device    File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
 ERROR nova.virt.block_device      return f(*args, **kwargs)
 ERROR nova.virt.block_device    File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 499, in connect_volume
 ERROR nova.virt.block_device      raise exception.VolumeDeviceNotFound(device=host_devices)
 ERROR nova.virt.block_device  VolumeDeviceNotFound: Volume device not found at [u'/dev/disk/by-path/ip-9.11.101.17:3260-iscsi-iqn.1986-03.com.ibm:2145.arctbd3.arc64g00gf-2-lun-4'].
 ERROR nova.virt.block_device 
 ERROR nova.compute.manager [req-02370405-6fa4-414d-a588-39b84cf2caa0 tempest-TestEncryptedCinderVolumes-586258091 tempest-TestEncryptedCinderVolumes-19401]  Failed to attach b70c823c-7992-445a-8d24-5668fe762818 at /dev/vdb
 ERROR nova.compute.manager  Traceback (most recent call last):
 ERROR nova.compute.manager    File "/opt/stack/new/nova/nova/compute/manager.py", line 4536, in _attach_volume
 ERROR nova.compute.manager      do_check_attach=False, do_driver_attach=True)
 ERROR nova.compute.manager    File "/opt/stack/new/nova/nova/virt/block_device.py", line 52, in wrapped
 ERROR nova.compute.manager      ret_val = method(obj, context, *args, **kwargs)
 ERROR nova.compute.manager    File "/opt/stack/new/nova/nova/virt/block_device.py", line 265, in attach
 ERROR nova.compute.manager      connector)
 ERROR nova.compute.manager    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
 ERROR nova.compute.manager      six.reraise(self.type_, self.value, self.tb)
 ERROR nova.compute.manager    File "/opt/stack/new/nova/nova/virt/block_device.py", line 256, in attach
 ERROR nova.compute.manager      device_type=self['device_type'], encryption=encryption)
 ERROR nova.compute.manager    File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1099, in attach_volume
 ERROR nova.compute.manager      self._connect_volume(connection_info, disk_info)
 ERROR nova.compute.manager    File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1049, in _connect_volume
 ERROR nova.compute.manager      driver.connect_volume(connection_info, disk_info)
 ERROR nova.compute.manager    File "/opt/stack/new/nova/nova/virt/libvirt/volume/iscsi.py", line 84, in connect_volume
 ERROR nova.compute.manager      device_info = self.connector.connect_volume(connection_info['data'])
 ERROR nova.compute.manager    File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
 ERROR nova.compute.manager      return f(*args, **kwargs)
 ERROR nova.compute.manager    File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 499, in connect_volume
 ERROR nova.compute.manager      raise exception.VolumeDeviceNotFound(device=host_devices)
 ERROR nova.compute.manager  VolumeDeviceNotFound: Volume device not found at [u'/dev/disk/by-path/ip-9.11.101.17:3260-iscsi-iqn.1986-03.com.ibm:2145.arctbd3.arc64g00gf-2-lun-4']

description:	updated
summary:	- Lun linked to an unexist file when detach a encrypted volume + Lun linked to an unexist file when detach an encrypted volume

zwei (suifeng20) on 2016-03-07

Changed in cinder:
assignee:	nobody → zwei (suifeng20)

Revision history for this message

Sean McGinnis (sean-mcginnis) wrote on 2016-10-07: Owner Expired

Unassigning due to no activity.

Changed in cinder:
assignee:	zwei (suifeng20) → nobody

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.