Lun linked to an unexist file when detach an encrypted volume, and then it would lead all volume attach failed with error unable to find a Fibre Channel volume device. Seems like the lun should be removed instead of linking to an unexist file
nova version: 2.28.1
cinder version :1.3.1
os-brick version: 0.4.0
Reproduce steps:
1. create a type and mark the volume type as encrypted
cinder type-create LUKS
cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
2. create an encrypted volume and attach to an instance
cinder create --display-name 'encrypted volume' --volume-type LUKS 1
nova volume-attach instance-id volume-id auto
3. Check the lun created:
under /dev/disk/by-path:
pci-0000:03:00.0-fc-0x500507680140a743-lun-0 -> /dev/mapper/pci-0000:03:00.0-fc-0x500507680140a743-lun-0
under /dev/mapper:
lrwxrwxrwx 1 root root 7 Sep 8 21:47 pci-0000:03:00.0-fc-0x500507680140a743-lun-0 -> ../dm-2
4. detach the volume: nova volume-detach instance-id volume-id
5. check the lun under /dev/disk/by-path, lun is still linked to another under /dev/mapper which already been removed. so it is marked as read when ls -al
6. create volume again and do attach, it would be failed and report error:
DEBUG os_brick.initiator.connector [^[[00;36m-^[[00;32m] ^[[00;32mLooking for Fibre Channel dev /dev/disk/by-path/pci-0000:03:00.0-fc-0x500507680140a743-lun-0 ^[[00;33mfrom (pid=19105) _wait_for_device_discovery /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py:981
ERROR os_brick.initiator.connector [^[[00;36m-] Fibre Channel volume device not found.
ERROR oslo.service.loopingcall [^[[00;36m-] Fixed interval looping call 'os_brick.initiator.connector._wait_for_device_discovery' failed
TRACE oslo.service.loopingcall Traceback (most recent call last):
TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
TRACE oslo.service.loopingcall result = func(*self.args, **self.kw)
TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 991, in _wait_for_device_discovery
TRACE oslo.service.loopingcall raise exception.NoFibreChannelVolumeDeviceFound()
TRACE oslo.service.loopingcall NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume device.
TRACE oslo.service.loopingcall
DEBUG oslo_concurrency.lockutils [^[[01;36mreq-37f0f319-b53e-4da7-a0f8-cf031a1fe727 ^[[00;36madmin admin^[[00;32m] ^[[00;32mLock "connect_volume" released by "os_brick.initiator.connector.connect_volume" :: held 10.638s ^[[00;33mfrom (pid=19105) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:265
ERROR nova.virt.block_device [^[[01;36mreq-37f0f319-b53e-4da7-a0f8-cf031a1fe727 ^[[00;36madmin admin] Driver failed to attach volume 7a7e1c42-1ca5-473d-a098-e5d307b55e1e at /dev/vdb
TRACE nova.virt.block_device Traceback (most recent call last):
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/block_device.py", line 256, in attach
TRACE nova.virt.block_device device_type=self['device_type'], encryption=encryption)
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1100, in attach_volume
TRACE nova.virt.block_device self._connect_volume(connection_info, disk_info)
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1050, in _connect_volume
TRACE nova.virt.block_device driver.connect_volume(connection_info, disk_info)
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/libvirt/volume/fibrechannel.py", line 54, in connect_volume
TRACE nova.virt.block_device device_info = self.connector.connect_volume(connection_info['data'])
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
TRACE nova.virt.block_device return f(*args, **kwargs)
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 1005, in connect_volume
TRACE nova.virt.block_device timer.start(interval=2).wait()
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
TRACE nova.virt.block_device return hubs.get_hub().switch()
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
TRACE nova.virt.block_device return self.greenlet.switch()
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
TRACE nova.virt.block_device result = func(*self.args, **self.kw)
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 991, in _wait_for_device_discovery
TRACE nova.virt.block_device raise exception.NoFibreChannelVolumeDeviceFound()
TRACE nova.virt.block_device NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume device.
It also can reproduced with iSCSI protocol. Following is the error logs: block_device Traceback (most recent call last): block_device File "/opt/stack/ new/nova/ nova/virt/ block_device. py", line 256, in attach block_device device_ type=self[ 'device_ type'], encryption= encryption) block_device File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 1099, in attach_volume block_device self._connect_ volume( connection_ info, disk_info) block_device File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 1049, in _connect_volume block_device driver. connect_ volume( connection_ info, disk_info) block_device File "/opt/stack/ new/nova/ nova/virt/ libvirt/ volume/ iscsi.py" , line 84, in connect_volume block_device device_info = self.connector. connect_ volume( connection_ info['data' ]) block_device File "/usr/local/ lib/python2. 7/dist- packages/ oslo_concurrenc y/lockutils. py", line 254, in inner block_device return f(*args, **kwargs) block_device File "/usr/local/ lib/python2. 7/dist- packages/ os_brick/ initiator/ connector. py", line 499, in connect_volume block_device raise exception. VolumeDeviceNot Found(device= host_devices) block_device VolumeDeviceNot Found: Volume device not found at [u'/dev/ disk/by- path/ip- 9.11.101. 17:3260- iscsi-iqn. 1986-03. com.ibm: 2145.arctbd3. arc64g00gf- 2-lun-4' ]. block_device manager [req-02370405- 6fa4-414d- a588-39b84cf2ca a0 tempest- TestEncryptedCi nderVolumes- 586258091 tempest- TestEncryptedCi nderVolumes- 19401] Failed to attach b70c823c- 7992-445a- 8d24-5668fe7628 18 at /dev/vdb manager Traceback (most recent call last): manager File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 4536, in _attach_volume manager do_check_ attach= False, do_driver_ attach= True) manager File "/opt/stack/ new/nova/ nova/virt/ block_device. py", line 52, in wrapped manager ret_val = method(obj, context, *args, **kwargs) manager File "/opt/stack/ new/nova/ nova/virt/ block_device. py", line 265, in attach manager connector) manager File "/usr/local/ lib/python2. 7/dist- packages/ oslo_utils/ excutils. py", line 195, in __exit__ manager six.reraise( self.type_ , self.value, self.tb) manager File "/opt/stack/ new/nova/ nova/virt/ block_device. py", line 256, in attach manager device_ type=self[ 'device_ type'], encryption= encryption) manager File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 1099, in attach_volume manager self._connect_ volume( connection_ info, disk_info) manager File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 1049, in _connect_volume manager driver. connect_ volume( connection_ info, d...
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.virt.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.
ERROR nova.compute.