| 2015-09-10 02:37:11 |
YuZhang |
description |
Lun linked to an unexist file when detach an encrypted volume, and then it would lead all volume attach failed with error unable to find a Fibre Channel volume device. Seems like the lun should be removed instead of linking to an unexist file
nova version: 2.28.1
cinder version :1.3.1
os-brick version: 0.4.0
Reproduce steps:
1. create a type and mark the volume type as encrypted
cinder type-create LUKS
cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
2. create an encrypted volume and attach to an instance
cinder create --display-name 'encrypted volume' --volume-type LUKS 1
nova volume-attach instance-id volume-id auto
3. Check the lun created:
under /dev/disk/by-path:
pci-0000:03:00.0-fc-0x500507680140a743-lun-0 -> /dev/mapper/pci-0000:03:00.0-fc-0x500507680140a743-lun-0
under /dev/mapper:
lrwxrwxrwx 1 root root 7 Sep 8 21:47 pci-0000:03:00.0-fc-0x500507680140a743-lun-0 -> ../dm-2
4. detach the volume: nova volume-detach instance-id volume-id
5. check the lun under /dev/disk/by-path, lun is still linked to another under /dev/mapper which already been removed. so it is marked as read when ls -al
6. create volume again and do attach, it would be failed and report error:
2015-09-08 05:44:57.108 DEBUG os_brick.initiator.connector [^[[00;36m-^[[00;32m] ^[[00;32mLooking for Fibre Channel dev /dev/disk/by-path/pci-0000:03:00.0-fc-0x500507680140a743-lun-0 ^[[00;33mfrom (pid=19105) _wait_for_device_discovery /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py:981
2015-09-08 05:44:57.109 ERROR os_brick.initiator.connector [^[[00;36m-] Fibre Channel volume device not found.
2015-09-08 05:44:57.112 ERROR oslo.service.loopingcall [^[[00;36m-] Fixed interval looping call 'os_brick.initiator.connector._wait_for_device_discovery' failed
2015-09-08 05:44:57.112 TRACE oslo.service.loopingcall Traceback (most recent call last):
2015-09-08 05:44:57.112 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
2015-09-08 05:44:57.112 TRACE oslo.service.loopingcall result = func(*self.args, **self.kw)
2015-09-08 05:44:57.112 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 991, in _wait_for_device_discovery
2015-09-08 05:44:57.112 TRACE oslo.service.loopingcall raise exception.NoFibreChannelVolumeDeviceFound()
2015-09-08 05:44:57.112 TRACE oslo.service.loopingcall NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume device.
2015-09-08 05:44:57.112 TRACE oslo.service.loopingcall
2015-09-08 05:44:57.117 DEBUG oslo_concurrency.lockutils [^[[01;36mreq-37f0f319-b53e-4da7-a0f8-cf031a1fe727 ^[[00;36madmin admin^[[00;32m] ^[[00;32mLock "connect_volume" released by "os_brick.initiator.connector.connect_volume" :: held 10.638s ^[[00;33mfrom (pid=19105) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:265
2015-09-08 05:44:57.118 ERROR nova.virt.block_device [^[[01;36mreq-37f0f319-b53e-4da7-a0f8-cf031a1fe727 ^[[00;36madmin admin] [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] Driver failed to attach volume 7a7e1c42-1ca5-473d-a098-e5d307b55e1e at /dev/vdb
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] Traceback (most recent call last):
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] File "/opt/stack/nova/nova/virt/block_device.py", line 256, in attach
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] device_type=self['device_type'], encryption=encryption)
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1100, in attach_volume
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] self._connect_volume(connection_info, disk_info)
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1050, in _connect_volume
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] driver.connect_volume(connection_info, disk_info)
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] File "/opt/stack/nova/nova/virt/libvirt/volume/fibrechannel.py", line 54, in connect_volume
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] device_info = self.connector.connect_volume(connection_info['data'])
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] return f(*args, **kwargs)
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 1005, in connect_volume
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] timer.start(interval=2).wait()
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] return hubs.get_hub().switch()
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] return self.greenlet.switch()
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] result = func(*self.args, **self.kw)
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 991, in _wait_for_device_discovery
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] raise exception.NoFibreChannelVolumeDeviceFound()
2015-09-08 05:44:57.118 TRACE nova.virt.block_device [instance: 497c8d66-c2d0-4117-a12e-eaae80c03168] NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume device. |
Lun linked to an unexist file when detach an encrypted volume, and then it would lead all volume attach failed with error unable to find a Fibre Channel volume device. Seems like the lun should be removed instead of linking to an unexist file
nova version: 2.28.1
cinder version :1.3.1
os-brick version: 0.4.0
Reproduce steps:
1. create a type and mark the volume type as encrypted
cinder type-create LUKS
cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
2. create an encrypted volume and attach to an instance
cinder create --display-name 'encrypted volume' --volume-type LUKS 1
nova volume-attach instance-id volume-id auto
3. Check the lun created:
under /dev/disk/by-path:
pci-0000:03:00.0-fc-0x500507680140a743-lun-0 -> /dev/mapper/pci-0000:03:00.0-fc-0x500507680140a743-lun-0
under /dev/mapper:
lrwxrwxrwx 1 root root 7 Sep 8 21:47 pci-0000:03:00.0-fc-0x500507680140a743-lun-0 -> ../dm-2
4. detach the volume: nova volume-detach instance-id volume-id
5. check the lun under /dev/disk/by-path, lun is still linked to another under /dev/mapper which already been removed. so it is marked as read when ls -al
6. create volume again and do attach, it would be failed and report error:
DEBUG os_brick.initiator.connector [^[[00;36m-^[[00;32m] ^[[00;32mLooking for Fibre Channel dev /dev/disk/by-path/pci-0000:03:00.0-fc-0x500507680140a743-lun-0 ^[[00;33mfrom (pid=19105) _wait_for_device_discovery /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py:981
ERROR os_brick.initiator.connector [^[[00;36m-] Fibre Channel volume device not found.
ERROR oslo.service.loopingcall [^[[00;36m-] Fixed interval looping call 'os_brick.initiator.connector._wait_for_device_discovery' failed
TRACE oslo.service.loopingcall Traceback (most recent call last):
TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
TRACE oslo.service.loopingcall result = func(*self.args, **self.kw)
TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 991, in _wait_for_device_discovery
TRACE oslo.service.loopingcall raise exception.NoFibreChannelVolumeDeviceFound()
TRACE oslo.service.loopingcall NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume device.
TRACE oslo.service.loopingcall
DEBUG oslo_concurrency.lockutils [^[[01;36mreq-37f0f319-b53e-4da7-a0f8-cf031a1fe727 ^[[00;36madmin admin^[[00;32m] ^[[00;32mLock "connect_volume" released by "os_brick.initiator.connector.connect_volume" :: held 10.638s ^[[00;33mfrom (pid=19105) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:265
ERROR nova.virt.block_device [^[[01;36mreq-37f0f319-b53e-4da7-a0f8-cf031a1fe727 ^[[00;36madmin admin] Driver failed to attach volume 7a7e1c42-1ca5-473d-a098-e5d307b55e1e at /dev/vdb
TRACE nova.virt.block_device Traceback (most recent call last):
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/block_device.py", line 256, in attach
TRACE nova.virt.block_device device_type=self['device_type'], encryption=encryption)
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1100, in attach_volume
TRACE nova.virt.block_device self._connect_volume(connection_info, disk_info)
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1050, in _connect_volume
TRACE nova.virt.block_device driver.connect_volume(connection_info, disk_info)
TRACE nova.virt.block_device File "/opt/stack/nova/nova/virt/libvirt/volume/fibrechannel.py", line 54, in connect_volume
TRACE nova.virt.block_device device_info = self.connector.connect_volume(connection_info['data'])
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
TRACE nova.virt.block_device return f(*args, **kwargs)
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 1005, in connect_volume
TRACE nova.virt.block_device timer.start(interval=2).wait()
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
TRACE nova.virt.block_device return hubs.get_hub().switch()
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
TRACE nova.virt.block_device return self.greenlet.switch()
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
TRACE nova.virt.block_device result = func(*self.args, **self.kw)
TRACE nova.virt.block_device File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 991, in _wait_for_device_discovery
TRACE nova.virt.block_device raise exception.NoFibreChannelVolumeDeviceFound()
TRACE nova.virt.block_device NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume device. |
|
