I created an encrypted volume and want to attach it to a vm.
The attaching succeeded but after I detached the volume from the vm, then the subsequent attaching failed with following nova error:
Mar 14 02:25:52 stein nova-compute[1046]: DEBUG os_brick.initiator.connectors.iscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] <== connect_volume: exception (44300ms) Volume
DeviceNotFound(u'Volume device not found at /dev/disk/by-id.',) {{(pid=1046) trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:156}}
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] Driver fai
led to attach volume 61b9c42e-08a6-4405-be7d-19dabbc45c6f at /dev/vdc: VolumeDeviceNotFound: Volume device not found at /dev/disk/by-id.
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] Traceback (most recent call last):
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/opt/stack/nova/nova/virt/block_device.py", line 559, in _volu
me_attach
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] device_type=self['device_type'], encryption=encryption)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1460, in at
tach_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] encryption=encryption)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1258, in _c
onnect_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] vol_driver.connect_volume(connection_info, instance)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 64, i
n connect_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] device_info = self.connector.connect_volume(connection_info['data'
])
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", lin
e 150, in trace_logging_wrapper
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] result = f(*args, **kwargs)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockut
ils.py", line 328, in inner
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] return f(*args, **kwargs)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/conn
ectors/iscsi.py", line 517, in connect_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] self._cleanup_connection(connection_properties, force=True)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py"
, line 220, in __exit__
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] self.force_reraise()
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py"
, line 196, in force_reraise
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] six.reraise(self.type_, self.value, self.tb)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 510, in connect_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] return self._connect_multipath_volume(connection_properties)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 61, in _wrapper
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] return r.call(f, *args, **kwargs)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] raise attempt.get()
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] six.reraise(self.value[0], self.value[1], self.value[2])
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 770, in _connect_multipath_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] mpath)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 551, in _get_connect_result
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] device = self._get_device_link(wwn, device, mpath)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 537, in _get_device_link
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] raise exception.VolumeDeviceNotFound(device='/dev/disk/by-id')
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] VolumeDeviceNotFound: Volume device not found at /dev/disk/by-id.
Enviroment
==========
- OpenStack: Stein devstack
- Ubuntu: 16.04.3
- os_brick: 2.7.0
- Linux kernel: 4.4.0-142-generic
- Cinder backend: DellEMC Unity
Reproduce steps
===============
1. Create an encrypted volume.
2. Create a vm.
3. Attach the volume to the vm.
4. Detach the volume from the vm.
5. Attach the volume to the vm again. >>>>> This step failed.
Basic analysis
==============
The Cinder finished its attachment job on storage backend successfully. In nova log, I saw the logs:
Mar 14 02:25:13 stein nova-compute[1046]: DEBUG os_brick.initiator.connectors.iscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] Connected to sde using {u'target_luns': [134, 134], u'target_iqns': [u'iqn.1992-04.com.emc:cx.fnm00150600267.a0', u'iqn.1992-04.com.emc:cx.fnm00150600267.b0'], u'target_discovered': True, u'encrypted': True, u'qos_specs': None, u'target_iqn': u'iqn.1992-04.com.emc:cx.fnm00150600267.a0', u'target_portals': [u'10.245.47.95:3260', u'10.245.47.96:3260'], u'volume_id': u'61b9c42e-08a6-4405-be7d-19dabbc45c6f', u'target_lun': 134, u'access_mode': u'rw', u'target_portal': u'10.245.47.95:3260'} {{(pid=1046) _connect_vol /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py:663}}
Mar 14 02:25:13 stein nova-compute[1046]: DEBUG os_brick.initiator.linuxscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] Searching for a device in session 2 and hctl (u'3', u'0', u'0', 134) yield: sdf {{(pid=1046) device_name_by_hctl /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:642}}
Mar 14 02:25:13 stein nova-compute[1046]: DEBUG os_brick.initiator.connectors.iscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] Connected to sdf using {u'target_luns': [134, 134], u'target_iqns': [u'iqn.1992-04.com.emc:cx.fnm00150600267.a0', u'iqn.1992-04.com.emc:cx.fnm00150600267.b0'], u'target_discovered': True, u'encrypted': True, u'qos_specs': None, u'target_iqn': u'iqn.1992-04.com.emc:cx.fnm00150600267.b0', u'target_portals': [u'10.245.47.95:3260', u'10.245.47.96:3260'], u'volume_id': u'61b9c42e-08a6-4405-be7d-19dabbc45c6f', u'target_lun': 134, u'access_mode': u'rw', u'target_portal': u'10.245.47.96:3260'} {{(pid=1046) _connect_vol /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py:663}}
Mar 14 02:25:14 stein nova-compute[1046]: DEBUG os_brick.initiator.connectors.iscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] All connection threads finished, giving 10 seconds for dm to appear. {{(pid=1046) _connect_multipath_volume /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py:748}}
Mar 14 02:25:24 stein nova-compute[1046]: WARNING os_brick.initiator.connectors.iscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] No dm was created, connection to volume is probably bad and will perform poorly.
`sde` and `sdf` devices were connected.
In os_brick/initiator/connectors/iscsi.py, _connect_multipath_volume uses below logic to get device wwn.
if not wwn and found:
wwn = self._linuxscsi.get_sysfs_wwn(found)
while `found` was device name `sde` and `sdf`, then `sde` and `sdf` were used to match the wwn in `/dev/disk/by-id` (logic of get_sysfs_wwn). However there was no device under /dev/disk/by-id with symbolic link to `/dev/sde` or `/dev/sdf`. I can only saw the multipath devices there.
$ ll /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root 10 Mar 14 03:11 dm-name-36006016015e03a0099ac885c7bda09ed -> ../../dm-0
lrwxrwxrwx 1 root root 10 Mar 14 03:11 dm-name-36006016015e03a00bfc0895c86abc36e -> ../../dm-1
lrwxrwxrwx 1 root root 10 Mar 14 03:11 dm-uuid-mpath-36006016015e03a0099ac885c7bda09ed -> ../../dm-0
lrwxrwxrwx 1 root root 10 Mar 14 03:11 dm-uuid-mpath-36006016015e03a00bfc0895c86abc36e -> ../../dm-1
lrwxrwxrwx 1 root root 9 Mar 13 06:50 scsi-0DGC_LUNZ_0000000000000000ffff000000000000 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 14 03:11 scsi-36006016015e03a0099ac885c7bda09ed -> ../../dm-0
lrwxrwxrwx 1 root root 10 Mar 14 05:44 scsi-36006016015e03a00bfc0895c86abc36e -> ../../dm-1
lrwxrwxrwx 1 root root 9 Mar 13 06:50 scsi-SDGC_LUNZ_FNM00150600267 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 14 03:11 wwn-0x6006016015e03a0099ac885c7bda09ed -> ../../dm-0
lrwxrwxrwx 1 root root 10 Mar 14 05:44 wwn-0x6006016015e03a00bfc0895c86abc36e -> ../../dm-1
$ sudo multipath -ll
36006016015e03a00bfc0895c86abc36e dm-1 DGC,VRAID
size=1.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 2:0:0:95 sdf 8:80 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
`- 3:0:0:95 sde 8:64 active ready running
36006016015e03a0099ac885c7bda09ed dm-0 DGC,VRAID
size=1.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 3:0:0:190 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
`- 2:0:0:190 sdc 8:32 active ready running
So the `wwn` and `mpath` in function _connect_multipath_volume were '' and `None` even after several times retry, which caused exception raised finally from below logic of function `_get_connect_result`
if con_props.get('encrypted'):
device = self._get_device_link(wwn, device, mpath)
I don't know if this is the same thing, but we've seen CI failures with a luks encrypted volume with a similar error:
https:/ /storage. gra1.cloud. ovh.net/ v1/AUTH_ dcaab5e32b234d5 6b626f72581e364 4c/zuul_ opendev_ logs_998/ 679640/ 32/gate/ nova-multi- cell/998bcd6/ testr_results. html.gz
https:/ /zuul.opendev. org/t/openstack /build/ 998bcd66b810467 fae1664aaf280d5 3d/log/ controller/ logs/screen- n-cpu.txt. gz?severity= 4#25797
Sep 19 08:33:40.910370 ubuntu- bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [None req-02bd4ed8- e521-4317- b434-b768168d28 ec tempest- TestVolumeBootP attern- 1345811753 tempest- TestVolumeBootP attern- 1345811753] [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] Instance failed to spawn: os_brick. exception. VolumeDeviceNot Found: Volume device not found at /dev/disk/by-id. bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] Traceback (most recent call last): bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] File "/opt/stack/ nova/nova/ compute/ manager. py", line 2478, in _build_resources bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] yield resources bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] File "/opt/stack/ nova/nova/ compute/ manager. py", line 2254, in _build_ and_run_ instance bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] block_device_ info=block_ device_ info) bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] File "/opt/stack/ nova/nova/ virt/libvirt/ driver. py", line 3467, in spawn bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] mdevs=mdevs) bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] File "/opt/stack/ nova/nova/ virt/libvirt/ driver. py", line 6027, in _get_guest_xml bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] context, mdevs) bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] File "/opt/stack/ nova/nova/ virt/libvirt/ driver. py", line 5690, in _get_guest_config bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892- 6c49-44b7- bbeb-cf7c846cf8 bf] flavor, guest.os_type) bionic- ovh-bhs1- 0011487265 nova-compute[ 27425]: ERROR nova.compute. manager [instance: e7adf892-6...
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-
Sep 19 08:33:40.910370 ubuntu-