Failed to attach encrypted volumes after detach: volume device not found at /dev/disk/by-id

[Ryan Liang]

I created an encrypted volume and want to attach it to a vm.
The attaching succeeded but after I detached the volume from the vm, then the subsequent attaching failed with following nova error:

Mar 14 02:25:52 stein nova-compute[1046]: DEBUG os_brick.initiator.connectors.iscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] <== connect_volume: exception (44300ms) Volume
DeviceNotFound(u'Volume device not found at /dev/disk/by-id.',) {{(pid=1046) trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:156}}
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] Driver fai
led to attach volume 61b9c42e-08a6-4405-be7d-19dabbc45c6f at /dev/vdc: VolumeDeviceNotFound: Volume device not found at /dev/disk/by-id.
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] Traceback (most recent call last):
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/opt/stack/nova/nova/virt/block_device.py", line 559, in _volu
me_attach
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] device_type=self['device_type'], encryption=encryption)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1460, in at
tach_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] encryption=encryption)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1258, in _c
onnect_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] vol_driver.connect_volume(connection_info, instance)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 64, i
n connect_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] device_info = self.connector.connect_volume(connection_info['data'
])
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", lin
e 150, in trace_logging_wrapper
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] result = f(*args, **kwargs)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockut
ils.py", line 328, in inner
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] return f(*args, **kwargs)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/conn
ectors/iscsi.py", line 517, in connect_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] self._cleanup_connection(connection_properties, force=True)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py"
, line 220, in __exit__
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] self.force_reraise()
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py"
, line 196, in force_reraise
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] six.reraise(self.type_, self.value, self.tb)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 510, in connect_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] return self._connect_multipath_volume(connection_properties)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/utils.py", line 61, in _wrapper
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] return r.call(f, *args, **kwargs)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] raise attempt.get()
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] six.reraise(self.value[0], self.value[1], self.value[2])
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 770, in _connect_multipath_volume
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] mpath)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 551, in _get_connect_result
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] device = self._get_device_link(wwn, device, mpath)
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 537, in _get_device_link
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] raise exception.VolumeDeviceNotFound(device='/dev/disk/by-id')
Mar 14 02:25:52 stein nova-compute[1046]: ERROR nova.virt.block_device [instance: f5158a9b-9584-4f1b-9da9-0d27a03628eb] VolumeDeviceNotFound: Volume device not found at /dev/disk/by-id.

Enviroment
==========
- OpenStack: Stein devstack
- Ubuntu: 16.04.3
- os_brick: 2.7.0
- Linux kernel: 4.4.0-142-generic
- Cinder backend: DellEMC Unity

Reproduce steps
===============
1. Create an encrypted volume.
2. Create a vm.
3. Attach the volume to the vm.
4. Detach the volume from the vm.
5. Attach the volume to the vm again. >>>>> This step failed.

Basic analysis
==============

The Cinder finished its attachment job on storage backend successfully. In nova log, I saw the logs:

Mar 14 02:25:13 stein nova-compute[1046]: DEBUG os_brick.initiator.connectors.iscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] Connected to sde using {u'target_luns': [134, 134], u'target_iqns': [u'iqn.1992-04.com.emc:cx.fnm00150600267.a0', u'iqn.1992-04.com.emc:cx.fnm00150600267.b0'], u'target_discovered': True, u'encrypted': True, u'qos_specs': None, u'target_iqn': u'iqn.1992-04.com.emc:cx.fnm00150600267.a0', u'target_portals': [u'10.245.47.95:3260', u'10.245.47.96:3260'], u'volume_id': u'61b9c42e-08a6-4405-be7d-19dabbc45c6f', u'target_lun': 134, u'access_mode': u'rw', u'target_portal': u'10.245.47.95:3260'} {{(pid=1046) _connect_vol /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py:663}}
Mar 14 02:25:13 stein nova-compute[1046]: DEBUG os_brick.initiator.linuxscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] Searching for a device in session 2 and hctl (u'3', u'0', u'0', 134) yield: sdf {{(pid=1046) device_name_by_hctl /usr/local/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:642}}
Mar 14 02:25:13 stein nova-compute[1046]: DEBUG os_brick.initiator.connectors.iscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] Connected to sdf using {u'target_luns': [134, 134], u'target_iqns': [u'iqn.1992-04.com.emc:cx.fnm00150600267.a0', u'iqn.1992-04.com.emc:cx.fnm00150600267.b0'], u'target_discovered': True, u'encrypted': True, u'qos_specs': None, u'target_iqn': u'iqn.1992-04.com.emc:cx.fnm00150600267.b0', u'target_portals': [u'10.245.47.95:3260', u'10.245.47.96:3260'], u'volume_id': u'61b9c42e-08a6-4405-be7d-19dabbc45c6f', u'target_lun': 134, u'access_mode': u'rw', u'target_portal': u'10.245.47.96:3260'} {{(pid=1046) _connect_vol /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py:663}}
Mar 14 02:25:14 stein nova-compute[1046]: DEBUG os_brick.initiator.connectors.iscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] All connection threads finished, giving 10 seconds for dm to appear. {{(pid=1046) _connect_multipath_volume /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py:748}}
Mar 14 02:25:24 stein nova-compute[1046]: WARNING os_brick.initiator.connectors.iscsi [None req-8e078834-d61b-4356-b511-be8a52df211c admin admin] No dm was created, connection to volume is probably bad and will perform poorly.

`sde` and `sdf` devices were connected.

In os_brick/initiator/connectors/iscsi.py, _connect_multipath_volume uses below logic to get device wwn.
    if not wwn and found:
        wwn = self._linuxscsi.get_sysfs_wwn(found)
while `found` was device name `sde` and `sdf`, then `sde` and `sdf` were used to match the wwn in `/dev/disk/by-id` (logic of get_sysfs_wwn). However there was no device under /dev/disk/by-id with symbolic link to `/dev/sde` or `/dev/sdf`. I can only saw the multipath devices there.

$ ll /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root 10 Mar 14 03:11 dm-name-36006016015e03a0099ac885c7bda09ed -> ../../dm-0
lrwxrwxrwx 1 root root 10 Mar 14 03:11 dm-name-36006016015e03a00bfc0895c86abc36e -> ../../dm-1
lrwxrwxrwx 1 root root 10 Mar 14 03:11 dm-uuid-mpath-36006016015e03a0099ac885c7bda09ed -> ../../dm-0
lrwxrwxrwx 1 root root 10 Mar 14 03:11 dm-uuid-mpath-36006016015e03a00bfc0895c86abc36e -> ../../dm-1
lrwxrwxrwx 1 root root 9 Mar 13 06:50 scsi-0DGC_LUNZ_0000000000000000ffff000000000000 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 14 03:11 scsi-36006016015e03a0099ac885c7bda09ed -> ../../dm-0
lrwxrwxrwx 1 root root 10 Mar 14 05:44 scsi-36006016015e03a00bfc0895c86abc36e -> ../../dm-1
lrwxrwxrwx 1 root root 9 Mar 13 06:50 scsi-SDGC_LUNZ_FNM00150600267 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 14 03:11 wwn-0x6006016015e03a0099ac885c7bda09ed -> ../../dm-0
lrwxrwxrwx 1 root root 10 Mar 14 05:44 wwn-0x6006016015e03a00bfc0895c86abc36e -> ../../dm-1

$ sudo multipath -ll
36006016015e03a00bfc0895c86abc36e dm-1 DGC,VRAID
size=1.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 2:0:0:95 sdf 8:80 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 3:0:0:95 sde 8:64 active ready running
36006016015e03a0099ac885c7bda09ed dm-0 DGC,VRAID
size=1.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 3:0:0:190 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 2:0:0:190 sdc 8:32 active ready running

So the `wwn` and `mpath` in function _connect_multipath_volume were '' and `None` even after several times retry, which caused exception raised finally from below logic of function `_get_connect_result`

    if con_props.get('encrypted'):
        device = self._get_device_link(wwn, device, mpath)

[Matt Riedemann (mriedem)]

I don't know if this is the same thing, but we've seen CI failures with a luks encrypted volume with a similar error:

https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_998/679640/32/gate/nova-multi-cell/998bcd6/testr_results.html.gz

https://zuul.opendev.org/t/openstack/build/998bcd66b810467fae1664aaf280d53d/log/controller/logs/screen-n-cpu.txt.gz?severity=4#25797

Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [None req-02bd4ed8-e521-4317-b434-b768168d28ec tempest-TestVolumeBootPattern-1345811753 tempest-TestVolumeBootPattern-1345811753] [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] Instance failed to spawn: os_brick.exception.VolumeDeviceNotFound: Volume device not found at /dev/disk/by-id.
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] Traceback (most recent call last):
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] File "/opt/stack/nova/nova/compute/manager.py", line 2478, in _build_resources
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] yield resources
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] File "/opt/stack/nova/nova/compute/manager.py", line 2254, in _build_and_run_instance
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] block_device_info=block_device_info)
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3467, in spawn
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] mdevs=mdevs)
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6027, in _get_guest_xml
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] context, mdevs)
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5690, in _get_guest_config
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6c49-44b7-bbeb-cf7c846cf8bf] flavor, guest.os_type)
Sep 19 08:33:40.910370 ubuntu-bionic-ovh-bhs1-0011487265 nova-compute[27425]: ERROR nova.compute.manager [instance: e7adf892-6...

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22os_brick.exception.VolumeDeviceNotFound%3A%20Volume%20device%20not%20found%20at%20%2Fdev%2Fdisk%2Fby-id.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

[Matt Riedemann (mriedem)]

This is what Lee Yarwood said about the problem when investigating the failed CI job run:

(11:52:13 AM) lyarwood: mriedem_afk: actually os-brick is doing some random lookups when encrypted volumes are used
(12:03:53 PM) lyarwood: mriedem_afk: so it looks like /dev/disk/by-id/ is just slow to populate on these CI nodes and we aren't retrying the lookup https://github.com/openstack/os-brick/blob/1c02d525bb77487d5de89c4bd58116b916d44207/os_brick/initiator/connectors/iscsi.py#L521-L540
(12:04:33 PM) lyarwood: mriedem_afk: the fact we are using /dev/disk/by-id/ is due to the volume being encrypted https://github.com/openstack/os-brick/blob/1c02d525bb77487d5de89c4bd58116b916d44207/os_brick/initiator/connectors/iscsi.py#L545-L553
(12:05:40 PM) lyarwood: mriedem_afk: and is due to the horrid design of the original encryptors that symlink things in a horrific way such that we can't use the actual block device path on the host otherwise we'd nuke it https://bugs.launchpad.net/os-brick/+bug/1703954
(12:05:41 PM) openstack: Launchpad bug 1703954 in OpenStack Compute (nova) "Attach/Detach encrypted volume problems with real paths" [Undecided,Incomplete]
(12:07:26 PM) lyarwood: mriedem_afk: I'll post something in the morning to retry _get_device_link but in the longer term I think I need to sort out the symlinking in the encryptors such that it's safe to provide them with a real block device path and they don't do anything stupid.

[Lee Yarwood (lyarwood)]

https://review.opendev.org/#/c/695564/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695564
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=331316827a60aa50ee26be08a06027525f5194ef
Submitter: Zuul
Branch: master

commit 331316827a60aa50ee26be08a06027525f5194ef
Author: Lee Yarwood <email address hidden>
Date: Thu Nov 21 20:01:50 2019 +0000

    iscsi: Add _get_device_link retry when waiting for /dev/disk/by-id/ to populate

    Bug #1820007 documents failures to find /dev/disk/by-id/ symlinks
    associated with encrypted volumes both in real world and CI
    environments. These failures appear to be due to udev on these slow or
    overloaded hosts failing to populate the required /dev/disk/by-id/
    symlinks in time after the iSCSI volume has been connected.

    This change seeks to avoid such failures by simply decorating
    _get_device_link with the @utils.retry to hopefully allow udev time to
    create the required symlinks under /dev/disk/by-id/.

    Closes-Bug: #1820007
    Change-Id: Ib9c8ebae7a6051e18538920139fecd123682a474

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/697115

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/697116

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/697117

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/697118

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697115
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=d8ed21e7d25f67c6d2bfe71515723f68ce15b887
Submitter: Zuul
Branch: stable/train

commit d8ed21e7d25f67c6d2bfe71515723f68ce15b887
Author: Lee Yarwood <email address hidden>
Date: Thu Nov 21 20:01:50 2019 +0000

    iscsi: Add _get_device_link retry when waiting for /dev/disk/by-id/ to populate

    Bug #1820007 documents failures to find /dev/disk/by-id/ symlinks
    associated with encrypted volumes both in real world and CI
    environments. These failures appear to be due to udev on these slow or
    overloaded hosts failing to populate the required /dev/disk/by-id/
    symlinks in time after the iSCSI volume has been connected.

    This change seeks to avoid such failures by simply decorating
    _get_device_link with the @utils.retry to hopefully allow udev time to
    create the required symlinks under /dev/disk/by-id/.

    Closes-Bug: #1820007
    Change-Id: Ib9c8ebae7a6051e18538920139fecd123682a474
    (cherry picked from commit 331316827a60aa50ee26be08a06027525f5194ef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697116
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=d1f84dfd897d35053d39c22ae484f6586d1268bd
Submitter: Zuul
Branch: stable/stein

commit d1f84dfd897d35053d39c22ae484f6586d1268bd
Author: Lee Yarwood <email address hidden>
Date: Thu Nov 21 20:01:50 2019 +0000

    iscsi: Add _get_device_link retry when waiting for /dev/disk/by-id/ to populate

    Bug #1820007 documents failures to find /dev/disk/by-id/ symlinks
    associated with encrypted volumes both in real world and CI
    environments. These failures appear to be due to udev on these slow or
    overloaded hosts failing to populate the required /dev/disk/by-id/
    symlinks in time after the iSCSI volume has been connected.

    This change seeks to avoid such failures by simply decorating
    _get_device_link with the @utils.retry to hopefully allow udev time to
    create the required symlinks under /dev/disk/by-id/.

    Closes-Bug: #1820007
    Change-Id: Ib9c8ebae7a6051e18538920139fecd123682a474
    (cherry picked from commit 331316827a60aa50ee26be08a06027525f5194ef)
    (cherry picked from commit d8ed21e7d25f67c6d2bfe71515723f68ce15b887)

[Matt Riedemann (mriedem)]

Still seeing some failures on master:

https://zuul.opendev.org/t/openstack/build/f78d4687c93e4e9a83df87b884ff7c50/log/compute1/logs/screen-n-cpu.txt.gz?severity=4

But maybe that's bug 1811729?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 2.11.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697117
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=1f811c412a172d5456d37cc748bc217fc4718cbf
Submitter: Zuul
Branch: stable/rocky

commit 1f811c412a172d5456d37cc748bc217fc4718cbf
Author: Lee Yarwood <email address hidden>
Date: Thu Nov 21 20:01:50 2019 +0000

    iscsi: Add _get_device_link retry when waiting for /dev/disk/by-id/ to populate

    Bug #1820007 documents failures to find /dev/disk/by-id/ symlinks
    associated with encrypted volumes both in real world and CI
    environments. These failures appear to be due to udev on these slow or
    overloaded hosts failing to populate the required /dev/disk/by-id/
    symlinks in time after the iSCSI volume has been connected.

    This change seeks to avoid such failures by simply decorating
    _get_device_link with the @utils.retry to hopefully allow udev time to
    create the required symlinks under /dev/disk/by-id/.

    Closes-Bug: #1820007
    Change-Id: Ib9c8ebae7a6051e18538920139fecd123682a474
    (cherry picked from commit 331316827a60aa50ee26be08a06027525f5194ef)
    (cherry picked from commit d8ed21e7d25f67c6d2bfe71515723f68ce15b887)
    (cherry picked from commit d1f84dfd897d35053d39c22ae484f6586d1268bd)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 2.10.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 2.8.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 2.5.9 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697118
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=8972aa4add00e8c27642e5a39ee5dd3aedb39d9f
Submitter: Zuul
Branch: stable/queens

commit 8972aa4add00e8c27642e5a39ee5dd3aedb39d9f
Author: Lee Yarwood <email address hidden>
Date: Thu Nov 21 20:01:50 2019 +0000

    iscsi: Add _get_device_link retry when waiting for /dev/disk/by-id/ to populate

    Bug #1820007 documents failures to find /dev/disk/by-id/ symlinks
    associated with encrypted volumes both in real world and CI
    environments. These failures appear to be due to udev on these slow or
    overloaded hosts failing to populate the required /dev/disk/by-id/
    symlinks in time after the iSCSI volume has been connected.

    This change seeks to avoid such failures by simply decorating
    _get_device_link with the @utils.retry to hopefully allow udev time to
    create the required symlinks under /dev/disk/by-id/.

    Closes-Bug: #1820007
    Change-Id: Ib9c8ebae7a6051e18538920139fecd123682a474
    (cherry picked from commit 331316827a60aa50ee26be08a06027525f5194ef)
    (cherry picked from commit d8ed21e7d25f67c6d2bfe71515723f68ce15b887)
    (cherry picked from commit 7ffa3fb7d9d497fd3848ab0313ba8aa84136a06e)
    (cherry picked from commit ed9feedfa3f9a434a0bbd48953262fbb2999465b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick queens-eol release.