| 2015-05-13 06:52:47 |
Tina Tang |
description |
We found this issue during testing volume detachment when iSCSI multipath is used. When a same iSCSI protal and iqn is shared by multiple LUNs, device from other volume maybe be deleted unexpected. This is found both in Kilo and the latest code.
For example, the devices under /dev/disk/by-path may looks like below when LUN 23 and 231 are from a same storage system and a same iSCSI protal and iqn are used. ls /dev/disk/by-path
ip-192.168.3.50:3260-iscsi-<iqna>-lun-23 -> ../../sdh
ip-192.168.3.50:3260-iscsi-<iqna>-lun-231 -> ../../sdk
ip-192.168.3.51:3260-iscsi-<iqnb>-lun-23 -> ../../sdd
ip-192.168.3.51:3260-iscsi-<iqnb>-lun-231 -> ../../sdi
When we try to detach volume corresponding LUN 23 from the host, the devices regarding to LUN 231 are also deleted which may cause the data unavailable.
Why this happen? After digging into the node code, below is the clue:
nova/virt/libvirt/volume.py
770 def _delete_mpath(self, iscsi_properties, multipath_device, ips_iqns):
771 entries = self._get_iscsi_devices()
772 # Loop through ips_iqns to construct all paths
773 iqn_luns = []
774 for ip, iqn in ips_iqns:
775 iqn_lun = '%s-lun-%s' % (iqn,
776 iscsi_properties.get('target_lun', 0))
777 iqn_luns.append(iqn_lun)
778 for dev in ['/dev/disk/by-path/%s' % dev for dev in entries]:
779 for iqn_lun in iqn_luns:
780 if iqn_lun in dev: ==> This is incorrect, device for LUN 231 will made this be True.
781 self._delete_device(dev)
782
783 self._rescan_multipath()
Due to the incorrect logic in line 780, detach LUN xx will deleted devices for other LUNs starts with xx, such as xxy, xxz
===================================
stack@openstack-performance:~/tina/nova_iscsi_mp/nova$ git log -1
commit f4504f3575b35ec14390b4b678e441fcf953f47b
Merge: 3f21f60 5fbd852
Author: Jenkins <jenkins@review.openstack.org>
Date: Tue May 12 22:46:43 2015 +0000
Merge "Remove db layer hard-code permission checks for network_get_all_by_host" |
We found this issue during testing volume detachment when iSCSI multipath is used. When a same iSCSI protal and iqn is shared by multiple LUNs, device from other volume maybe be deleted unexpected. This is found both in Kilo and the latest code.
For example, the devices under /dev/disk/by-path may looks like below when LUN 23 and 231 are from a same storage system and a same iSCSI protal and iqn are used. ls /dev/disk/by-path
ip-192.168.3.50:3260-iscsi-<iqna>-lun-23
ip-192.168.3.50:3260-iscsi-<iqna>-lun-231
ip-192.168.3.51:3260-iscsi-<iqnb>-lun-23
ip-192.168.3.51:3260-iscsi-<iqnb>-lun-231
When we try to detach volume corresponding LUN 23 from the host, we noticed that the devices regarding to LUN 231 are also deleted which may cause the data unavailable.
Why this happen? After digging into the nova code, below is the clue:
nova/virt/libvirt/volume.py
770 def _delete_mpath(self, iscsi_properties, multipath_device, ips_iqns):
771 entries = self._get_iscsi_devices()
772 # Loop through ips_iqns to construct all paths
773 iqn_luns = []
774 for ip, iqn in ips_iqns:
775 iqn_lun = '%s-lun-%s' % (iqn,
776 iscsi_properties.get('target_lun', 0))
777 iqn_luns.append(iqn_lun)
778 for dev in ['/dev/disk/by-path/%s' % dev for dev in entries]:
779 for iqn_lun in iqn_luns:
780 if iqn_lun in dev: ==> This is incorrect, device for LUN 231 will made this be True.
781 self._delete_device(dev)
782
783 self._rescan_multipath()
Due to the incorrect logic in line 780, detach LUN xx will deleted devices for other LUNs starts with xx, such as xxy, xxz
===================================
stack@openstack-performance:~/tina/nova_iscsi_mp/nova$ git log -1
commit f4504f3575b35ec14390b4b678e441fcf953f47b
Merge: 3f21f60 5fbd852
Author: Jenkins <jenkins@review.openstack.org>
Date: Tue May 12 22:46:43 2015 +0000
Merge "Remove db layer hard-code permission checks for network_get_all_by_host" |
|
| 2015-05-13 06:55:01 |
Tina Tang |
description |
We found this issue during testing volume detachment when iSCSI multipath is used. When a same iSCSI protal and iqn is shared by multiple LUNs, device from other volume maybe be deleted unexpected. This is found both in Kilo and the latest code.
For example, the devices under /dev/disk/by-path may looks like below when LUN 23 and 231 are from a same storage system and a same iSCSI protal and iqn are used. ls /dev/disk/by-path
ip-192.168.3.50:3260-iscsi-<iqna>-lun-23
ip-192.168.3.50:3260-iscsi-<iqna>-lun-231
ip-192.168.3.51:3260-iscsi-<iqnb>-lun-23
ip-192.168.3.51:3260-iscsi-<iqnb>-lun-231
When we try to detach volume corresponding LUN 23 from the host, we noticed that the devices regarding to LUN 231 are also deleted which may cause the data unavailable.
Why this happen? After digging into the nova code, below is the clue:
nova/virt/libvirt/volume.py
770 def _delete_mpath(self, iscsi_properties, multipath_device, ips_iqns):
771 entries = self._get_iscsi_devices()
772 # Loop through ips_iqns to construct all paths
773 iqn_luns = []
774 for ip, iqn in ips_iqns:
775 iqn_lun = '%s-lun-%s' % (iqn,
776 iscsi_properties.get('target_lun', 0))
777 iqn_luns.append(iqn_lun)
778 for dev in ['/dev/disk/by-path/%s' % dev for dev in entries]:
779 for iqn_lun in iqn_luns:
780 if iqn_lun in dev: ==> This is incorrect, device for LUN 231 will made this be True.
781 self._delete_device(dev)
782
783 self._rescan_multipath()
Due to the incorrect logic in line 780, detach LUN xx will deleted devices for other LUNs starts with xx, such as xxy, xxz
===================================
stack@openstack-performance:~/tina/nova_iscsi_mp/nova$ git log -1
commit f4504f3575b35ec14390b4b678e441fcf953f47b
Merge: 3f21f60 5fbd852
Author: Jenkins <jenkins@review.openstack.org>
Date: Tue May 12 22:46:43 2015 +0000
Merge "Remove db layer hard-code permission checks for network_get_all_by_host" |
We found this issue during testing volume detachment when iSCSI multipath is used. When a same iSCSI protal and iqn is shared by multiple LUNs, device from other volume maybe be deleted unexpected. This is found both in Kilo and the latest code.
For example, the devices under /dev/disk/by-path may looks like below when LUN 23 and 231 are from a same storage system and a same iSCSI protal and iqn are used. ls /dev/disk/by-path
ip-192.168.3.50:3260-iscsi-<iqna>-lun-23
ip-192.168.3.50:3260-iscsi-<iqna>-lun-231
ip-192.168.3.51:3260-iscsi-<iqnb>-lun-23
ip-192.168.3.51:3260-iscsi-<iqnb>-lun-231
When we try to detach volume corresponding LUN 23 from the host, we noticed that the devices regarding to LUN 231 are also deleted which may cause the data unavailable.
Why this happen? After digging into the nova code, below is the clue:
nova/virt/libvirt/volume.py
770 def _delete_mpath(self, iscsi_properties, multipath_device, ips_iqns):
771 entries = self._get_iscsi_devices()
772 # Loop through ips_iqns to construct all paths
773 iqn_luns = []
774 for ip, iqn in ips_iqns:
775 iqn_lun = '%s-lun-%s' % (iqn,
776 iscsi_properties.get('target_lun', 0))
777 iqn_luns.append(iqn_lun)
778 for dev in ['/dev/disk/by-path/%s' % dev for dev in entries]:
779 for iqn_lun in iqn_luns:
780 if iqn_lun in dev: ==> This is incorrect, device for LUN 231 will made this be True.
781 self._delete_device(dev)
782
783 self._rescan_multipath()
Due to the incorrect logic in line 780, detach LUN xx will deleted devices for other LUNs starts with xx, such as xxy, xxz. We could use dev.endswith(iqn_lun) to avoid it.
===================================
stack@openstack-performance:~/tina/nova_iscsi_mp/nova$ git log -1
commit f4504f3575b35ec14390b4b678e441fcf953f47b
Merge: 3f21f60 5fbd852
Author: Jenkins <jenkins@review.openstack.org>
Date: Tue May 12 22:46:43 2015 +0000
Merge "Remove db layer hard-code permission checks for network_get_all_by_host" |
|
