terminate instances boot from volume used multipath have residual device

Bug #1402535 reported by YaoZheng_ZTE
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
ugvddm

Bug Description

Reproducing method as following:
1、nova.conf configure iscsi_used_multipath_tool=multipath, restart nova-compute service
2、launch instance vm1 boot from volume(used HpSan),then attach volume1 for this vm1
3、launch instance vm2 boot from volume(used HpSan),then attach volume2 for this vm2 at the same host
4、terminate vm2
5、vm2 has been destoryed , but /dev/disk/by-path/ device can not be completely removed

Revision history for this message
YaoZheng_ZTE (zheng-yao1) wrote :

the reason for this issue :
in /nova/compute/manager.py _shutdown_instance function :
first, query one instances's two volume, then disconnect volume and delete /dev/disk/by-path/ device.
 disconnect the first volume is ok , but the first volume is not removed from san's mapping group, so when disconnect the second volume , in /nova/virt/libvirt/volume.py _disconnect_volume_multipath_iscsi function , can run self._rescan_iscsi() ,because the first volume has not removed san's mapping group yet ,so recsan the first device . so when the vm have destoryed, but the first device is residual .
 Reproducing as following:
1、 launch instance vm1 boot from volume(used HpSan),then attach volume1 for this vm1. /dev/disk/by-path/ as following:
[root@2c514-1-13-SBCJ cinder(keystone_admin)]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 Dec 13 18:41 ip-192.14.1.232:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr-4c:09:b4:b0:02:30-lun-0 -> ../../sdb
lrwxrwxrwx 1 root root 9 Dec 13 18:41 ip-192.14.1.232:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr-4c:09:b4:b0:02:30-lun-1 -> ../../sdc
2、 launch instance vm2 boot from volume(used HpSan),then attach volume2 for this vm2. /dev/disk/by-path/ as following:
[root@2c514-1-13-SBCJ cinder(keystone_admin)]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 Dec 13 18:41 ip-192.14.1.232:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr-4c:09:b4:b0:02:30-lun-0 -> ../../sdb
lrwxrwxrwx 1 root root 9 Dec 13 18:41 ip-192.14.1.232:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr-4c:09:b4:b0:02:30-lun-1 -> ../../sdc
lrwxrwxrwx 1 root root 9 Dec 13 19:49 ip-192.14.1.232:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr-4c:09:b4:b0:02:30-lun-2 -> ../../sdd
lrwxrwxrwx 1 root root 9 Dec 13 19:49 ip-192.14.1.232:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr-4c:09:b4:b0:02:30-lun-3 -> ../../sde
3、terminat instances vm2, /dev/disk/by-path/ as following:

[root@2c514-1-13-SBCJ cinder(keystone_admin)]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 Dec 13 18:41 ip-192.14.1.232:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr-4c:09:b4:b0:02:30-lun-0 -> ../../sdb
lrwxrwxrwx 1 root root 9 Dec 13 18:41 ip-192.14.1.232:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr-4c:09:b4:b0:02:30-lun-1 -> ../../sdc
lrwxrwxrwx 1 root root 9 Dec 13 19:49 ip-192.14.1.232:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr-4c:09:b4:b0:02:30-lun-2 -> ../../sdd

the device ip-192.14.1.232:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr-4c:09:b4:b0:02:30-lun-2 should be deleted ,but now cannt be removed

melanie witt (melwitt)
summary: - terminate instances boot from volume used multipath have esidual
+ terminate instances boot from volume used multipath have residual
device
tags: added: libvirt volumes
Revision history for this message
YaoZheng_ZTE (zheng-yao1) wrote :

Hi melanie witt,
   this issue will be solved by in /nova/virt/libvirt/volume.py function _disconnect_volume_multipath_iscsi , remove call self._rescan_iscsi(). because disconnect volume will not need rescan_iscsi.

Changed in nova:
status: New → Confirmed
importance: Undecided → Low
ugvddm (271025598-9)
Changed in nova:
assignee: nobody → ugvddm (271025598-9)
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
ugvddm (271025598-9) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/184676
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2833f8c08fcfb7961b3c64b285ceff958bf5a05e
Submitter: Jenkins
Branch: master

commit 2833f8c08fcfb7961b3c64b285ceff958bf5a05e
Author: Zhengguang <email address hidden>
Date: Thu May 21 02:31:50 2015 +0000

    remove _rescan_iscsi from disconnect_volume_multipath_iscsi

    terminating instance that attached more than one volume, disconnect
    the first volume is ok, but the first volume is not removed, then
    disconnect the second volume, disconnect_volume_multipath_iscsi
    will call _rescan_iscsi so that rescan the first device, although
    the instance is destroyed, the first device is residual, therefor
    we don't need rescan when disconnect volume.

    Change-Id: I7f2c688aba9e69afaf370b2badc86a2bb3ee899d
    Closes-Bug:#1402535

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-2 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.