Comment 0 for bug 1419577

Revision history for this message
Hyun Ha (raymon-ha) wrote :

Hi, guys

When live-migrate failed with error, lun-id of connection_info column in Nova's block_deivce_mapping table couldn't be rollback.
and failed VM can have others volume.

my test environment is following :

Openstack Version : Havana ( 2013.2.3)
Compute Node OS : 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Compute Node multipath : multipath-tools 0.4.9-3ubuntu7.2

test step is :

1) create 2 Compute node (host#1 and host#2)
2) create 1 VM on host#1 (vm01)
3) create 1 cinder volume (vol01)
4) attach 1 volume to vm01 (/dev/vdb)
5) live-migrate vm01 from host#1 to host#2
6) live-migrate success
     - please check the mapper by using multipath command in host#1 (# multipath -ll), then you can find mapper is not deleted.
       and the status of devices is "failed faulty"
     - please check the lun-id of vol01
7) Again, live-migrate vm01 from host#2 to host#1 (vm01 was migrated to host#2 at step 4)
8) live-migrate fail
     - please check the mapper in host#1
     - please check the lun-id of vol01, then you can find the lun hav "two" igroups
     - please check the connection_info column in Nova's block_deivce_mapping table, then you can find lun-id couldn't be rollback

This Bug is critical security issue because the failed VM can have others volume.

and every backend storage of cinder-volume can have same problem because this is the bug of live-migration's rollback process.

I suggest bellow methods to solve issue :

1) when live-migrate is complete, nova should delete mapper devices at origin host
2) when live-migrate is failed, nova should rollback lun-id in connection_info column
3) when live-migrate is failed, cinder should delete the mapping between lun and host (Netapp : igroup, EMC : storage_group ...)
4) when volume-attach is requested , cinder volume driver of vendors should make lun-id randomly for reduce of probability of mis-mapping

please check this bug.

Thank you.