| 2016-04-23 14:02:46 |
jangseon ryu |
description |
Description
===========
target-lun id of volume changed when live-migration failed
I tried to live-migrate vm with attahed volume, but failed.
I think nova-compute should rollback vm as old status before live-migration.
But target-lun id of volume didn't changed as old lun-id.
Environment
===========
- OpenStack Release : Liberty
- OS : Ubuntu 14.04.2 LTS
- Hypervisor : KVM
- Cinder Storage : iSCSI (EMC VNX)
Steps to reproduce
==================
1. Create VM and Attach Volume to VM(target-lun id is 174)
2. Try to Live-migration... but failed (target-lun id changed 97)
3. Try to rollback (target-lun id still is 97)
* target-lun id didn't changed old one.
> before live-migration(nova.block_device_mapping table.connection_info)
{"driver_volume_type": "iscsi", "serial": "6352f542-819f-477e-a588-b15d75008178",
"data": {"target_luns": [174, 174, 174, 174],
"device_path": "/dev/mapper/360060160a7d03800b0dcf791f008e611", "target_discovered": true, "encrypted": false, "qos_specs": null,
"target_iqn": "iqn.1992-04.com.emc:cx.ckm00142100690.b0", "target_portal": "x.x.x.x:3260", "volume_id": "6352f542-819f-477e-a588-b15d75008178",
"target_lun": 174, "access_mode": "rw",
"target_portals": ["x.x.x.x:3260", "x.x.x.x:3260", "x.x.x.x:3260", "x.x.x.x:3260"]}}
(*) target_lun id was 174.
> after live-migration and rollback (nova.block_device_mapping table.connection_info)
{"driver_volume_type": "iscsi", "serial": "6352f542-819f-477e-a588-b15d75008178",
"data": {"target_luns": [97, 97, 97, 97],
"target_discovered": true, "encrypted": false, "qos_specs": null, "target_iqn": "iqn.1992-04.com.emc:cx.ckm00142100690.b1",
"target_portal": "x.x.x.x:3260", "volume_id": "6352f542-819f-477e-a588-b15d75008178",
"target_lun": 97, "access_mode": "rw",
"target_portals": ["x.x.x.x:3260", "x.x.x.x:3260", "x.x.x.x:3260", "x.x.x.x:3260"]}}
(*) target_lun id was changed 97.
Expected result
===============
If live-migration success, it is normal status that target-lun id changes 97.
Actual result
=============
otherwise target-lun id must change as old target id(174).
In this environment, if nova reboot the vm, vm will fail rebooting.
Because target-lun(97) doesn't exist in the server.
Or
If target-lun(97) used by other vm exist, vm will attach volume used by other vm.
It occurs critical situation that single volume attached by multi-vm. |
Description
===========
target-lun id of volume changed when live-migration failed
I tried to live-migrate vm with attahed volume, but failed.
I think nova-compute should rollback vm as old status before live-migration.
But target-lun id of volume didn't changed as old lun-id.
Environment
===========
- OpenStack Release : Liberty
- OS : Ubuntu 14.04.2 LTS
- Hypervisor : KVM
- Cinder Storage : iSCSI (EMC VNX)
Steps to reproduce
==================
1. Create VM and Attach Volume to VM(target-lun id is 174)
2. Try to Live-migration... but failed (target-lun id changed 97)
3. Try to rollback (target-lun id still is 97)
* target-lun id didn't changed old one.
> before live-migration(nova.block_device_mapping table.connection_info)
{"driver_volume_type": "iscsi", "serial": "6352f542-819f-477e-a588-b15d75008178",
"data": {"target_luns": [174, 174, 174, 174],
"device_path": "/dev/mapper/360060160a7d03800b0dcf791f008e611", "target_discovered": true, "encrypted": false, "qos_specs": null,
"target_iqn": "iqn.1992-04.com.emc:cx.ckm00142100690.b0", "target_portal": "x.x.x.x:3260", "volume_id": "6352f542-819f-477e-a588-b15d75008178",
"target_lun": 174, "access_mode": "rw",
"target_portals": ["x.x.x.x:3260", "x.x.x.x:3260", "x.x.x.x:3260", "x.x.x.x:3260"]}}
(*) target_lun id was 174.
> after live-migration and rollback (nova.block_device_mapping table.connection_info)
{"driver_volume_type": "iscsi", "serial": "6352f542-819f-477e-a588-b15d75008178",
"data": {"target_luns": [97, 97, 97, 97],
"target_discovered": true, "encrypted": false, "qos_specs": null, "target_iqn": "iqn.1992-04.com.emc:cx.ckm00142100690.b1",
"target_portal": "x.x.x.x:3260", "volume_id": "6352f542-819f-477e-a588-b15d75008178",
"target_lun": 97, "access_mode": "rw",
"target_portals": ["x.x.x.x:3260", "x.x.x.x:3260", "x.x.x.x:3260", "x.x.x.x:3260"]}}
(*) target_lun id was changed 97.
Expected result
===============
If live-migration success, it is normal status that target-lun id changes 97.
Actual result
=============
otherwise target-lun id must change as old target id(174).
In this environment, if nova reboot the vm, vm will fail rebooting.
Because target-lun(97) doesn't exist in the server.
Or
If target-lun(97) used by other vm exist, vm will attach volume used by other vm.
It occurs critical situation that single volume attached by multi-vm.
Logs & Configs
=============
we don't have volume(target-lun:97) in the server,
nova-compute couldn't reboot vm.
$ nova list
+--------------------------------------+--------------------+--------+-----------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------+--------+-----------------+-------------+--------------------------+
| 44a127c3-e7b9-4538-8e5c-068fa8f0262f | vm2 | ERROR | - | Running | vmware_net01=66.66.66.54 |
2016-04-23 22:51:48.704 28334 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 816, in connect_volume
2016-04-23 22:51:48.704 28334 ERROR oslo_messaging.rpc.dispatcher raise exception.VolumeDeviceNotFound(device=host_devices)
2016-04-23 22:51:48.704 28334 ERROR oslo_messaging.rpc.dispatcher VolumeDeviceNotFound: Volume device not found at [u'/dev/disk/by-path/ip-x.x.x.x:3260-iscsi-iqn.1992-04.com.emc:cx.ckm00142100690.b1-lun-97', u'/dev/disk/by-path/ip-x.x.x.x:3260-iscsi-iqn.1992-04.com.emc:cx.ckm00142100690.b0-lun-97', u'/dev/disk/by-path/ip-x.x.x.x:3260-iscsi-iqn.1992-04.com.emc:cx.ckm00142100690.a1-lun-97', u'/dev/disk/by-path/ip-x.x.x.x:3260-iscsi-iqn.1992-04.com.emc:cx.ckm00142100690.a0-lun-97']. |
|
