HPE 3PAR: migrating the vm attached with volume has been retyped will cause old volume attach back again

[Guangyu Suo]

# Description

Cinder is configured with two backends, both are HPE 3PAR, but with different cpg, one is ssd, another is hdd, then create two corresponding volume types. VMs are all booted from volume, for the capacity reason, we need retype some bootable volumes attached to active vms from one type to another, after retyping, it seems to be working well, the volume type has changed without downtime, except that the temporary volume created during retyping won't be deleted. After a happy time, we do migration to these vms, after that, we found some data on the volume has been lost, but in the end, we found the "lost data" actually on the temporary volume created during retyping.

# Environment

OpenStack is rocky, but after some code investigation, I think this problem is still on the master.

cinder.conf
[DEFAULT]
enabled_backends = 3parfc-1,3parssd-1

[3parfc-1]
backend_host = control
volume_driver = cinder.volume.drivers.hpe.hpe_3par_fc.HPE3PARFCDriver
volume_backend_name = 3parfc-1
hpe3par_api_url = https://192.168.0.10/api/v1
hpe3par_username = <username>
hpe3par_password = <password>
hpe3par_cpg = FC_r6
san_ip = 192.168.0.10
san_login = <username>
san_password = <password>
max_over_subscription_ratio = 10.0
reserved_percentage = 15
image_volume_cache_enabled = True
image_upload_use_cinder_backend = True
image_upload_use_internal_tenant = True

[3parssd-1]
backend_host = control
volume_driver = cinder.volume.drivers.hpe.hpe_3par_fc.HPE3PARFCDriver
volume_backend_name = 3parssd-1
hpe3par_api_url = https://192.168.0.10/api/v1
hpe3par_username = <username>
hpe3par_password = <password>
hpe3par_cpg = SSD_r6
san_ip = 192.168.0.10
san_login = <username>
san_password = <password>
max_over_subscription_ratio = 10.0
reserved_percentage = 15
image_volume_cache_enabled = True
image_upload_use_cinder_backend = True
image_upload_use_internal_tenant = True

# How to Reproduce

## before retype:

[root@control01 ~]# cinder list | grep nice
| 4e59408d-a517-46c9-8589-91f71ec3182b | in-use | nice | 500 | 3parssd | false | 07715bd4-6b78-43bd-925f-9e3638daf0a7 |

[root@control01 ~]# cinder show 4e59408d-a517-46c9-8589-91f71ec3182b
+--------------------------------+------------------------------------------+
| Property | Value |
+--------------------------------+------------------------------------------+
| attached_servers | ['07715bd4-6b78-43bd-925f-9e3638daf0a7'] |
| attachment_ids | ['82074978-cc6f-4389-a1df-ea8e05cfecab'] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-11-09T10:58:31.000000 |
| description | None |
| encrypted | False |
| id | 4e59408d-a517-46c9-8589-91f71ec3182b |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | nice |
| os-vol-host-attr:host | control@3parssd-1#SSD_r6 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 63ff5e00e4d84d058a15e5b20c67dec8 |
| replication_status | None |
| size | 500 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| updated_at | 2021-11-09T10:58:50.000000 |
| user_id | c6eaad2425fa4d13a888b3d27d2a30d5 |
| volume_type | 3parssd |
+--------------------------------+------------------------------------------+

hpe
* name: TllAjaUXRsmFiZH3HsMYKw
* lun: 21
* WWN: 60002AC0000000000000058B00027A9

[root@compute03 ~]# docker exec -it nova_libvirt virsh domblklist 2
 目标 源
-------------------------------------------------------------------------
 vda /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000057900027a92
 vdb /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000058b00027a92

[root@compute03 ~]# ll /dev/disk/by-path/ | grep lun-21
lrwxrwxrwx. 1 root root 10 Nov 9 18:58 pci-0000:37:00.0-fc-0x20310002ac027a92-lun-21 -> ../../sdcj
lrwxrwxrwx. 1 root root 10 Nov 9 18:58 pci-0000:37:00.0-fc-0x21310002ac027a92-lun-21 -> ../../sdck
lrwxrwxrwx. 1 root root 10 Nov 9 18:58 pci-0000:86:00.0-fc-0x20320002ac027a92-lun-21 -> ../../sdcl
lrwxrwxrwx. 1 root root 10 Nov 9 18:58 pci-0000:86:00.0-fc-0x21320002ac027a92-lun-21 -> ../../sdcm

[root@compute03 ~]# ll /dev/disk/by-id/ | grep -i 60002AC0000000000000058B00027A9
lrwxrwxrwx. 1 root root 11 Nov 9 18:58 dm-uuid-mpath-360002ac0000000000000058b00027a92 -> ../../dm-22
lrwxrwxrwx. 1 root root 10 Nov 9 18:58 scsi-360002ac0000000000000058b00027a92 -> ../../sdcm
lrwxrwxrwx. 1 root root 10 Nov 9 18:58 wwn-0x60002ac0000000000000058b00027a92 -> ../../sdcm

## during retype：

[root@control01 ~]# cinder list | grep nice
| 1829e4f0-60d1-4f6b-b37d-ed34a3294d19 | attaching | nice | 500 | 3parfc | false | |
| 4e59408d-a517-46c9-8589-91f71ec3182b | retyping | nice | 500 | 3parssd | false | 07715bd4-6b78-43bd-925f-9e3638daf0a7 |

[root@control01 ~]# cinder show 4e59408d-a517-46c9-8589-91f71ec3182b
+--------------------------------+------------------------------------------+
| Property | Value |
+--------------------------------+------------------------------------------+
| attached_servers | ['07715bd4-6b78-43bd-925f-9e3638daf0a7'] |
| attachment_ids | ['82074978-cc6f-4389-a1df-ea8e05cfecab'] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-11-09T10:58:31.000000 |
| description | None |
| encrypted | False |
| id | 4e59408d-a517-46c9-8589-91f71ec3182b |
| metadata | |
| migration_status | migrating |
| multiattach | False |
| name | nice |
| os-vol-host-attr:host | control@3parssd-1#SSD_r6 |
| os-vol-mig-status-attr:migstat | migrating |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 63ff5e00e4d84d058a15e5b20c67dec8 |
| replication_status | None |
| size | 500 |
| snapshot_id | None |
| source_volid | None |
| status | retyping |
| updated_at | 2021-11-10T02:51:23.000000 |
| user_id | c6eaad2425fa4d13a888b3d27d2a30d5 |
| volume_type | 3parssd |
+--------------------------------+------------------------------------------+

[root@control01 ~]# cinder show 1829e4f0-60d1-4f6b-b37d-ed34a3294d19
+--------------------------------+---------------------------------------------+
| Property | Value |
+--------------------------------+---------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-11-09T10:58:31.000000 |
| description | None |
| encrypted | False |
| id | 1829e4f0-60d1-4f6b-b37d-ed34a3294d19 |
| metadata | |
| migration_status | target:4e59408d-a517-46c9-8589-91f71ec3182b |
| multiattach | False |
| name | nice |
| os-vol-host-attr:host | control@3parfc-1#FC_r6 |
| os-vol-mig-status-attr:migstat | target:4e59408d-a517-46c9-8589-91f71ec3182b |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 63ff5e00e4d84d058a15e5b20c67dec8 |
| replication_status | None |
| size | 500 |
| snapshot_id | None |
| source_volid | None |
| status | attaching |
| updated_at | 2021-11-10T02:51:27.000000 |
| user_id | c6eaad2425fa4d13a888b3d27d2a30d5 |
| volume_type | 3parfc |
+--------------------------------+---------------------------------------------+

hpe:
* name: TllAjaUXRsmFiZH3HsMYKw
* lun: 21
* WWN: 60002AC0000000000000058B00027A9

* name: GCnk8GDRT2uzfe00oylNGQ
* lun: 22
* wwn: 60002AC0000000000000058C00027A92

[root@compute03 ~]# docker exec -it nova_libvirt virsh blockjob 2 /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000058b00027a92
块复制: [ 4 %]

[root@compute03 ~]# docker exec -it nova_libvirt virsh domblklist 2 目标 源
-------------------------------------------------------------------------
 vda /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000057900027a92
 vdb /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000058b00027a92

[root@compute03 ~]# ll /dev/disk/by-path/ | grep lun-21
lrwxrwxrwx. 1 root root 10 Nov 9 18:58 pci-0000:37:00.0-fc-0x20310002ac027a92-lun-21 -> ../../sdcj
lrwxrwxrwx. 1 root root 10 Nov 9 18:58 pci-0000:37:00.0-fc-0x21310002ac027a92-lun-21 -> ../../sdck
lrwxrwxrwx. 1 root root 10 Nov 9 18:58 pci-0000:86:00.0-fc-0x20320002ac027a92-lun-21 -> ../../sdcl
lrwxrwxrwx. 1 root root 10 Nov 9 18:58 pci-0000:86:00.0-fc-0x21320002ac027a92-lun-21 -> ../../sdcm

[root@compute03 ~]# ll /dev/disk/by-path/ | grep lun-22
lrwxrwxrwx. 1 root root 10 Nov 10 10:51 pci-0000:37:00.0-fc-0x20310002ac027a92-lun-22 -> ../../sdcn
lrwxrwxrwx. 1 root root 10 Nov 10 10:51 pci-0000:37:00.0-fc-0x21310002ac027a92-lun-22 -> ../../sdco
lrwxrwxrwx. 1 root root 10 Nov 10 10:51 pci-0000:86:00.0-fc-0x20320002ac027a92-lun-22 -> ../../sdcp
lrwxrwxrwx. 1 root root 10 Nov 10 10:51 pci-0000:86:00.0-fc-0x21320002ac027a92-lun-22 -> ../../sdcq

avg-cpu: %user %nice %system %iowait %steal %idle
           2.12 0.00 1.30 0.00 0.00 96.58

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-22 0.00 0.00 119.00 0.00 119.00 0.00 2048.00 0.24 1.98 1.98 0.00 0.73 8.70
dm-47 0.00 0.00 0.00 119.00 0.00 119.00 2048.00 15.74 128.50 0.00 128.50 8.40 100.00

## after retype：

[root@control01 ~]# cinder list | grep nice
| 1829e4f0-60d1-4f6b-b37d-ed34a3294d19 | available | nice | 500 | 3parfc | false | |
| 4e59408d-a517-46c9-8589-91f71ec3182b | in-use | nice | 500 | 3parfc | false | 07715bd4-6b78-43bd-925f-9e3638daf0a7 |

[root@control01 ~]# cinder show 1829e4f0-60d1-4f6b-b37d-ed34a3294d19
+--------------------------------+--------------------------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-11-09T10:58:31.000000 |
| description | migration src for 4e59408d-a517-46c9-8589-91f71ec3182b |
| encrypted | False |
| id | 1829e4f0-60d1-4f6b-b37d-ed34a3294d19 |
| metadata | |
| migration_status | deleting |
| multiattach | False |
| name | nice |
| os-vol-host-attr:host | control@3parssd-1#SSD_r6 |
| os-vol-mig-status-attr:migstat | deleting |
| os-vol-mig-status-attr:name_id | 4e59408d-a517-46c9-8589-91f71ec3182b |
| os-vol-tenant-attr:tenant_id | 63ff5e00e4d84d058a15e5b20c67dec8 |
| replication_status | None |
| size | 500 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2021-11-10T03:02:55.000000 |
| user_id | c6eaad2425fa4d13a888b3d27d2a30d5 |
| volume_type | 3parfc |
+--------------------------------+--------------------------------------------------------+
[root@control01 ~]# cinder show 4e59408d-a517-46c9-8589-91f71ec3182b
+--------------------------------+------------------------------------------+
| Property | Value |
+--------------------------------+------------------------------------------+
| attached_servers | ['07715bd4-6b78-43bd-925f-9e3638daf0a7'] |
| attachment_ids | ['330ee88e-017e-4913-a546-45fc1086c57f'] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-11-09T10:58:31.000000 |
| description | None |
| encrypted | False |
| id | 4e59408d-a517-46c9-8589-91f71ec3182b |
| metadata | |
| migration_status | success |
| multiattach | False |
| name | nice |
| os-vol-host-attr:host | control@3parfc-1#FC_r6 |
| os-vol-mig-status-attr:migstat | success |
| os-vol-mig-status-attr:name_id | 1829e4f0-60d1-4f6b-b37d-ed34a3294d19 |
| os-vol-tenant-attr:tenant_id | 63ff5e00e4d84d058a15e5b20c67dec8 |
| replication_status | None |
| size | 500 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| updated_at | 2021-11-10T03:02:55.000000 |
| user_id | c6eaad2425fa4d13a888b3d27d2a30d5 |
| volume_type | 3parfc |
+--------------------------------+------------------------------------------+

hpe:
* name: TllAjaUXRsmFiZH3HsMYKw
* lun: there is no export
* WWN: 60002AC0000000000000058B00027A9

* name: GCnk8GDRT2uzfe00oylNGQ
* lun: 22
* wwn: 60002AC0000000000000058C00027A92

[root@compute03 ~]# docker exec -it nova_libvirt virsh domblklist 2
 目标 源
-------------------------------------------------------------------------
 vda /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000057900027a92
 vdb /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000058c00027a92

[root@compute03 ~]# ll /dev/disk/by-path/ | grep lun-21
[root@compute03 ~]# ll /dev/disk/by-path/ | grep lun-22
lrwxrwxrwx. 1 root root 10 Nov 10 10:51 pci-0000:37:00.0-fc-0x20310002ac027a92-lun-22 -> ../../sdcn
lrwxrwxrwx. 1 root root 10 Nov 10 10:51 pci-0000:37:00.0-fc-0x21310002ac027a92-lun-22 -> ../../sdco
lrwxrwxrwx. 1 root root 10 Nov 10 10:51 pci-0000:86:00.0-fc-0x20320002ac027a92-lun-22 -> ../../sdcp
lrwxrwxrwx. 1 root root 10 Nov 10 10:51 pci-0000:86:00.0-fc-0x21320002ac027a92-lun-22 -> ../../sdcq

## after migration or detach/attach：

[root@compute03 ~]# docker exec -it nova_libvirt virsh domblklist 7
 目标 源
-------------------------------------------------------------------------
 vda /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000057900027a92
 vdb /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000058b00027a92

[root@compute03 ~]# ll /dev/disk/by-path/ | grep lun-22
[root@compute03 ~]# ll /dev/disk/by-path/ | grep lun-21
[root@compute03 ~]# ll /dev/disk/by-path/ | grep lun-21
lrwxrwxrwx. 1 root root 10 Nov 10 12:03 pci-0000:37:00.0-fc-0x20310002ac027a92-lun-21 -> ../../sdcf
lrwxrwxrwx. 1 root root 10 Nov 10 12:03 pci-0000:37:00.0-fc-0x21310002ac027a92-lun-21 -> ../../sdcg
lrwxrwxrwx. 1 root root 10 Nov 10 12:03 pci-0000:86:00.0-fc-0x20320002ac027a92-lun-21 -> ../../sdch
lrwxrwxrwx. 1 root root 10 Nov 10 12:03 pci-0000:86:00.0-fc-0x21320002ac027a92-lun-21 -> ../../sdci

hpe:
* name: TllAjaUXRsmFiZH3HsMYKw
* lun: 21
* WWN: 60002AC0000000000000058B00027A9

* name: GCnk8GDRT2uzfe00oylNGQ
* lun: 22
* wwn: 60002AC0000000000000058C00027A92