ceph backend volume migration fails when in the same pool

[xinliang]

We want to disable/delete some volume ceph cinder_volume/backup services(or node who runs these two services). From note[1], we know that we should migrate the volume to another host first before disable/delete the cinder_volume/backup.

Be it seems that migrating volume which in state available or in-use on the same ceph pool fails.

Tested on Rocky/Stein/Train all fail.
We deploy openstack with kolla-ansible(internal ceph)

[1] "If you plan to decommission a block storage node, you must stop the cinder volume service on the node after performing the migration." at https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-migration.html

[xinliang (xin3liang)]

migrate volume with cli and migration_status in error state

linaro@j12-d05:~$ openstack volume migrate --host j12-d05@rbd-1#rbd-1 6ff6e918-dedd-4da9-8122-43d729987e61
linaro@j12-d05:~$ openstack volume show 6ff6e918-dedd-4da9-8122-43d729987e61
+--------------------------------+--------------------------------------+
| Field | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-04-08T02:14:49.000000 |
| description | None |
| encrypted | False |
| id | 6ff6e918-dedd-4da9-8122-43d729987e61 |
| migration_status | error |
| multiattach | False |
| name | test-vol-03 |
| os-vol-host-attr:host | uk-dc-cavium-07@rbd-1#rbd-1 |
| os-vol-mig-status-attr:migstat | error |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | fe2459242c0f4fcf83e499d7ea2f5eba |
| properties | |
| replication_status | None |
| size | 20 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| type | None |
| updated_at | 2020-04-08T02:52:33.000000 |
| user_id | 1010a53f5266403e8c3a8937d0a05cf2 |
+--------------------------------+--------------------------------------+

[xinliang (xin3liang)]

cinder-volume.log:

default] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=5). _do_conn /var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:336
2020-04-08 03:24:02.041 34 ERROR cinder.volume.drivers.rbd [req-abde6ea7-0ac0-4e25-b42b-a8c5fa4fb730 1010a53f5266403e8c3a8937d0a05cf2 fe2459242c0f4fcf83e499d7ea2f5eba - default default] Error copying rbd image volume-0f19481a-32cb-4aa7-9b19-5e4100d34cd0 to target pool volumes.: ImageExists: [errno 17] error copying image volume-0f19481a-32cb-4aa7-9b19-5e4100d34cd0 to volume-0f19481a-32cb-4aa7-9b19-5e4100d34cd0
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server [req-abde6ea7-0ac0-4e25-b42b-a8c5fa4fb730 1010a53f5266403e8c3a8937d0a05cf2 fe2459242c0f4fcf83e499d7ea2f5eba - default default] Exception during message handling: ImageExists: [errno 17] error copying image volume-0f19481a-32cb-4aa7-9b19-5e4100d34cd0 to volume-0f19481a-32cb-4aa7-9b19-5e4100d34cd0
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/manager.py", line 2397, in migrate_volume
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server volume.save()
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/manager.py", line 2375, in migrate_volume
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server host)
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 1688, in migrate_volume
2020-04-08 03:24:02.330 34 ERROR oslo_messaging.rpc.server self.RBDPr...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/720423

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720423
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ea5a9c35ea0fe5456cc002730cb72cd3bfca218b
Submitter: Zuul
Branch: master

commit ea5a9c35ea0fe5456cc002730cb72cd3bfca218b
Author: Xinliang Liu <email address hidden>
Date: Thu Apr 16 09:13:22 2020 +0000

    Fix volume migration fails in the same ceph RBD pool

    For the same ceph RBD pool, volume migration only needs to update
    volume’s host value to destination host.
    In this case, we can migrate volume in available or in-use status.

    Change-Id: I01039b7651a354761d034a9118b55b96cf32445e
    Closes-Bug: #1871524

[Radosław Piliszek (yoctozepto)]

Nothing to fix in Kolla-Ansible.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/724599

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/724599
Reason: Needs to go to stable/ussuri first, then can be backported once it is merged there.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724599
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=a11460f36de9a86abc994c738cd918de4933245c
Submitter: Zuul
Branch: stable/train

commit a11460f36de9a86abc994c738cd918de4933245c
Author: Xinliang Liu <email address hidden>
Date: Thu Apr 16 09:13:22 2020 +0000

    Fix volume migration fails in the same ceph RBD pool

    For the same ceph RBD pool, volume migration only needs to update
    volume’s host value to destination host.
    In this case, we can migrate volume in available or in-use status.

    Change-Id: I01039b7651a354761d034a9118b55b96cf32445e
    Closes-Bug: #1871524
    (cherry picked from commit ea5a9c35ea0fe5456cc002730cb72cd3bfca218b)

    Fix nits from RBD volume migration changes

    This addresses some minor issues in
    I01039b7651a354761d034a9118b55b96cf32445e

    Change-Id: I98dbc1f843a99405f12b6b9d1d189e27f5cdce6c
    Signed-off-by: Sean McGinnis <email address hidden>
    (cherry picked from commit 8819a1d6b4c5aa48f73beb344764becf511bf2e8)
    (cherry picked from commit d3b1d5c4f08de760f29c257dc1e84524d3ca3948)