migrate attached volume fails

[Ramy Asselin]

Migrating an attached volume from one host to another doesn't work.

Steps to reproduce:
1. Create a volume
2. Attach to a host
3. Migrate the volume to another host.

Actual results:
1. Volume still on original host
2. A New volume is created also on the original host
3. No errors noticed in the c-vol log file.

master+* ± cinder migrate d862ea9d-cc2b-4460-9ed8-2126fe839f07 ramy-stack2@3pariscsi1
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 1db2976f-264c-466b-84eb-1c221d17adbf | available | my_vol | 2 | None | false | |
| d862ea9d-cc2b-4460-9ed8-2126fe839f07 | in-use | my_vol | 2 | None | false | 826709db-40d8-477c-8f8f-af81df9bc6b0 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+

 master+* ± cinder show 1db2976f-264c-466b-84eb-1c221d17adbf
+--------------------------------+---------------------------------------------+
| Property | Value |
+--------------------------------+---------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2014-02-22T00:44:21.000000 |
| display_description | |
| display_name | my_vol |
| id | 1db2976f-264c-466b-84eb-1c221d17adbf |
| metadata | {} |
| os-vol-host-attr:host | ramy-stack2@3pariscsi1 |
| os-vol-mig-status-attr:migstat | target:d862ea9d-cc2b-4460-9ed8-2126fe839f07 |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 63be499ec75e48d597b83679112d32bd |
| size | 2 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| volume_type | None |
+--------------------------------+---------------------------------------------+

cinder show d862ea9d-cc2b-4460-9ed8-2126fe839f07
+--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'device': u'/dev/vdb', u'server_id': u'826709db-40d8-477c-8f8f-af81df9bc6b0', u'id': u'd862ea9d-cc2b-4460-9ed8-2126fe839f07', u'host_name': None, u'volume_id': u'd862ea9d-cc2b-4460-9ed8-2126fe839f07'}] |
| availability_zone | nova |
| bootable | false |
| created_at | 2014-02-22T00:44:21.000000 |
| display_description | |
| display_name | my_vol |
| id | d862ea9d-cc2b-4460-9ed8-2126fe839f07 |
| metadata | {u'readonly': u'False', u'attached_mode': u'rw'} |
| os-vol-host-attr:host | ramy-stack2@3pariscsi2 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 63be499ec75e48d597b83679112d32bd |
| size | 2 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| volume_type | None |
+--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[Ramy Asselin (ramy-asselin)]

May be duplicate of https://bugs.launchpad.net/cinder/+bug/1278289
Awaiting confirmation from Avishay.

[Avishay Traeger (avishay-il)]

Ramy: Migration of attached volumes is handled by Nova. Can you please see if there is anything in the Nova logs?

[Ramy Asselin (ramy-asselin)]

Avishay, I reproduced it. Here is the nova log "snippet":
http://paste.openstack.org/show/69927/

[Avishay Traeger (avishay-il)]

OK, so this is your error:
virDomain instance has no attribute 'blockRebase'

This means that your libvirt isn't up to date - you need to use at least version 0.9.10 for this feature to work. If this isn't documented anywhere it should be...

[Ramy Asselin (ramy-asselin)]

So in this case the migration fails. Should the new volumes be deleted? Currently they cannot because they're still 'migrating'.
Manual cleanup fails as well. How do you remove these?

[stack@ramy-stack2:~/openstack/cinder/cinder/volume/drivers/san/hp]↥ master ± cinder delete 7e3d3af6-19a1-41e6-87b0-e03aa8a65f48
Delete for volume 7e3d3af6-19a1-41e6-87b0-e03aa8a65f48 failed: Invalid volume: Volume cannot be deleted while migrating (HTTP 400) (Request-ID: req-5778f857-3378-47ca-9989-4afb26f88f54)
ERROR: Unable to delete any of the specified volumes.
[stack@ramy-stack2:~/openstack/cinder/cinder/volume/drivers/san/hp]↥ master 1 ± cinder force-delete 7e3d3af6-19a1-41e6-87b0-e03aa8a65f48
Delete for volume 7e3d3af6-19a1-41e6-87b0-e03aa8a65f48 failed: Invalid volume: Volume cannot be deleted while migrating (HTTP 400) (Request-ID: req-b6c50996-3aa8-46df-8ad5-3ea1162a67a4)
ERROR: Unable to force delete any of the specified volumes.
[stack@ramy-stack2:~/openstack/cinder/cinder/volume/drivers/san/hp]↥ master 1 ± cinder reset-state 7e3d3af6-19a1-41e6-87b0-e03aa8a65f48
[stack@ramy-stack2:~/openstack/cinder/cinder/volume/drivers/san/hp]↥ master ± cinder delete 7e3d3af6-19a1-41e6-87b0-e03aa8a65f48
Delete for volume 7e3d3af6-19a1-41e6-87b0-e03aa8a65f48 failed: Invalid volume: Volume cannot be deleted while migrating (HTTP 400) (Request-ID: req-129c0f1a-03f0-462f-94b0-aa9f10c1fc83)
ERROR: Unable to delete any of the specified volumes.
[stack@ramy-stack2:~/openstack/cinder/cinder/volume/drivers/san/hp]↥ master 1 ± cinder force-delete 7e3d3af6-19a1-41e6-87b0-e03aa8a65f48
Delete for volume 7e3d3af6-19a1-41e6-87b0-e03aa8a65f48 failed: Invalid volume: Volume cannot be deleted while migrating (HTTP 400) (Request-ID: req-acba7e53-12e0-46e6-ad6d-650d44a3cc75)
ERROR: Unable to force delete any of the specified volumes.

[Avishay Traeger (avishay-il)]

Hmm that's not good. Nova may not be passing the call to Cinder to clean up. I will look into it.

[Ramy Asselin (ramy-asselin)]

My notes on working around this issue.

1. I'm using Ubuntu 12.04 which has 0.9.8

>apt-show-versions python-libvirt
python-libvirt/precise-security uptodate 0.9.8-2ubuntu17.17

ref: https://ask.openstack.org/en/question/8285/cannot-create-live-snapshot-with-error-virdomain-instance-has-no-attribute-blockrebase/

2. To get the latest version in 12.04, you need to follow these instructions:
https://wiki.ubuntu.com/ServerTeam/CloudArchive

>sudo add-apt-repository cloud-archive:havana
>sudo apt-get update
>sudo apt-get upgrade python-libvirt
>apt-show-versions python-libvirt
python-libvirt/precise-updates upgradeable from 0.9.8-2ubuntu17.17 to 1.1.1-0ubuntu8.5~cloud0

>aptitude (console-ui tool. Needed to mark the package to force the update (+), then execute update(g))

>apt-show-versions python-libvirt
python-libvirt/precise-updates uptodate 1.1.1-0ubuntu8.5~cloud0

..that didn't work completely. After more research, the correct answer seems to be:

>sudo apt-get dist-upgrade
http://askubuntu.com/questions/601/the-following-packages-have-been-kept-back-why-and-how-do-i-solve-it

Dependencies are now up to date, but still it doesn't seem to work:

http://paste.openstack.org/show/70326/

[Ramy Asselin (ramy-asselin)]

Avishay,

Seems n-cpu tries to delete the volume after restarting it, but it's failing. See logs here:
http://paste.openstack.org/show/70303/

thanks,
Ramy

[haruka tanizawa (h-tanizawa)]

What is the different with https://bugs.launchpad.net/cinder/+bug/1316079 ?
Duplicate?

[Ramy Asselin (ramy-asselin)]

Steps to reproduce are same/similar, but they have different stack traces.

[Avishay Traeger (avishay-il)]

I believe this is a duplicate of https://bugs.launchpad.net/cinder/+bug/1316079

The logs from after the restart show that Nova failed to find the Cinder endpoint, which seems to indicate cinder-api wasn't running, or some configuration issue.

Closing for now. Ramy if this still happens with the latest code, please re-open.