Cinder volume stuck in swap_volume

[Takashi Natsume]

Cinder volumes are stuck in 'attaching'/'detaching' state when 'swap volume' is executed.

A cinder volume is attached to a VM instance.
Then the cinder volume is swapped for a new volume by 'swap volume'.
Concretely the following API is called.

PUT /v2/{tenant_id}/servers/{server_id}/os-volume_attachments/{attachment_id}

After the API is called, the attached volume(old volume) becomes stuck in 'detaching' state
and the new volume becomes stuck in 'attaching' state.

[How to reproduce]
stack@devstack-kilo:/opt/devstack$ cinder list
+--------------------------------------+-----------+-------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------+------+-------------+----------+--------------------------------------+
| c3aff356-7545-444c-8b4f-33670b3e483c | in-use | TEST2 | 1 | lvmdriver-1 | false | adc00700-c7c8-4fe9-b3b4-df9beed40405 |
| da251e5b-a783-4f96-8b9f-a8db5dc070c1 | available | TEST1 | 1 | lvmdriver-1 | false | |
+--------------------------------------+-----------+-------+------+-------------+----------+--------------------------------------+
stack@devstack-kilo:/opt/devstack$ nova show server1
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack-kilo |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack-kilo |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-07-02T00:53:25.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | True |
| created | 2015-07-02T00:53:19Z |
| flavor | m1.tiny (1) |
| hostId | ea09a5e13b086e757a5c21f093c46d0aa6ae373d82a34ef7ac798816 |
| id | adc00700-c7c8-4fe9-b3b4-df9beed40405 |
| image | cirros-0.3.2-x86_64-uec (abe0afbf-7f82-4361-a308-a69d7206989f) |
| key_name | - |
| metadata | {} |
| name | server1 |
| os-extended-volumes:volumes_attached | [{"id": "c3aff356-7545-444c-8b4f-33670b3e483c"}] |
| progress | 0 |
| public network | 10.0.2.195 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 252be0c183fa42f788947cd18004ae63 |
| updated | 2015-07-02T00:53:25Z |
| user_id | 4f870a0e7b4a420b874fcb3f9718f040 |
+--------------------------------------+----------------------------------------------------------------+
stack@devstack-kilo:/opt/devstack$ curl -i -X PUT http://10.0.2.15:8774/v2/252be0c183fa42f788947cd18004ae63/servers/adc00700-c7c8-4fe9-b3b4-df9beed40405/os-volume_attachments/c3aff356-7545-444c-8b4f-33670b3e483c -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: e6f83c8ca44644a39d787fd7308f4920" -d '{"volumeAttachment": {"volumeId": "da251e5b-a783-4f96-8b9f-a8db5dc070c1"}}'
HTTP/1.1 202 Accepted
Content-Type: text/html; charset=UTF-8
Content-Length: 0
X-Compute-Request-Id: req-9300c544-95c1-4993-abb6-d0cd1a7bddcf
Date: Thu, 02 Jul 2015 00:59:16 GMT

stack@devstack-kilo:/opt/devstack$ cinder list
+--------------------------------------+-----------+-------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------+------+-------------+----------+--------------------------------------+
| c3aff356-7545-444c-8b4f-33670b3e483c | detaching | TEST2 | 1 | lvmdriver-1 | false | adc00700-c7c8-4fe9-b3b4-df9beed40405 |
| da251e5b-a783-4f96-8b9f-a8db5dc070c1 | attaching | TEST1 | 1 | lvmdriver-1 | false | |
+--------------------------------------+-----------+-------+------+-------------+----------+--------------------------------------+
stack@devstack-kilo:/opt/devstack$ nova show server1
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack-kilo |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack-kilo |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-07-02T00:53:25.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | True |
| created | 2015-07-02T00:53:19Z |
| flavor | m1.tiny (1) |
| hostId | ea09a5e13b086e757a5c21f093c46d0aa6ae373d82a34ef7ac798816 |
| id | adc00700-c7c8-4fe9-b3b4-df9beed40405 |
| image | cirros-0.3.2-x86_64-uec (abe0afbf-7f82-4361-a308-a69d7206989f) |
| key_name | - |
| metadata | {} |
| name | server1 |
| os-extended-volumes:volumes_attached | [{"id": "da251e5b-a783-4f96-8b9f-a8db5dc070c1"}] |
| progress | 0 |
| public network | 10.0.2.195 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 252be0c183fa42f788947cd18004ae63 |
| updated | 2015-07-02T00:59:31Z |
| user_id | 4f870a0e7b4a420b874fcb3f9718f040 |
+--------------------------------------+----------------------------------------------------------------+

[Environment]
OS: Ubuntu 14.04 LTS
nova: stable/kilo(commit d22167e08388f6f557484e469ab559255cc273c2)
cinder: stable/kilo(commit 6c055943e70675585ef45301318c47ccc1139ce9)

[Markus Zoeller (markus_z) (mzoeller)]

@Takashi NATSUME:

Since you are set as assignee, I switch the status to 'In Progress'.

[Takashi Natsume (natsume-takashi)]

IMO, the 'migrate_volume_completion' process in cinder should be fixed.

[Takashi Natsume (natsume-takashi)]

Cinder has 'multi-attach-volume' function.
So if attaching and detaching the volumes is executed in cinder, attachment-id or instance uuid is needed.
If attaching and detaching the volumes will be executed in 'migrate_volume_completion' ,
 attachment-id or instance uuid has to be passed.
In this case, API has to be changed.

So I will fix this bug by adding attaching and detaching the volumes in nova side.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/207385

[Matt Riedemann (mriedem)]

Removing liberty-rc-potential. This is a bug reported against kilo so we can backport to stable/liberty and stable/kilo once we have a fix, but this shouldn't hold up releasing liberty since it's not a regression in liberty.

[Takashi Natsume (natsume-takashi)]

Could any core reviewers review the following patch for this bug?

https://review.openstack.org/#/c/207385/

This fix should be backported to stable/liberty and stable/kilo.
It cannot be backported until the bug is fixed in master branch.
So this bug should be fixed in master branch as soon as possible.

[Ryan McNair (rdmcnair)]

Think this is a duplicate of https://bugs.launchpad.net/cinder/+bug/1489744 which a fix just landed in Cinder for.

I have not done anything to backport that bugfix - if we want I can propose that fix to the stable streams now.

Though look at the proposed Nova fix, since it has a hold of the BDM think this would work better with multiattach devices... so the other option would be reverting my patch and using this. Only strange part of doing it with proposed patch is the final attach/detach coming from Nova side half the time and Cinder side the other half.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/207385
Reason: Looks like this has been abandoned in favor of https://review.openstack.org/#/c/257135/ now.