A volume remains attached and cannot be detached after attaching it fails

[Takashi Natsume]

A volume has been attached in nova after attaching it fails.
But its status is 'available' in cinder.
And then, the volume cannot be detached
because an error(Invalid volume: Volume must be attached in order to detach) occurs.

The cause of the failure of attaching the volume is a rpc call timeout between nova-api and nova-compute
in 'reserve_block_device_name' call.
(a 'reserve_block_device_name' rpc call is in '_attach_volume' method in nova/compute/api.py)

A rpc call timeout occurs because of waiting to get the semaphore in nova-compute.
It takes long time when many requests for same VM instance comes.

The cause of the failure of detaching the volume is that the record remains in a 'block_device_mapping' table.

stack@devstack-juno:/opt/devstack$ nova list
+--------------------------------------+---------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+-------------------+
| 16eb66e3-3cda-44c5-9134-cb759ffb211b | server1 | ACTIVE | - | Running | public=10.0.2.195 |
+--------------------------------------+---------+--------+------------+-------------+-------------------+
stack@devstack-juno:/opt/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 474964ed-e334-44c0-a904-4612c52e6618 | available | TEST | 1 | lvmdriver-1 | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
stack@devstack-juno:/opt/devstack$ nova volume-attach server1 474964ed-e334-44c0-a904-4612c52e6618
ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-c371e287-5e28-410f-812c-43c193bdcffc)

* RPC time out occurred between nova-api and nova-compute in 'reserve_block_device_name' call.

stack@devstack-juno:/opt/devstack$ nova show server1
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack-juno |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack-juno |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-02-24T07:54:06.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-02-24T07:53:54Z |
| flavor | m1.tiny (1) |
| hostId | 358f8c0b777e6a5218b4e297676604bbc730afe5bcc2760e64a32408 |
| id | 16eb66e3-3cda-44c5-9134-cb759ffb211b |
| image | cirros-0.3.2-x86_64-uec (7f38517f-adb5-4fe6-86be-f60d9065f00d) |
| key_name | - |
| metadata | {} |
| name | server1 |
| os-extended-volumes:volumes_attached | [{"id": "474964ed-e334-44c0-a904-4612c52e6618"}] |
| progress | 0 |
| public network | 10.0.2.195 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 47b5fac1e13c4e97b36cd76bcb4ca661 |
| updated | 2015-02-24T07:54:06Z |
| user_id | 94cf43ae770c4a32bf8ce817cd2608cb |
+--------------------------------------+----------------------------------------------------------------+
stack@devstack-juno:/opt/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 474964ed-e334-44c0-a904-4612c52e6618 | available | TEST | 1 | lvmdriver-1 | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
stack@devstack-juno:/opt/devstack$ nova volume-detach server1 474964ed-e334-44c0-a904-4612c52e6618
ERROR (BadRequest): Invalid volume: Volume must be attached in order to detach. (HTTP 400) (Request-ID: req-cdc4b447-60b8-424c-b151-73d2f2874068)
stack@devstack-juno:/opt/devstack$ nova show server1
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack-juno |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack-juno |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-02-24T07:54:06.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-02-24T07:53:54Z |
| flavor | m1.tiny (1) |
| hostId | 358f8c0b777e6a5218b4e297676604bbc730afe5bcc2760e64a32408 |
| id | 16eb66e3-3cda-44c5-9134-cb759ffb211b |
| image | cirros-0.3.2-x86_64-uec (7f38517f-adb5-4fe6-86be-f60d9065f00d) |
| key_name | - |
| metadata | {} |
| name | server1 |
| os-extended-volumes:volumes_attached | [{"id": "474964ed-e334-44c0-a904-4612c52e6618"}] |
| progress | 0 |
| public network | 10.0.2.195 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 47b5fac1e13c4e97b36cd76bcb4ca661 |
| updated | 2015-02-24T07:54:06Z |
| user_id | 94cf43ae770c4a32bf8ce817cd2608cb |
+--------------------------------------+----------------------------------------------------------------+

[Environment]
OS: Ubuntu 14.04 LTS
nova: stable/juno(2014.2.1)
cinder: stable/juno(2014.2.1)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/167510

[OpenStack Infra (hudson-openstack)]

Change abandoned by Joe Gordon (<email address hidden>) on branch: master
Review: https://review.openstack.org/167510
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Sean Dague (sdague)]

Automatically discovered version juno in description. If this is incorrect, please update the description to include 'nova version: ...'

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/692940

[Matt Riedemann (mriedem)]

This is likely fixed in Ussuri with this change: https://review.opendev.org/#/c/693537/

A better long-term fix is to create the BDM in the API: https://review.opendev.org/#/c/692940/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/692940