Volume of 'in-use' remain by a timeout during the attach

[Yasuaki Nagata]

Remain as the 'in-use' volume due to timeout during the 'attach'.
If the problem occurs, The instance can't detach/attach the volume.
 detach -> volume_id not found
 attach -> libvirtError: Requested operation is not valid: target vdb already exists
 (other volume)

This problems are caused by the mismatching DB because c-vol does not rollback it when happend the timeout of RPC(CALL).
At first, c-vol takes time in the attaching process, and the attaching process failed in the c-api by timeout.
But in this time, the attaching process does not failed yet in the c-vol.

By this, BDM is deleted and volume is updated to in-use.
(If successful the attaching process in the c-vol.)

<Repro>
used master
 cinder: commit d4b77484c5d41f207d54f40dcdd530fb8a1b1ea6
 nova : commit eaeecdaf4743463888c3ee24fb08128eac15dee7

1. attach volume
   (in cinder/volume/manage.py def attach_volume)
    note: I reproduced this problem by inserting sleep in this method.
2. RPC(CALL) timeout in the c-api(attach_volume process takes time beyond the rpc_response_timeout)

<About the volume and BDM >
block_device_mapping
+---------------------+----+-------------+-----------------------+--------------------------------------+-----------------+--------------------------------------+---------+
| deleted_at | id | device_name | delete_on_termination | volume_id | connection_info | instance_uuid | deleted |
+---------------------+----+-------------+-----------------------+--------------------------------------+-----------------+--------------------------------------+---------+
| NULL | 1 | /dev/vda | 1 | NULL | NULL | 4683d4fb-758c-459e-9def-b8d247a56954 | 0 |
| 2015-03-17 06:12:36 | 2 | /dev/vdb | 0 | 46d1bfbb-bdf2-472f-8bf6-2d2367b1edb1 | NULL | 4683d4fb-758c-459e-9def-b8d247a56954 | 2 |
+---------------------+----+-------------+-----------------------+--------------------------------------+-----------------+--------------------------------------+---------+
volumes
+------------+---------+--------------------------------------+--------+---------------+---------------+-----------------------------------------------------------------------------------------------+--------------------------------------------+
| deleted_at | deleted | id | status | attach_status | terminated_at | provider_location | provider_auth |
+------------+---------+--------------------------------------+--------+---------------+---------------+-----------------------------------------------------------------------------------------------+--------------------------------------------+
| NULL | 0 | 46d1bfbb-bdf2-472f-8bf6-2d2367b1edb1 | in-use | attached | NULL | 192.168.58.172:3260,4 iqn.2010-10.org.openstack:volume-46d1bfbb-bdf2-472f-8bf6-2d2367b1edb1 1 | CHAP 2W7r5XQcZJ5BHVctM8YY NogCxmq4VswXWHWE |
+------------+---------+--------------------------------------+--------+---------------+---------------+-----------------------------------------------------------------------------------------------+--------------------------------------------+

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/167183

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/167185

[Seiji Aguchi (seiji-aguchi)]

Hi,
---
    note: I reproduced this problem by inserting sleep in this method.
---

Why do you need to reproduce it by changing the code?
If possible, can you share an actual use case scenario that this problem happens?

Seiji

[OpenStack Infra (hudson-openstack)]

Change abandoned by Joe Gordon (<email address hidden>) on branch: master
Review: https://review.openstack.org/167183
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mike Perez (<email address hidden>) on branch: master
Review: https://review.openstack.org/167185
Reason: Over a month with no update.