Deleting 2 instances with a common multi-attached volume can leave the volume attached
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Confirmed
|
Medium
|
Lee Yarwood |
Bug Description
CAVEAT: The following is only from code inspection. I have not reproduced the issue.
During instance delete, we call:
driver.cleanup():
foreach volume:
_
if _should_
There is no volume-specific or global locking around _disconnect_volume that I can see in this call graph.
_should_
volume = cinder-
connection_count = count of volume.attachments where instance is on this host
As there is no locking between the above operation and the subsequent disconnect_
Also note that there are many call paths which call _disconnect_
The result would be that all attachments are deleted, but the volume remains attached to the host.
tags: | added: cinder volumes |
tags: | added: libvirt multiattach |
Changed in nova: | |
assignee: | nobody → Lee Yarwood (lyarwood) |
Yes it is a bug, i have tested on my system. below are the reproducing steps :
Precondition: ebb0-41d2- a435-6f56791a96 04, 0efcd163- 13ab-4b70- 9449-ab89301be1 cf
create two instance and attach them with a multiattach volume.
there instances id are : c78bdc4b-
1. delete first attachment :
[root@openstackq ~(keystone_admin)]# nova volume-detach c78bdc4b- ebb0-41d2- a435-6f56791a96 04 4361ce05- e325-40c3- 8b2c-5bcaeedf42 60
2. now, delete second attachment :
[root@openstackq ~(keystone_admin)]# nova volume-detach 0efcd163- 13ab-4b70- 9449-ab89301be1 cf 4361ce05- e325-40c3- 8b2c-5bcaeedf42 60
Second step executed without any message.
however, if we check output of cinder list, volume's status is shown as a "deattaching". but after few seconds it changes from "deattaching" to "in-use".
3.
[root@openstackq ~(keystone_admin)]# cinder list ------- ------- ------- ------- ----+-- ------- --+---- ------- +------ +------ ------- +------ ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ---+ ------- ------- ------- ------- ----+-- ------- --+---- ------- +------ +------ ------- +------ ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ---+ e325-40c3- 8b2c-5bcaeedf42 60 | detaching | - | 1 | multiattach | false | 0efcd163- 13ab-4b70- 9449-ab89301be1 cf | a08f-46d4- 88f8-c302d2ec9d 75 | in-use | sharedvol | 1 | multiattach | false | 2bf44c23- 87b4-4d84- 81b6-bc2b1f792d 12,c79ad598- be11-490c- 8fd0-58c6d72bbe 4f | ------- ------- ------- ------- ----+-- ------- --+---- ------- +------ +------ ------- +------ ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ---+ ------- ------- ------- ------- ----+-- ------- --+---- ------- +------ +------ ------- +------ ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ---+ ------- ------- ------- ------- ----+-- ------- --+---- ------- +------ +------ ------- +------ ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ---+ e325-40c3- 8b2c-5bcaeedf42 60 | detaching | - | 1 | multiattach | false | 0efcd163- 13ab-4b70- 9449-ab89301be1 cf | a08f-46d4- 88f8-c302d2ec9d 75 | in-use | sharedvol | 1 | multiattach | false | 2bf44c23- 87b4-4d84- 81b6-bc2b1f792d 12,c79ad598- be11-490c- 8fd0-58c6d72bbe 4f | ------- ------- ------- ------- ----+-- ------- --+---- ------- +------ +------ ------- +------ ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ---+ ------- ------- ------- ------- ----+-- ------- --+---- ------- +------ +------ ------- +------ ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ---+
+------
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+------
| 4361ce05-
| 946009e9-
+------
[root@openstackq ~(keystone_admin)]# cinder list
+------
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+------
| 4361ce05-
| 946009e9-
+------
[root@openstackq ~(keystone_admin)]# cinder list
+------
| ID ...