Concurrent requests to attach the same non-multiattach volume to multiple instances can succeed
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Fix Released
|
High
|
Matt Riedemann | ||
Bug Description
Description
===========
Discovered this by chance yesterday, at first glance this appears to be due to a lack of locking within c-api when we initial create the attachments and no additional validation when we update with the connector later in the attach flow. Reporting this against both nova and cinder for now.
$ nova volume-attach 77c092c2-
[1] 24949
+------
| Property | Value |
+------
| device | /dev/vdb |
| id | b4240f39-
| serverId | 91a7c490-
| volumeId | b4240f39-
+------
+------
| Property | Value |
+------
| device | /dev/vdb |
| id | b4240f39-
| serverId | 77c092c2-
| volumeId | b4240f39-
+------
$ cinder show b4240f39-
+------
| Property | Value |
+------
| attached_servers | ['91a7c490-
| attachment_ids | ['31b8c16f-
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2018-04-
| description | None |
| encrypted | False |
| id | b4240f39-
| metadata | attached_mode : rw |
| migration_status | None |
| multiattach | False |
| name | None |
| os-vol-
| os-vol-
| os-vol-
| os-vol-
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| updated_at | 2018-04-
| user_id | 57293b0839da449
| volume_type | lvmdriver-1 |
+------
$ ll /dev/disk/
lrwxrwxrwx. 1 root root 9 Apr 9 15:04 /dev/disk/
$ sudo virsh domblklist 77c092c2-
Target Source
-------
vda /opt/stack/
vdb /dev/sdc
$ sudo virsh domblklist 91a7c490-
Target Source
-------
vda /opt/stack/
vdb /dev/sdc
Steps to reproduce
==================
$ nova volume-attach $instance_1_uuid $volume_uuid & nova volume-attach $instance_2_uuid $volume_uuid
Expected result
===============
Only one of the requests succeeds.
Actual result
=============
Both requests succeed.
Environment
===========
1. Exact version of OpenStack you are running. See the following
list for all releases: http://
$ cd ../nova
$ git describe --tags
17.0.0.
$ cd ../cinder/
$ git describe --tags
12.0.0.
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
Libvirt + KVM
2. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?
LVM + iSCSI
3. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)
N/A
Changed in cinder: | |
importance: | Undecided → High |
Changed in nova: | |
status: | New → Invalid |
no longer affects: | nova |
FWIW this does reproduce on stable/queens but not stable/pike.