Repeated volume attach can cause u'message': u'The supplied device path (/dev/vdc) is in use.'
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Expired
|
Undecided
|
Unassigned |
Bug Description
If you attach and detach the same volume to same server in loop, the n-api may report the device name is already use.
I used the following stress test https:/
With the blow configuration.
[{"action": "tempest.
"threads": 1,
"use_admin": false,
"use_
"kwargs": {"vm_extra_args": {},
}
]
The issue happens with all config options, but this is the fastest way.
The issue can happen even after the device disappearance confirmed via ssh, ie. not listed in /proc/partitions anymore.
I used similar devstack setup as the gate uses with multiple nova api/cond workers.
NOTE: libvirt/qemu/linux disregards the device name.
For reproducing the issue
1. add tempest to enabled devstack services.
2. apply the https:/
3. change the logging options in the tempst.conf [DEFAULT]
4. ./tempest/
If 128 attempt is not enough, you can increase the number of threads (in the json config) or the attempts as cli option.
affects: | openstack-ci → nova |
tags: | added: volumes |
Changed in nova: | |
assignee: | nobody → jichenjc (jichenjc) |
summary: |
- Repeated volume attche can cause u'message': u'The supplied device path + Repeated volume attach can cause u'message': u'The supplied device path (/dev/vdc) is in use.' |
Changed in nova: | |
assignee: | jichenjc (jichenjc) → nobody |
Changed in nova: | |
status: | Triaged → Confirmed |
Yep - there seems to be a race window (not huge to be honest) that can cause this kind of issue with stress testing. The solution is to do two things:
1) In the API part of attach - we should reserve the volume before we call out to the compute service to reserve the device name
2) I the compute detach path - We should release the device name before we release the volume in detach
I will target this for Juno as it is somewhat risky at this point. It should be an easy backport to Icehouse once we get it in.