Volume detach error when use NFS as the cinder backend
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Undecided
|
SamP |
Bug Description
Tested Environment
-------
OS: Ubuntu 14.04 LST
Cinder NFS driver:
volume_
Error description
-------
I used NFS as the cinder storage backend and successfully attached multiple volumes to nova instances.
However, when I tried to detach one them, I found following error on nova-compute.log.
-----------Error log------
2014-07-07 17:48:46.175 3195 ERROR nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
2014-07-07 17:48:46.175 3195 TRACE nova.virt.
-----------End of the Log--
For NFS volumes, every time you detach a volume, nova tries to umount the device path.
/nova/virt/
Line 632: class LibvirtNFSVolum
Line 653: def disconnect_
Line 661: utils.execute(
This works when the device path is not busy.
If the device path is busy (or in use), it should output a message to log and continue.
The problem is, Instead of output a log message, it raise exception and that cause the above error.
I think the reason is, the ‘if’ statement at Line 663 fails to catch the device busy massage from the content of the exc.message. It looking for the ‘target is busy’ in the exc.message, but umount error code returns ‘device is busy’.
Therefore, current code skip the ‘if’ statement and run the ‘else’ and raise the exception.
How to reproduce
-------
(1) Prepare a NFS share storage and set it as the storage backend of you cinder
(refer http://
In cinder.conf
volume_
nfs_shares_
(2) Create 2 empty volumes from cinder
(3) Create a nova instance and attach above 2 volumes
(4) Then, try to detach one of them.
You will get the error in nova-compute.log “Couldn't unmount the NFS share <your NFS mount path on nova-compute>”
Proposed Fix
-------
I’m not sure about any other OSs who outputs the ‘target is busy’ in the umount error code.
Therefore, first fix comes to my mind is fix the ‘if’ statement to:
Before fix;
if 'target is busy' in exc.message:
After fix;
if 'device is busy' in exc.message:
Changed in nova: | |
assignee: | nobody → Thang Pham (thang-pham) |
tags: | added: volumes |
Changed in nova: | |
status: | New → In Progress |
description: | updated |
tags: | added: icehouse-backport-potential |
Changed in nova: | |
milestone: | none → juno-rc1 |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | juno-rc1 → 2014.2 |
The feature above was put in by this commit: https:/ /github. com/openstack/ nova/commit/ dc716bd0ce77b56 f4aabe54d6633b7 f3bf9b0a5d. I agree with your proposed fix. Most of the time, I see "device is busy" and not "target is busy". This should be a quick fix.