ImageCacheManager raises Permission denied error on nova compute in race condition
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Ankit Agrawal | ||
Liberty |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
ImageCacheManager raises Permission denied error on nova compute in race condition
While creating an instance snapshot nova calls guest.launch method from libvirt driver which changes the base file permissions and updates base file user from openstack to libvirt-qemu (in case of qcow2 image backend). In race condition when ImageCacheManager is trying to update last access time of this base file and guest.launch is called by instance snapshot just before updating the access time, ImageCacheManager raise Permission denied error in nova compute for os.utime().
Steps to reproduce:
1. Configure image_cache_
2. Add a sleep for 60 sec in _handle_base_image method of libvirt.imagecache just before calling os.utime().
3. Restart nova services.
4. Create an instance using image.
$ nova boot --image 5e1659aa-
5. Check that instance is in active state.
6. Go to the n-cpu screen and check imagecache manager logs at the point it waits to execute sleep statement added in step #2.
7. Send instance snapshot request when imagecache manger is waiting to execute sleep.
$ nova image-create 19c7900b-
8. instance snapshot request updates the base file owner to libvirt-qemu by calling guest.launch method from libvirt driver.
9. Now when imagecache manger comes out from sleep and executes os.utime it raise following Permission denied error in nova compute.
2015-07-01 01:51:46.794 ERROR nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
self._age_
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
2015-07-01 01:51:46.794 TRACE nova.openstack.
Expected result: guest.launch should not update the base file permissions and owner to libvirt-qemu. Base file owner should remain unchanged.
Actual result: Libvirt is updating the base file owner which causes permission issues in nova.
Changed in nova: | |
assignee: | nobody → Ankit Agrawal (ankitagrawal) |
Changed in nova: | |
importance: | Undecided → Medium |
I see this error intermittently in compute logs. When I was analyzing the source code to fix this issue I found that there is a provision in /etc/libvirt/ qemu.conf to configure user and group via the user=$USERNAME and group=$GROUPNAME parameters.
If I change libvirt user to same as nova user I do not see this Permission issue any more in imagecache manager for updating the base/backing file access time.
Is this a valid way to fix this issue by making changes in libvirt configuration? Please suggest.