Resource could not be updated when deleting instances

[Eric Zhou]

In my openstack compute node. a kvm-based vm exists and it has a iso image configured in the cdrom. The iso image path is /root/RHEL6.5-20131111.0-Server-x86_64-DVD1.iso.
When deleting the nova instances, the following error happend in the /var/log/nova/compute.log. And this error caused that the resources could not be updated and the resources occupied by the deleted instances could not be released. This error also happend when restaring the nova-compute service and so that nova-compute could not restarted successfully.

We should catch this exception for some existing libvirt instances otherwise all resources for nova instances could not be updated correctly.

2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task task(self, context)
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5445, in update_available_resource
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task rt.update_available_resource(context)
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task return f(*args, **kwargs)
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py", line 293, in update_available_resource
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task resources = self.driver.get_available_resource(self.nodename)
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4142, in get_available_resource
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task stats = self.get_host_stats(refresh=True)
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4817, in get_host_stats
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task return self.host_state.get_host_stats(refresh=refresh)
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5218, in get_host_stats
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task self.update_status()
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5261, in update_status
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task data['disk_available_least'] = _get_disk_available_least()
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5234, in _get_disk_available_least
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task disk_over_committed = (self.driver.
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4787, in get_disk_over_committed_size_total
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task self.get_instance_disk_info(i_name))
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4759, in get_instance_disk_info
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task dk_size = int(os.path.getsize(path))
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/genericpath.py", line 49, in getsize
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task return os.stat(filename).st_size
2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task OSError: [Errno 13] Permission denied: '/root/RHEL6.5-20131111.0-Server-x86_64-DVD1.iso'

[s iwata (s-iwata)]

2014-05-20 09:34:26.802 4061 TRACE nova.openstack.common.periodic_task OSError: [Errno 13] Permission denied: '/root/RHEL6.5-20131111.0-Server-x86_64-DVD1.iso'

It looks that deleted instance still occupied the iso file.
(But the log say ' permission denied ')
Please tell me the way to reproduce it. I'll try to reproduce it in my environment.
So, just to make sure, is 'the resources' iso image?

I think following
1. Boot the instance with iso image
2. Delete the booted server
3. Boot the instance again with same iso image
Does it correct?

[Eric Zhou (zyouzhou)]

re s-iwata, it is not the fact that the deleted instance occupied the iso file. Actually you can reproduce it with the following steps:

1. Create an virtual machine using virsh tool and install the operating system using an iso image file for the virtual machine. It means that this virtual machine is not managed by nova.

2. Install openstack on the same host.

3. Boot multiple instances using nova boot.

4. Try to delete one instance that managed by the nova and booted in step 3.

5. you will see the error reported in the description. Nova could not access the disk path occupied by the virtual machine not managed by nova, which is created in the step 1.

[s iwata (s-iwata)]

Hi Eric

Thank you for reply!
Do i have to use an iso image file in the step3 ?

Thank you.
Saki

[s iwata (s-iwata)]

1. queme need to be started by root user to create VM in libvirt using an iso image file in /root dir.
In this case, the VM is owned by root user.

2. In Nova, checking physical file of VM and attached iso image(periodic task), to check disk space.
Normally Nova is started by nova user.
Therefore, nova can't check the VM owned by root.
I think, Permission Error caused by those reason.

In my local environment, i could see Permission Error without deleting VM.

Given these facts, Permission Error caused by that two VMs(created by virsh tool and nova boot) existing at the same time.
Is my understanding correct?

Thanks.

[Eric Zhou (zyouzhou)]

Yes, correct.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95105
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ebecfd496ec8aaef07a401d3c90e2e5da406d16f
Submitter: Jenkins
Branch: master

commit ebecfd496ec8aaef07a401d3c90e2e5da406d16f
Author: ericzhou <email address hidden>
Date: Fri May 23 16:19:35 2014 +0800

    Catch permission denied exception when update host

    When an existing non-openstack instance in the compute node contains
    disk file which can not be accessed by nova non-root user, resources can
    not be updated successfully when periodic task is running. Also the
    nova-compute service could not be restared again. We should catch such
    exception.

    Change-Id: I75195417e123f135252470cf1322843830556b0a
    Closes-Bug: 1322467

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/102862

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthew Treinish (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/102862