When some instance aren't deleted correctly and libvirt still keep the domain for the instance, the resource tracker will failed to update available resource

Bug #1553144 reported by Alex Xu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Triaged
High
Alex Xu
Liberty
New
Undecided
Unassigned

Bug Description

When instance was deleted in the db, but it is still at compute node, the resource tracker will fail to update available resource.

2016-03-04 10:58:28.143 ERROR nova.compute.manager [req-d2f1c99b-0e81-4b6d-9361-a40bd2218141 None None] Error updating resources for node vm6.

2016-03-04 10:58:28.143 TRACE nova.compute.manager Traceback (most recent call last):
2016-03-04 10:58:28.143 TRACE nova.compute.manager File "/opt/stack/nova/nova/compute/manager.py", line 6379, in update_available_resource
2016-03-04 10:58:28.143 TRACE nova.compute.manager rt.update_available_resource(context)
2016-03-04 10:58:28.143 TRACE nova.compute.manager File "/opt/stack/nova/nova/compute/resource_tracker.py", line 491, in update_available_resource
2016-03-04 10:58:28.143 TRACE nova.compute.manager resources = self.driver.get_available_resource(self.nodename)
2016-03-04 10:58:28.143 TRACE nova.compute.manager File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5414, in get_available_resource
2016-03-04 10:58:28.143 TRACE nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()
2016-03-04 10:58:28.143 TRACE nova.compute.manager File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7047, in _get_disk_over_committed_size_total
2016-03-04 10:58:28.143 TRACE nova.compute.manager local_instances[guest.uuid], bdms[guest.uuid])
2016-03-04 10:58:28.143 TRACE nova.compute.manager KeyError: '49505c88-b38a-4100-ab56-97958b48b533'
2016-03-04 10:58:28.143 TRACE nova.compute.manager

The available resource won't get update until periodic_task '_cleanup_running_deleted_instances' if running_deleted_instance_action is 'reap'

Alex Xu (xuhj)
Changed in nova:
assignee: nobody → Alex Xu (xuhj)
Revision history for this message
Matt Riedemann (mriedem) wrote :

Actually I think this is already fixed:

https://review.openstack.org/#/c/221162/

Now it checks:

if guest.uuid in local_instances:

tags: added: compute libvirt
Changed in nova:
status: New → Triaged
importance: Undecided → High
tags: added: liberty-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.