I see that for Grizzly update_capabilities method for host manager doesn't reset self.num_instances_by_project if the last VM for some project is deleted on the host.
Is reproduced when working from several tenants simultaneously. For some reason in some cases update messages miss 0 counter for one of tenant;
If it is really the bug, fix can be like
--- a/nova/scheduler/host_manager.py
+++ b/nova/scheduler/host_manager.py
@@ -171,9 +171,12 @@ class HostState(object):
# Track number of instances by project_id
project_id_keys = [k for k in statmap.keys() if
k.startswith("num_proj_")]
- for key in project_id_keys:
- project_id = key[9:]
- self.num_instances_by_project[project_id] = int(statmap[key])
+ project2instance_count = dict([(key[9:], int(statmap[key]))
+ for key in project_id_keys])
+ for key in self.num_instances_by_project:
+ if key not in project2instance_count:
+ self.num_instances_by_project[key] = 0
+ self.num_instances_by_project.update(project2instance_count)
# Track number of instances in certain vm_states
vm_state_keys = [k for k in statmap.keys() if k.startswith("num_vm_")]
Sumanth Nagadavalli are you still working on this?