The _sync_power_states task should filter out instances.task_state != None up front

Bug #1572730 reported by Matt Riedemann on 2016-04-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Sarafraj Singh

Bug Description

The _sync_power_states periodic task queries all instances on the compute host:

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6164

Then later it skips any that are in the middle of an operation:

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6269

We should avoid the roundtrip to the DB and RPC traffic to load up all of the instances on the compute host that are in the middle of a task and will just be skipped in code anyway and filter out the instance list by task_state in the initial DB query.

Matt Riedemann (mriedem) on 2016-04-20
tags: added: low-hanging-fruit
Changed in nova:
assignee: nobody → Chuck Carmack (chuckcarmack75)
Chuck Carmack (chuckcarmack75) wrote :

Matt, the periodic task compares the numbers of instances in the db vs. the actual number of VMs.

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6180

Should that warning be removed?

Matt Riedemann (mriedem) wrote :

I'll have to think about this. I guess the question is more about if we start filtering on the query to the database, then we might always get that warning, even though the numbers would otherwise be the same. That could be a problem yeah.

Matt Riedemann (mriedem) wrote :

There is handling later in the looping where the instance from the DB isn't on the hypervisor, and in that case we assume no power state:

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6223-L6227

Which actually makes this probably wrong:

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6282-L6294

So let's consider this scenario: we have 5 instances on the hypervisor and 5 in the database, that's all normal. However, we're currently taking snapshots of two of those instances. If we filter on task_state != None, then we'd get the warning in:

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6180

Even though the instances are on the hypervisor.

I think the thing we need to warn about is if there are more instances for that host in the DB than what's in the hypervisor, so change:

if num_vm_instances != num_db_instances:

to

if num_vm_instances < num_db_instances:

Fix proposed to branch: master
Review: https://review.openstack.org/309970

Changed in nova:
status: Triaged → In Progress
Changed in nova:
assignee: Chuck Carmack (chuckcarmack75) → Sarafraj Singh (sarafraj-singh)
Sarafraj Singh (sarafraj-singh) wrote :

This is no longer valid if you exclude all the things that are not running a task.
I think the cloud steady state will be no instances running a task, so I don't think we need to worry about the extra DB load of getting instances that are running a task. It should be tiny.

Changed in nova:
status: In Progress → Won't Fix

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/309970

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers