The _sync_power_states task should filter out instances.task_state != None up front

Bug #1572730 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Won't Fix
Medium
Sarafraj Singh

Bug Description

The _sync_power_states periodic task queries all instances on the compute host:

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6164

Then later it skips any that are in the middle of an operation:

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6269

We should avoid the roundtrip to the DB and RPC traffic to load up all of the instances on the compute host that are in the middle of a task and will just be skipped in code anyway and filter out the instance list by task_state in the initial DB query.

Matt Riedemann (mriedem)
tags: added: low-hanging-fruit
Changed in nova:
assignee: nobody → Chuck Carmack (chuckcarmack75)
Revision history for this message
Chuck Carmack (chuckcarmack75) wrote :

Matt, the periodic task compares the numbers of instances in the db vs. the actual number of VMs.

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6180

Should that warning be removed?

Revision history for this message
Matt Riedemann (mriedem) wrote :

I'll have to think about this. I guess the question is more about if we start filtering on the query to the database, then we might always get that warning, even though the numbers would otherwise be the same. That could be a problem yeah.

Revision history for this message
Matt Riedemann (mriedem) wrote :

There is handling later in the looping where the instance from the DB isn't on the hypervisor, and in that case we assume no power state:

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6223-L6227

Which actually makes this probably wrong:

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6282-L6294

So let's consider this scenario: we have 5 instances on the hypervisor and 5 in the database, that's all normal. However, we're currently taking snapshots of two of those instances. If we filter on task_state != None, then we'd get the warning in:

https://github.com/openstack/nova/blob/4ad414f3b1216393301ef268a64e61ca1a3d5be9/nova/compute/manager.py#L6180

Even though the instances are on the hypervisor.

I think the thing we need to warn about is if there are more instances for that host in the DB than what's in the hypervisor, so change:

if num_vm_instances != num_db_instances:

to

if num_vm_instances < num_db_instances:

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/309970

Changed in nova:
status: Triaged → In Progress
Changed in nova:
assignee: Chuck Carmack (chuckcarmack75) → Sarafraj Singh (sarafraj-singh)
Revision history for this message
Sarafraj Singh (sarafraj-singh) wrote :

This is no longer valid if you exclude all the things that are not running a task.
I think the cloud steady state will be no instances running a task, so I don't think we need to worry about the extra DB load of getting instances that are running a task. It should be tiny.

Changed in nova:
status: In Progress → Won't Fix
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/309970

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.