_sync_power_state locks during deployment

Bug #1362194 reported by Kyle L. Henderson
This bug report is a duplicate of:  Bug #1363231: Periodic thread lockup. Edit Remove
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
High
melanie witt

Bug Description

In nova/compute/manager.py, _sync_power_states() uses a synchronized() lock to lock on the instance uuid before reconciling the power states of the instance between OpenStack and the compute driver. It purposely conflicts with the instance uuid lock on deployment (do_build_and_run_instance) . However, inside the lock (in _query_driver_power_state_and_sync) the first thing it does is to see if there is a task state and if so skips sync'ing the power state.

The result of synchronizing with the deployment is that all periodic tasks can be hung up behind a single deploy of an instance. There is no reason, that I can see, why the checking the task state can't be done outside the synchronized lock which would make it skip instances being deployed and would prevent periodic tasks from being hung behind a deployment.

So basically, the check for a task state should be moved to be outside the lock and that would avoid hanging periodic tasks and would result in the same semantics of the existing syncing of power states.

Tags: compute
Tracy Jones (tjones-i)
tags: added: compute
Changed in nova:
importance: Undecided → High
Revision history for this message
melanie witt (melwitt) wrote :

Here's the patch which added the lock to _sync_power_states:

https://review.openstack.org/#/c/103174/

In the first version I pushed, I left the skip logic outside the lock for that reason -- but I got convinced otherwise. I think you're right that it could be moved outside the lock as it was read from the database before the lock (object isn't refreshed) so I think maybe the lock isn't helping anything there.

Changed in nova:
assignee: nobody → melanie witt (melwitt)
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/117634

Changed in nova:
assignee: melanie witt (melwitt) → Kyle L. Henderson (kyleh)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/117639

Changed in nova:
assignee: Kyle L. Henderson (kyleh) → melanie witt (melwitt)
Revision history for this message
melanie witt (melwitt) wrote :

@Kyle So, I didn't realize you intended to work on this -- next time assign the bug to yourself so people know you're working on it and don't spend time.

Revision history for this message
Kyle L. Henderson (kyleh) wrote :

@Melanie, Yep, sounds like the right approach. I saw it wasn't assigned this morning and thought I'd look into it. When I started I didn't even have a development environment, so I didn't know if it would all come together. That's why I didn't assign it to myself. I didn't see you assigned it to yourself until I was mostly done so, I thought I'd put it up for review. I didn't realize it would pull the assignment from you. Sorry about that.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Kyle L. Henderson (<email address hidden>) on branch: master
Review: https://review.openstack.org/117634
Reason: Superseded by https://review.openstack.org/#/c/117639/.

Revision history for this message
melanie witt (melwitt) wrote :

@Kyle No worries -- was just trying to help avoid future collisions. Please feel free to assign a bug to yourself if you're looking into it. You can simply unassign yourself if it doesn't work out.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Another patch for a duplicate bug: https://review.openstack.org/#/c/117817/

Who will win?!

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by melanie witt (<email address hidden>) on branch: master
Review: https://review.openstack.org/117639
Reason: Abandoning this for https://review.openstack.org/#/c/117817

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Since https://review.openstack.org/#/c/117817/ is the one in progress, marking this as a dup of the one against which that review is in active review.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.