Periodic thread lockup

[Brian Elliott]

The instance locking introduced in
cc5388bbe81aba635fb757e202d860aeed98f3e8 keeps the power state sane
between stop and the periodic task power sync. However, locking on
an instance in the periodic task thread can potentially lock
that thread for a long time.

Example:
1) User boots an instance. The instance gets locked by uuid.
2) Driver spawn begins and the image starts downloading from glance.
3) During spawn, periodic tasks run. Sync power states tries to grab
the same instance lock by uuid.
4) Periodic task thread hangs until the driver spawn completes in
another greenthread.

This scenario results in nova-compute appearing unresponsive for
a long time.

[Brian Elliott (belliott)]

https://review.openstack.org/#/c/117817/ (not sure why review wasn't linking up)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/117817
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=baabab45e0ae0e9e35872cae77eb04bdb5ee0545
Submitter: Jenkins
Branch: master

commit baabab45e0ae0e9e35872cae77eb04bdb5ee0545
Author: Brian D. Elliott <email address hidden>
Date: Fri Aug 29 15:12:07 2014 +0000

    Process power state syncs asynchronously

    The instance locking introduced in
    cc5388bbe81aba635fb757e202d860aeed98f3e8 keeps the power state sane
    between stop and the periodic task power sync. However, locking on
    an instance in the periodic task thread can potentially lock
    that thread for a long time.

    Fix this condition by executing power syncs in a background GreenPool.
    Limit the syncs to 1 at a time per instance.

    Closes-Bug: 1363231

    Change-Id: Idbc00ddfcb7a6816433863ed1686ad8f7477d99b