Need to delay instance state events

[Drew Thorstensen]

The libvirt code understands that multiple events can be sent for a VM when it is transitioning through its lifecycle (and that these events can affect tasks). For instance, a reboot will go through a power off and a power on quickly. Two events get emitted from the hypervisor.

This can affect the VM lifecycle code and can generally upset a couple operations.

To deal with this, libvirt has introduced a 'delayed emit event' concept.

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L267-L294

It only delays the stopped events (or in PowerVM the 'not activated' event).

We need to introduce a similar counter in the PowerVM driver. Certain tests in tempest seem to be affected by the lack of this, and certain timing issues can arise in the CI due to us not having it. For instance the error here:

http://184.172.12.213/10/361410/3/check/nova-powervm-pvm-dsvm-tempest-full/9dbc055/powervm_os_ci.html.gz

I think the PowerVM driver needs to utilize similar logic to delay the stopped events.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/391581

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391048
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=3a78c42ff0040df5a60531d536e8f1689659a696
Submitter: Jenkins
Branch: master

commit 3a78c42ff0040df5a60531d536e8f1689659a696
Author: Drew Thorstensen <email address hidden>
Date: Thu Oct 27 10:14:24 2016 +0200

    Refactor events into own class

    The eventing model needs to be updated to handle delayed events (and in
    future needs more logic to handle resumes on target during LPM). There
    is going to be additional complexity in events. This change set moves
    it out into its own class so as not to keep growing the already long
    driver.py file.

    Change-Id: I1eadc930d9a55a5fb92f7cd6ffdd37965bb187b6
    Partial-Bug: 1617746

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391581
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=24dbf1b7e61649b5001706190b054dcd07d8d4dd
Submitter: Jenkins
Branch: stable/newton

commit 24dbf1b7e61649b5001706190b054dcd07d8d4dd
Author: Drew Thorstensen <email address hidden>
Date: Thu Oct 27 10:14:24 2016 +0200

    Refactor events into own class

    The eventing model needs to be updated to handle delayed events (and in
    future needs more logic to handle resumes on target during LPM). There
    is going to be additional complexity in events. This change set moves
    it out into its own class so as not to keep growing the already long
    driver.py file.

    Change-Id: I1eadc930d9a55a5fb92f7cd6ffdd37965bb187b6
    Partial-Bug: 1617746

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391164
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=f98f022a28e66b003fb256d2cc328c5696d66d7a
Submitter: Jenkins
Branch: master

commit f98f022a28e66b003fb256d2cc328c5696d66d7a
Author: Drew Thorstensen <email address hidden>
Date: Thu Oct 27 13:36:38 2016 +0200

    Add delay queue for events

    Certain events can come in a series. For instance, if you run a VM
    reboot, the hypervisor will emit a SHUTTING_DOWN, NOT_ACTIVATED and
    STARTING state.

    If these states are just fed to nova raw, due to the async nature it
    could come in out of order. So the ACTIVE event could come in before
    the NOT_ACTIVATED (even though that doesn't reflect how it came in to
    the host).

    This change follows the libvirt event mechanism and builds a delay
    queue. For certain powering off events, they will be delayed before
    they are emitted up to nova. If a corresponding starting event comes in
    afterwards, it will void out the shutting down event.

    Change-Id: Iefd33b6bf9e3621a9f716488bf43b02c7d3cb0c4
    Closes-Bug: 1617746