Event execution failures for back to back leases

[Pierre Riteau]

If two leases have at least one compute host in common, and the second lease starts when the first lease ends, there is the possibility of a race. The Blazar manager can first run the start_lease event of the second lease. This would fail since the end_lease event of the first lease wouldn't have been run yet: the compute host(s) in common would still be in aggregate(s) associated with the first lease, instead of being in the freepool.

[Masahito Muroi (muroi-masahito)]

I imagine one of the purposes of cleaning time BP is resolving this issue. The first usecase of the new feature comes from Ironic usecase, but basically it fixes this issue, too.

https://blueprints.launchpad.net/blazar/+spec/cleaning-time-allowance

[Masahito Muroi (muroi-masahito)]

Oops, I mean we've already fixed the bug and don't need to care it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/589899

[Pierre Riteau (priteau)]

Cleaning time would help, but it's not enabled by default. Please check my patch instead: it fixes other issues, such as running before_end_lease after start_lease has completed.

[Pierre Riteau (priteau)]

I will push an updated patch.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/blazar/+/589899
Committed: https://opendev.org/openstack/blazar/commit/c92edb8a177de51862ad2a4f9cbac2c50d31ef84
Submitter: "Zuul (22348)"
Branch: master

commit c92edb8a177de51862ad2a4f9cbac2c50d31ef84
Author: Pierre Riteau <email address hidden>
Date: Wed Aug 8 12:46:28 2018 +0200

    Prevent conflicting events from running concurrently

    If two leases have compute hosts in common, and the second lease starts
    exactly when the first lease ends, there is the possibility of a race.
    The Blazar manager can first run the start_lease event of the second
    lease. This event would fail since the end_lease event of the first
    lease would still be UNDONE, and the compute hosts in common would still
    be in the aggregate associated with the first lease, instead of being in
    the freepool.

    This patch changes event execution code so that events are executed
    concurrently if possible, with the following constraints:

    - events are executed strictly in order, i.e. events are started only
      after all previous events have completed
    - when events are at the same time, we first execute before_end_lease
      events (unless there is a start_lease at the same time), then
      end_lease events, followed by start_lease events, ensuring the bug
      described above does not happen. Finally, we run any before_end_lease
      which had a corresponding start_lease event at the same time.

    It also has the side effect of providing better stack traces for event
    execution failures, since we call wait() on all GreenThread objects.

    Co-Authored-By: Jason Anderson <email address hidden>
    Change-Id: Ie2339db18e8baee379fbea082f1238ec44fca6b1
    Closes-Bug: #1785841

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/blazar/+/831509

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/blazar 9.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/blazar/+/831509
Committed: https://opendev.org/openstack/blazar/commit/c3b851937ebf110184ef46d7bc2ad42e163f92b1
Submitter: "Zuul (22348)"
Branch: stable/xena

commit c3b851937ebf110184ef46d7bc2ad42e163f92b1
Author: Pierre Riteau <email address hidden>
Date: Wed Aug 8 12:46:28 2018 +0200

    Prevent conflicting events from running concurrently

    If two leases have compute hosts in common, and the second lease starts
    exactly when the first lease ends, there is the possibility of a race.
    The Blazar manager can first run the start_lease event of the second
    lease. This event would fail since the end_lease event of the first
    lease would still be UNDONE, and the compute hosts in common would still
    be in the aggregate associated with the first lease, instead of being in
    the freepool.

    This patch changes event execution code so that events are executed
    concurrently if possible, with the following constraints:

    - events are executed strictly in order, i.e. events are started only
      after all previous events have completed
    - when events are at the same time, we first execute before_end_lease
      events (unless there is a start_lease at the same time), then
      end_lease events, followed by start_lease events, ensuring the bug
      described above does not happen. Finally, we run any before_end_lease
      which had a corresponding start_lease event at the same time.

    It also has the side effect of providing better stack traces for event
    execution failures, since we call wait() on all GreenThread objects.

    Co-Authored-By: Jason Anderson <email address hidden>
    Change-Id: Ie2339db18e8baee379fbea082f1238ec44fca6b1
    Closes-Bug: #1785841
    (cherry picked from commit c92edb8a177de51862ad2a4f9cbac2c50d31ef84)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/blazar 8.0.1 release.