Need to audit when notifications are sent during live migration

[Matt Riedemann]

We do a pretty good job of testing that notifications are sent during certain operations, like live migration, but not so great a job at making sure that notifications are sent at expected times, like that start and end notifications actually happen at the start and end of a method (seems we should really use a decorator function for something like this for future proofing...).

This is a follow on from bug 1762876 where I thought about relying on the "live_migration._post.end" notification to be able to tell when a migration record should be 'completed' but that notification is sent *before* we change the status on the migration record:

https://github.com/openstack/nova/blob/fe976dcc559d059589a9ccf953a28e855abf50fb/nova/compute/manager.py#L6323

If you look at the beginning of the same method, the start notification is sent well after we've already started doing some work:

https://github.com/openstack/nova/blob/fe976dcc559d059589a9ccf953a28e855abf50fb/nova/compute/manager.py#L6261

So this bug is primarily meant to be an audit of at least the live migration flows where the methods are big and hairy so it's easy to see how over the years, the notifications got pushed into weird spots in those methods, and should be moved back to the appropriate start/end locations (or write a decorator to handle this).

[Balazs Gibizer (balazs-gibizer)]

I agree to fix this. I will also open a blueprint to do the audit on the rest of the start/end notification as it feels we have more such places.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/610739

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/610739
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7a2228142b8f8f015a554e2b25198ab37811a058
Submitter: Zuul
Branch: master

commit 7a2228142b8f8f015a554e2b25198ab37811a058
Author: Matt Riedemann <email address hidden>
Date: Mon Oct 15 18:16:40 2018 -0400

    Move live_migration.pre.start to the start of the method

    This simply moves the pre_live_migration start notification
    back to the start of the method before creating new volume
    attachments. While in here, the list of bdms are passed to
    the notify_about_instance_action method so if
    [notifications]/bdms_in_notifications is True we don't have
    to pull them out of the database again for each versioned
    notification.

    Change-Id: I372eddb9e356b02328f937917a856bc631691f53
    Partial-Bug: #1763051

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/612714

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/612773

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/612714
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ae2da62955e5c856fec71eebb4f3458fb4c7fb79
Submitter: Zuul
Branch: stable/rocky

commit ae2da62955e5c856fec71eebb4f3458fb4c7fb79
Author: Matt Riedemann <email address hidden>
Date: Mon Oct 15 18:16:40 2018 -0400

    Move live_migration.pre.start to the start of the method

    This simply moves the pre_live_migration start notification
    back to the start of the method before creating new volume
    attachments. While in here, the list of bdms are passed to
    the notify_about_instance_action method so if
    [notifications]/bdms_in_notifications is True we don't have
    to pull them out of the database again for each versioned
    notification.

    Change-Id: I372eddb9e356b02328f937917a856bc631691f53
    Partial-Bug: #1763051
    (cherry picked from commit 7a2228142b8f8f015a554e2b25198ab37811a058)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/612773
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6b065dad8015ea4fdacb0aedc763b38c5c8a9130
Submitter: Zuul
Branch: stable/queens

commit 6b065dad8015ea4fdacb0aedc763b38c5c8a9130
Author: Matt Riedemann <email address hidden>
Date: Mon Oct 15 18:16:40 2018 -0400

    Move live_migration.pre.start to the start of the method

    This simply moves the pre_live_migration start notification
    back to the start of the method before creating new volume
    attachments. While in here, the list of bdms are passed to
    the notify_about_instance_action method so if
    [notifications]/bdms_in_notifications is True we don't have
    to pull them out of the database again for each versioned
    notification.

    Conflicts in nova/compute/manager.py due to absence of comment from
    f0b9d5204f3.

    Change-Id: I372eddb9e356b02328f937917a856bc631691f53
    Partial-Bug: #1763051
    (cherry picked from commit 7a2228142b8f8f015a554e2b25198ab37811a058)
    (cherry picked from commit ae2da62955e5c856fec71eebb4f3458fb4c7fb79)