device_path not saved into bdm.connection_info during pre_live_migration

[Lee Yarwood]

Description
===========

Various block based volume drivers attempt to save a device_path back into the stashed connection_info stored within Nova's block device mappings *after* connecting a volume to the underlying host. Thanks to the indirection caused by the various data structures used between the virt and compute layers this isn't actually saved into the underlying block device mapping database record during a typical attach flow until we get back into the compute and driver block device layer:

https://github.com/openstack/nova/blob/84b61790763f91e12eebb96d955e2f83abc00d56/nova/virt/block_device.py#L613-L619

However when an instance is being live migrated these volumes are connected as part of pre_live_migration on the destination and no attempt is made to save the updates made to the connection_info of the volume into the database. This isn't a massive problem as os-brick can for the most part lookup the device during future operations but it is obviously inefficient.

This was initially hit in bug #1936439 but that bug is now being used to track in a trivial DEBUG log change while this bug will track in the underlying fix for the above issue.

Steps to reproduce
==================
* Attach an iSCSI/FC/NVME etc volume to an instance
* Live migrate the instance
* Confirm that device_path isn't present in the connection_info stashed in the bdm

Expected result
===============
device_path is stashed in the connection_info of the bdm

Actual result
=============
device_path isn't stashed in the connection_info of the bdm

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

master

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

libvirt

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

LVM/iSCSI/FC/NVMe, any block based volume backends.

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

N/A

Logs & Configs
==============

See bug #1936439

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/804229

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/804230

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/804229
Committed: https://opendev.org/openstack/nova/commit/268b7169554f871d3813d78e79861934d086484d
Submitter: "Zuul (22348)"
Branch: master

commit 268b7169554f871d3813d78e79861934d086484d
Author: Lee Yarwood <email address hidden>
Date: Wed Aug 11 10:51:33 2021 +0100

    Add a regression test for bug 1939545

    Assert the behaviour of the libvirt driver during pre_live_migration
    with instances that have block based volumes attached.

    Bug #1939545 covering the case where the returned path from os-brick
    isn't being saved into the connection_info of the associated bdm in
    Nova.

    The cinder fixture is extended as part of this change to offer
    connection_info listing the iscsi volume type in order to correctly
    assert the behaviour of the associated libvirt volume driver.

    Related-Bug: #1939545
    Change-Id: I3f1073d70e4332737e33f3c9c00cf129d0ef76b4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/804230
Committed: https://opendev.org/openstack/nova/commit/962eda94d52321c3237da870c3d0455c6f0e851b
Submitter: "Zuul (22348)"
Branch: master

commit 962eda94d52321c3237da870c3d0455c6f0e851b
Author: Lee Yarwood <email address hidden>
Date: Wed Aug 11 12:04:11 2021 +0100

    compute: Ensure updates to bdms during pre_live_migration are saved

    When connecting volumes to the underlying host the virt drivers can
    attempt to stash additional metadata returned from os-brick into the
    connection_info associated with a bdm. This pretty janky behaviour
    relies on someone later calling .save() against the underlying
    BlockDeviceMapping object to persist these changes into the database as
    happens in the driver block device layer during a standard volume
    attach.

    However during pre_live_migration no call was made to .save() resulting
    in the changes made to the connection_info being lost. This change
    simply introduces this call at the end of the pre_live_migration method
    on the destination via the driver bdm objects we provide in
    block_device_info.

    Closes-Bug: #1939545
    Change-Id: Iea8896682fc28e3a5cd25afa45238272bee745e1

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/843679

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/843680

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/843702

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/843948

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/843949

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/843951

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/843952

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/843959

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/843960

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Artom Lifshitz <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/843702

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/843679
Committed: https://opendev.org/openstack/nova/commit/9f7b81c4cfe9f6a41fb28ec3b9d20616817ceab5
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9f7b81c4cfe9f6a41fb28ec3b9d20616817ceab5
Author: Lee Yarwood <email address hidden>
Date: Wed Aug 11 10:51:33 2021 +0100

    Add a regression test for bug 1939545

    Assert the behaviour of the libvirt driver during pre_live_migration
    with instances that have block based volumes attached.

    Bug #1939545 covering the case where the returned path from os-brick
    isn't being saved into the connection_info of the associated bdm in
    Nova.

    The cinder fixture is extended as part of this change to offer
    connection_info listing the iscsi volume type in order to correctly
    assert the behaviour of the associated libvirt volume driver.

    Conflicts: nova/tests/fixtures/cinder.py did not exist, applied
    changes to CinderFixture class in nova/tests/fixtures.py. Also:
    * Switched to using uuidsentinel instead of uuids, as the import is
      done as the former in stable/wallaby.
    * Fixed the fakelibvirt import to match where it lives in
      stable/wallaby.
    * Replace use of OSBrickFixture() that didn't exist in stable/wallaby
      with the fake_os_brick_connector from the libvirt unit tests.
    * Fix fakelibvirt migrateToURI3 monkeypatch to use correct path for
      stable/wallaby.

    Related-Bug: #1939545
    Change-Id: I3f1073d70e4332737e33f3c9c00cf129d0ef76b4
    (cherry picked from commit 268b7169554f871d3813d78e79861934d086484d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/843680
Committed: https://opendev.org/openstack/nova/commit/9bccc5a04354b0490fbe7e0f0b98cf55d4b7c9c4
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9bccc5a04354b0490fbe7e0f0b98cf55d4b7c9c4
Author: Lee Yarwood <email address hidden>
Date: Wed Aug 11 12:04:11 2021 +0100

    compute: Ensure updates to bdms during pre_live_migration are saved

    When connecting volumes to the underlying host the virt drivers can
    attempt to stash additional metadata returned from os-brick into the
    connection_info associated with a bdm. This pretty janky behaviour
    relies on someone later calling .save() against the underlying
    BlockDeviceMapping object to persist these changes into the database as
    happens in the driver block device layer during a standard volume
    attach.

    However during pre_live_migration no call was made to .save() resulting
    in the changes made to the connection_info being lost. This change
    simply introduces this call at the end of the pre_live_migration method
    on the destination via the driver bdm objects we provide in
    block_device_info.

    Closes-Bug: #1939545
    Change-Id: Iea8896682fc28e3a5cd25afa45238272bee745e1
    (cherry picked from commit 962eda94d52321c3237da870c3d0455c6f0e851b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 23.2.2 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/843960
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/843959
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/843952
Reason: stable/ussuri branch of openstack/nova transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/843951
Reason: stable/ussuri branch of openstack/nova transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/843949
Reason: stable/victoria branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/843948
Reason: stable/victoria branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.