cannot live-migrate VMs booted from image across charm apps if using ceph libvirt-image-backend=rbd

[Rodrigo Barbieri]

[jammy-yoga] rev 681

When using ceph-ephemeral (libvirt-image-backend=rbd), and VMs booted from image, live migration across different charm apps (such as from nova-compute-kvm to nova-compute-kvm2) fails with the following error:

2023-07-24 14:44:40.004 66943 ERROR nova.virt.libvirt.driver [-] [instance: 392e13f3-1a22-42b7-a2b4-498c469cf159] Live Migration failure: internal error: process exited while connecting to monitor: 2023-07-24T14:44:39.741332Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"nova","image":"392e13f3-1a22-42b7-a2b4-498c469cf159_disk","server":[{"host":"10.5.3.246","port":"6789"}],"user":"nova-compute-qua","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-auth-secret0","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error connecting: Permission denied: libvirt.libvirtError: internal error: process exited while connecting to monitor: 2023-07-24T14:44:39.741332Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"nova","image":"392e13f3-1a22-42b7-a2b4-498c469cf159_disk","server":[{"host":"10.5.3.246","port":"6789"}],"user":"nova-compute-qua","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-auth-secret0","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error connecting: Permission denied
2023-07-24 14:44:40.157 66943 ERROR nova.virt.libvirt.driver [-] [instance: 392e13f3-1a22-42b7-a2b4-498c469cf159] Migration operation has aborted

The reason for the above error is that the ceph keys are different for each nova-compute charm app, so the src/dst do not have each other's keys to accomplish the migration successfully.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

https://review.opendev.org/c/openstack/charm-nova-compute/+/889642

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/889642
Committed: https://opendev.org/openstack/charm-nova-compute/commit/c3c2cf0349c086dad7f23b180c3ee9ea0f865e8f
Submitter: "Zuul (22348)"
Branch: master

commit c3c2cf0349c086dad7f23b180c3ee9ea0f865e8f
Author: Rodrigo Barbieri <email address hidden>
Date: Tue Jul 25 14:02:30 2023 -0300

    Fix migration across apps when using VMs created from image

    When using the config libvirt-image-backend=rbd, VMs
    created from image have their disk data stored in ceph
    instead of the compute node itself.

    When performing live-migrations, both nodes need to
    access the same ceph credentials to access the VM's
    disk in ceph, but this is currently not possible
    if the nodes involved pertain to different
    nova-compute charm apps.

    This patch changes app name sent to ceph to 'nova-compute',
    allowing all nova-compute apps to use the same ceph auth.

    This patch also includes the charmhelpers sync
    from PR: #840

    Closes-bug: #2028559
    Change-Id: I7222661017655fd7225db0c677f1a8f5ebb7984d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/893194

[Rodrigo Barbieri (rodrigo-barbieri2010)]

charm-guide update: https://review.opendev.org/c/openstack/charm-guide/+/893228

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-guide/+/893228
Committed: https://opendev.org/openstack/charm-guide/commit/e4539bb2053893d48a02fe767fffc6425a2b1d28
Submitter: "Zuul (22348)"
Branch: master

commit e4539bb2053893d48a02fe767fffc6425a2b1d28
Author: Rodrigo Barbieri <email address hidden>
Date: Wed Aug 30 13:46:12 2023 -0300

    Release note for unify nova-compute Ceph credentials

    Closes-bug: #2028559
    Change-Id: I32dcda326733563e18d9960564988153bea6bacb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/894875

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/894876

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Edward Hope-Morley <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/894875
Reason: master branch has been reverted so abandoning these backports

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Edward Hope-Morley <email address hidden>" on branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/894876
Reason: master branch has been reverted so abandoning these backports

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Edward Hope-Morley <email address hidden>" on branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/893194
Reason: master branch has been reverted so abandoning these backports

[Peter Matulis (petermatulis)]

Removing bug task for charm-guide due to code revert.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/897549
Committed: https://opendev.org/openstack/charm-nova-compute/commit/650f3a5d511690ec27648b30f3b24532378a33a1
Submitter: "Zuul (22348)"
Branch: master

commit 650f3a5d511690ec27648b30f3b24532378a33a1
Author: Rodrigo Barbieri <email address hidden>
Date: Fri Oct 6 10:24:48 2023 -0300

    [v2] Fix migration across nova-compute apps using ceph

    This change reworks previous changes [1] and [2] that had
    been respectively reverted and abandoned.

    When using the config libvirt-image-backend=rbd, VMs
    created from image have their disk data stored in ceph
    instead of the compute node itself.

    When performing live-migrations, both nodes need to
    access the same ceph credentials to access the VM's
    disk in ceph, but this is currently not possible
    if the nodes involved pertain to different
    nova-compute charm apps.

    This patch changes app name sent to ceph to
    'nova-compute-ceph-auth-c91ce26f', a unique name common to
    all nova-compute apps, allowing all nova-compute apps to
    use the same ceph auth.

    This change also ensures newly deployed nodes install
    the old credentials first on ceph-joined hook,
    and then supercedes it with the new credentials
    on ceph-changed hook, therefore also retaining
    the old credentials.

    This patch also includes the charmhelpers sync
    from PR: #840

    [1] https://review.opendev.org/889642
    [2] https://review.opendev.org/896155

    Closes-bug: #2028559
    Related-bug: #2037003

    Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1149

    Change-Id: I1ae12d787a1f8e7761ca06b5a80049c1c62e9e90

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/905312

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/905312
Committed: https://opendev.org/openstack/charm-nova-compute/commit/28233f9a943f9c6c67d4934cf693869c9c03b1e4
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 28233f9a943f9c6c67d4934cf693869c9c03b1e4
Author: Rodrigo Barbieri <email address hidden>
Date: Fri Oct 6 10:24:48 2023 -0300

    [v2] Fix migration across nova-compute apps using ceph

    This change reworks previous changes [1] and [2] that had
    been respectively reverted and abandoned.

    When using the config libvirt-image-backend=rbd, VMs
    created from image have their disk data stored in ceph
    instead of the compute node itself.

    When performing live-migrations, both nodes need to
    access the same ceph credentials to access the VM's
    disk in ceph, but this is currently not possible
    if the nodes involved pertain to different
    nova-compute charm apps.

    This patch changes app name sent to ceph to
    'nova-compute-ceph-auth-c91ce26f', a unique name common to
    all nova-compute apps, allowing all nova-compute apps to
    use the same ceph auth.

    This change also ensures newly deployed nodes install
    the old credentials first on ceph-joined hook,
    and then supercedes it with the new credentials
    on ceph-changed hook, therefore also retaining
    the old credentials.

    [1] https://review.opendev.org/889642
    [2] https://review.opendev.org/896155

    Closes-bug: #2028559
    Related-bug: #2037003

    Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1183

    Change-Id: I1ae12d787a1f8e7761ca06b5a80049c1c62e9e90
    (cherry picked from commit 650f3a5d511690ec27648b30f3b24532378a33a1)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/906298

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/906298
Committed: https://opendev.org/openstack/charm-nova-compute/commit/ecaf8010fc6ee2e28e6a0f05cb2ae0a8ce25b5e6
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit ecaf8010fc6ee2e28e6a0f05cb2ae0a8ce25b5e6
Author: Rodrigo Barbieri <email address hidden>
Date: Fri Oct 6 10:24:48 2023 -0300

    [v2] Fix migration across nova-compute apps using ceph

    This change reworks previous changes [1] and [2] that had
    been respectively reverted and abandoned.

    When using the config libvirt-image-backend=rbd, VMs
    created from image have their disk data stored in ceph
    instead of the compute node itself.

    When performing live-migrations, both nodes need to
    access the same ceph credentials to access the VM's
    disk in ceph, but this is currently not possible
    if the nodes involved pertain to different
    nova-compute charm apps.

    This patch changes app name sent to ceph to
    'nova-compute-ceph-auth-c91ce26f', a unique name common to
    all nova-compute apps, allowing all nova-compute apps to
    use the same ceph auth.

    This change also ensures newly deployed nodes install
    the old credentials first on ceph-joined hook,
    and then supercedes it with the new credentials
    on ceph-changed hook, therefore also retaining
    the old credentials.

    [1] https://review.opendev.org/889642
    [2] https://review.opendev.org/896155

    Closes-bug: #2028559
    Related-bug: #2037003

    Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1183

    Change-Id: I1ae12d787a1f8e7761ca06b5a80049c1c62e9e90
    (cherry picked from commit 650f3a5d511690ec27648b30f3b24532378a33a1)
    (cherry picked from commit 28233f9a943f9c6c67d4934cf693869c9c03b1e4)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/908187

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/908187
Committed: https://opendev.org/openstack/charm-nova-compute/commit/8c3549d828c5ba0d6306b76494fdbd1abd9d8471
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 8c3549d828c5ba0d6306b76494fdbd1abd9d8471
Author: Rodrigo Barbieri <email address hidden>
Date: Fri Oct 6 10:24:48 2023 -0300

    [v2] Fix migration across nova-compute apps using ceph

    This change reworks previous changes [1] and [2] that had
    been respectively reverted and abandoned.

    When using the config libvirt-image-backend=rbd, VMs
    created from image have their disk data stored in ceph
    instead of the compute node itself.

    When performing live-migrations, both nodes need to
    access the same ceph credentials to access the VM's
    disk in ceph, but this is currently not possible
    if the nodes involved pertain to different
    nova-compute charm apps.

    This patch changes app name sent to ceph to
    'nova-compute-ceph-auth-c91ce26f', a unique name common to
    all nova-compute apps, allowing all nova-compute apps to
    use the same ceph auth.

    This change also ensures newly deployed nodes install
    the old credentials first on ceph-joined hook,
    and then supercedes it with the new credentials
    on ceph-changed hook, therefore also retaining
    the old credentials.

    [1] https://review.opendev.org/889642
    [2] https://review.opendev.org/896155

    Closes-bug: #2028559
    Related-bug: #2037003

    Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1183

    Change-Id: I1ae12d787a1f8e7761ca06b5a80049c1c62e9e90
    (cherry picked from commit 650f3a5d511690ec27648b30f3b24532378a33a1)
    (cherry picked from commit 28233f9a943f9c6c67d4934cf693869c9c03b1e4)
    (cherry picked from commit ecaf8010fc6ee2e28e6a0f05cb2ae0a8ce25b5e6)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/911153

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/911153
Committed: https://opendev.org/openstack/charm-nova-compute/commit/810d4b0fbac967d47ff793cc02f87da37cb5b578
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 810d4b0fbac967d47ff793cc02f87da37cb5b578
Author: Rodrigo Barbieri <email address hidden>
Date: Fri Oct 6 10:24:48 2023 -0300

    [v2] Fix migration across nova-compute apps using ceph

    This change reworks previous changes [1] and [2] that had
    been respectively reverted and abandoned.

    When using the config libvirt-image-backend=rbd, VMs
    created from image have their disk data stored in ceph
    instead of the compute node itself.

    When performing live-migrations, both nodes need to
    access the same ceph credentials to access the VM's
    disk in ceph, but this is currently not possible
    if the nodes involved pertain to different
    nova-compute charm apps.

    This patch changes app name sent to ceph to
    'nova-compute-ceph-auth-c91ce26f', a unique name common to
    all nova-compute apps, allowing all nova-compute apps to
    use the same ceph auth.

    This change also ensures newly deployed nodes install
    the old credentials first on ceph-joined hook,
    and then supercedes it with the new credentials
    on ceph-changed hook, therefore also retaining
    the old credentials.

    [1] https://review.opendev.org/889642
    [2] https://review.opendev.org/896155

    Closes-bug: #2028559
    Related-bug: #2037003

    Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1190

    Change-Id: I1ae12d787a1f8e7761ca06b5a80049c1c62e9e90
    (cherry picked from commit 650f3a5d511690ec27648b30f3b24532378a33a1)
    (cherry picked from commit 28233f9a943f9c6c67d4934cf693869c9c03b1e4)
    (cherry picked from commit ecaf8010fc6ee2e28e6a0f05cb2ae0a8ce25b5e6)
    (cherry picked from commit 8c3549d828c5ba0d6306b76494fdbd1abd9d8471)