libvirt live migration sometimes fails with "libvirt.libvirtError: internal error: migration was active, but no RAM info was set"

[melanie witt]

We have seen this downstream where live migration randomly fails with the following error [1]:

  libvirt.libvirtError: internal error: migration was active, but no RAM info was set

Discussion on [1] gravitated toward a possible race condition issue in qemu around the query-migrate command [2]. The query-migrate command is used (indirectly) by the libvirt driver during monitoring of live migrations [3][4][5].

While searching for info about this error, I found a thread on libvir-list from the past [6] where someone else encountered the same error and for them it happened if they called query-migrate *after* a live migration had completed.

Based on this, it seemed possible that our live migration monitoring thread sometimes races and calls jobStats() after the migration has completed, resulting in this error being raised and the migration being considered failed when it was actually complete.

A patch has since been proposed and committed [7] to address the possible issue.

Meanwhile, on our side in nova, we can mitigate this problematic behavior by catching the specific error from libvirt and ignoring it so that a live migration in this situation will be considered completed by the libvirt driver.

Doing this would improve the experience for users that are hitting this error and getting erroneous live migration failures.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2074205
[2] https://qemu.readthedocs.io/en/latest/interop/qemu-qmp-ref.html#qapidoc-1848
[3] https://github.com/openstack/nova/blob/bcb96f362ab12e297f125daa5189fb66345b4976/nova/virt/libvirt/driver.py#L10123
[4] https://github.com/openstack/nova/blob/bcb96f362ab12e297f125daa5189fb66345b4976/nova/virt/libvirt/guest.py#L655
[5] https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainGetJobStats
[6] https://listman.redhat.com/archives/libvir-list/2021-January/213631.html
[7] https://github.com/qemu/qemu/commit/552de79bfdd5e9e53847eb3c6d6e4cd898a4370e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/852002

[Brett Milford (brettmilford)]

Hi Melanie,

We have an large openstack ussuri environment that encounters this issue with some frequency.
I've proposed the fix above, looking forward to your feedback.

[Tyler Stachecki (tstachecki)]

Hello,

I am able to successfully reproduce this issue in a development cluster. It does take a few hours to trigger, so not something I can easily provide steps for... but happy to test candidate fixes.

Taking the opendev PR and running it through the gauntlet now.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/859358

[OpenStack Infra (hudson-openstack)]

Change abandoned by "melanie witt <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/842687
Reason: This patch does not work properly, so we will proceed with a different patch: https://review.opendev.org/c/openstack/nova/+/852002

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/852002
Committed: https://opendev.org/openstack/nova/commit/9fea934c71d3c2fa7fdd80c67d94e18466c5cf9a
Submitter: "Zuul (22348)"
Branch: master

commit 9fea934c71d3c2fa7fdd80c67d94e18466c5cf9a
Author: Brett Milford <email address hidden>
Date: Thu Aug 4 16:52:33 2022 +1000

    Handle "no RAM info was set" migration case

    This handles the case where the live migration monitoring thread may
    race and call jobStats() after the migration has completed resulting in
    the following error:

        libvirt.libvirtError: internal error: migration was active, but no
        RAM info was set

    Closes-Bug: #1982284
    Change-Id: I77fdfa9cffbd44b2889f49f266b2582bcc6a4267

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/860732

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/860733

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/860734

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/860735

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/860736

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/860737

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/860739

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/860732
Committed: https://opendev.org/openstack/nova/commit/00396fa9396324780c09161ed57a86b7e458c26f
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 00396fa9396324780c09161ed57a86b7e458c26f
Author: Brett Milford <email address hidden>
Date: Thu Aug 4 16:52:33 2022 +1000

    Handle "no RAM info was set" migration case

    This handles the case where the live migration monitoring thread may
    race and call jobStats() after the migration has completed resulting in
    the following error:

        libvirt.libvirtError: internal error: migration was active, but no
        RAM info was set

    Closes-Bug: #1982284
    Change-Id: I77fdfa9cffbd44b2889f49f266b2582bcc6a4267
    (cherry picked from commit 9fea934c71d3c2fa7fdd80c67d94e18466c5cf9a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/860733
Committed: https://opendev.org/openstack/nova/commit/4316234e63b76e4f9877ec6e924b5c54ea761bbb
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 4316234e63b76e4f9877ec6e924b5c54ea761bbb
Author: Brett Milford <email address hidden>
Date: Thu Aug 4 16:52:33 2022 +1000

    Handle "no RAM info was set" migration case

    This handles the case where the live migration monitoring thread may
    race and call jobStats() after the migration has completed resulting in
    the following error:

        libvirt.libvirtError: internal error: migration was active, but no
        RAM info was set

    Closes-Bug: #1982284
    Change-Id: I77fdfa9cffbd44b2889f49f266b2582bcc6a4267
    (cherry picked from commit 9fea934c71d3c2fa7fdd80c67d94e18466c5cf9a)
    (cherry picked from commit 00396fa9396324780c09161ed57a86b7e458c26f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/860734
Committed: https://opendev.org/openstack/nova/commit/98d9936e54b900db1bd2d5477a9a1d7e5a7be104
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 98d9936e54b900db1bd2d5477a9a1d7e5a7be104
Author: Brett Milford <email address hidden>
Date: Thu Aug 4 16:52:33 2022 +1000

    Handle "no RAM info was set" migration case

    This handles the case where the live migration monitoring thread may
    race and call jobStats() after the migration has completed resulting in
    the following error:

        libvirt.libvirtError: internal error: migration was active, but no
        RAM info was set

    Closes-Bug: #1982284
    Change-Id: I77fdfa9cffbd44b2889f49f266b2582bcc6a4267
    (cherry picked from commit 9fea934c71d3c2fa7fdd80c67d94e18466c5cf9a)
    (cherry picked from commit 00396fa9396324780c09161ed57a86b7e458c26f)
    (cherry picked from commit 4316234e63b76e4f9877ec6e924b5c54ea761bbb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/859358
Committed: https://opendev.org/openstack/nova/commit/ca9b7defe857404162bfd3909652bbd98514ffa8
Submitter: "Zuul (22348)"
Branch: master

commit ca9b7defe857404162bfd3909652bbd98514ffa8
Author: melanie witt <email address hidden>
Date: Mon Sep 26 22:43:51 2022 +0000

    Unit test exceptions raised duing live migration monitoring

    This adds some test coverage to verify the expected handling of
    exceptions raised in Guest.get_job_info() during live migration
    monitoring.

    In the case of the related bug fix, this testing verifies that the post
    live migration method is called after handling the special case and
    workaround of libvirt error "migration was active, but no RAM info was
    set".

    Related-Bug: #1982284

    Change-Id: Ibab54aa4f2e929b384b75e7e0a21c8b059b680c5

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 24.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 25.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 26.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 27.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/860739
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/860737
Reason: stable/ussuri branch of openstack/nova transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/860736
Reason: stable/victoria branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/860735
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.