Nova fails to parse new libvirt mediated device name format

[Joe Kralicky]

The name format of mediated devices in libvirt was recently changed from `mdev_<uuid>` to `mdev_<uuid>_<parent>`, e.g.:
Old: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5`
New: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5_0000_c1_00_0`

This results in the following error:

2021-11-19 22:51:45.952 7 ERROR nova.compute.manager [req-570c7e8f-0540-49fb-b2b0-8c2ac932e4dc - - - - -] Error updating resources for node: ValueError: badly formed hexadecimal UUID string
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager Traceback (most recent call last):
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 9993, in _update_available_resource_for_node
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 895, in update_available_resource
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return f(*args, **kwargs)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 975, in _update_available_resource
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager self._update(context, cn, startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1227, in _update
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 49, in wrapped_f
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 206, in call
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 247, in get
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/usr/local/lib/python3.6/site-packages/six.py", line 719, in reraise
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager raise value
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 200, in call
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1163, in _update_to_placement
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager self.driver.update_provider_tree(prov_tree, nodename)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8355, in update_provider_tree
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager provider_tree, nodename, allocations=allocations)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8757, in _update_provider_tree_for_vgpu
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager inventories_dict = self._get_gpu_inventories()
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7597, in _get_gpu_inventories
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager count_per_parent = self._count_mediated_devices(enabled_mdev_types)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7538, in _count_mediated_devices
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager mediated_devices = self._get_mediated_devices(types=enabled_mdev_types)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7788, in _get_mediated_devices
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager device = self._get_mediated_device_information(name)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7769, in _get_mediated_device_information
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager "uuid": libvirt_utils.mdev_name2uuid(cfgdev.name),
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/utils.py", line 583, in mdev_name2uuid
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return str(uuid.UUID(mdev_name[5:].replace('_', '-')))
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File "/usr/lib64/python3.6/uuid.py", line 140, in __init__
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager raise ValueError('badly formed hexadecimal UUID string')
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager ValueError: badly formed hexadecimal UUID string
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager

Nova is assuming everything after `mdev_` is the UUID, but this string now includes the parent PCI address.

Relevant commit in libvirt: https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d

Environment: qemu/kvm, latest openstack xena (kolla)

[Joe Kralicky (kralicky)]

Fixing this properly might be non-trivial but the following patch is a quick fix that I used to work around the issue:

--- a/nova/virt/libvirt/config.py
+++ b/nova/virt/libvirt/config.py
@@ -3136,6 +3136,8 @@ class LibvirtConfigNodeDevice(LibvirtConfigObject):
         for c in xmldoc:
             if c.tag == "name":
                 self.name = c.text
+ if len(self.name) > 41:
+ self.name = self.name[:41]
             elif c.tag == "parent":
                 self.parent = c.text
             elif c.tag == "capability" and c.get("type") in ['pci', 'net']:

[Sylvain Bauza (sylvain-bauza)]

Looks like it's modified since libvirt 7.7.0 version.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/832489

[Billy Olsen (billy-olsen)]

Looks like the device name was changed in libvirt commit which appends the parent name https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d.

From reading the commit message, it implies that using the UUID for identification is not unique enough since multiple parent devices may get the same UUID.

The UUID should be part of the XML returned so I suspect that it's better to use the nested UUID attribute rather than parse it from the name of the mediated device.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/838976

[Tobias Urdin (tobias-urdin)]

Hitting this when running Libvirt 8.0 with Xena as well.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/850673

[OpenStack Infra (hudson-openstack)]

Change abandoned by "sean mooney <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/832489
Reason: we are going to move forward with https://review.opendev.org/c/openstack/nova/+/838976

[Steven Relf (srelf)]

If im running into this issue on Yoga at the moment, what would be the best way to resolve? is it using https://review.opendev.org/c/openstack/nova/+/838976

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/850673
Committed: https://opendev.org/openstack/nova/commit/185201974775bab966f4e5ca3bbdc31b8269fa4c
Submitter: "Zuul (22348)"
Branch: master

commit 185201974775bab966f4e5ca3bbdc31b8269fa4c
Author: Sylvain Bauza <email address hidden>
Date: Thu Jul 21 18:21:51 2022 +0200

    Reproducer for bug 1951656

    Due to a new mdev naming, we can't parse it.

    Change-Id: I0f785178b132dfef668829558dea9f7e674abadb
    Related-Bug: #1951656

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/838976
Committed: https://opendev.org/openstack/nova/commit/a28b907c4f0dbba6e141a8fbea807e6cb0438977
Submitter: "Zuul (22348)"
Branch: master

commit a28b907c4f0dbba6e141a8fbea807e6cb0438977
Author: Billy Olsen <email address hidden>
Date: Thu Apr 21 19:42:27 2022 -0700

    Handle mdev devices in libvirt 7.7+

    Libvirt 7.7 changed the mdev device naming to include the parent PCI
    device when listing node devices. The domain, however, will still only
    see the UUID and not see the parent PCI device. Changing the parsing to
    simply drop the PCI identifier is not enough as the device cannot be
    found when attempting to lookup the new ID.

    Modify the Libvirt Driver's _get_mediated_device_information to tolerate
    different formats of the mdev name. This first uses the legacy behavior
    by trying to lookup the device name that is passed in (typically
    mdev_<uuid> format) and if that is not found, iterates the list of mdev
    node devices until the right UUID is found and selects that one.

    Note that the lookup of the mdev device by UUID are needed in order
    to keep the ability to recreate assigned mediated devices on a reboot of
    the compute node.

    Additionally, the libvirt utils parsing method mdev_name2uuid, has
    been updated to tolerate both mdev_<uuid> and mdev_<uuid>_<pciid>
    formats.

    Closes-Bug: 1951656

    Change-Id: Ifed0fa16053228990a6a8df8d4c666521db7e329

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/866151

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/866152

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/866153

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/866154

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/866155

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/866156

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/866157

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/866158

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 27.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/866151
Committed: https://opendev.org/openstack/nova/commit/857df72d3166a8f7e8a8cdfeabb62ad6ead46565
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 857df72d3166a8f7e8a8cdfeabb62ad6ead46565
Author: Sylvain Bauza <email address hidden>
Date: Thu Jul 21 18:21:51 2022 +0200

    Reproducer for bug 1951656

    Due to a new mdev naming, we can't parse it.

    Change-Id: I0f785178b132dfef668829558dea9f7e674abadb
    Related-Bug: #1951656
    (cherry picked from commit 185201974775bab966f4e5ca3bbdc31b8269fa4c)

[Christian Rohmann (christian-rohmann)]

Could you please consider creating new point releases for Nova with this fix included?

I just mentioned this issue in relation to a new stable/yoga point releases for Ubuntu Cloud Archive, https://bugs.launchpad.net/cloud-archive/+bug/2011713/comments/4.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/866152
Committed: https://opendev.org/openstack/nova/commit/98d8c9eaa3c415cc234193e6a9115db887751363
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 98d8c9eaa3c415cc234193e6a9115db887751363
Author: Billy Olsen <email address hidden>
Date: Thu Apr 21 19:42:27 2022 -0700

    Handle mdev devices in libvirt 7.7+

    Libvirt 7.7 changed the mdev device naming to include the parent PCI
    device when listing node devices. The domain, however, will still only
    see the UUID and not see the parent PCI device. Changing the parsing to
    simply drop the PCI identifier is not enough as the device cannot be
    found when attempting to lookup the new ID.

    Modify the Libvirt Driver's _get_mediated_device_information to tolerate
    different formats of the mdev name. This first uses the legacy behavior
    by trying to lookup the device name that is passed in (typically
    mdev_<uuid> format) and if that is not found, iterates the list of mdev
    node devices until the right UUID is found and selects that one.

    Note that the lookup of the mdev device by UUID are needed in order
    to keep the ability to recreate assigned mediated devices on a reboot of
    the compute node.

    Additionally, the libvirt utils parsing method mdev_name2uuid, has
    been updated to tolerate both mdev_<uuid> and mdev_<uuid>_<pciid>
    formats.

    Closes-Bug: 1951656

    Change-Id: Ifed0fa16053228990a6a8df8d4c666521db7e329
    (cherry picked from commit a28b907c4f0dbba6e141a8fbea807e6cb0438977)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/866153
Committed: https://opendev.org/openstack/nova/commit/71aa17a487136be5e938192857721d9119222811
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 71aa17a487136be5e938192857721d9119222811
Author: Sylvain Bauza <email address hidden>
Date: Thu Jul 21 18:21:51 2022 +0200

    Reproducer for bug 1951656

    Due to a new mdev naming, we can't parse it.

    Change-Id: I0f785178b132dfef668829558dea9f7e674abadb
    Related-Bug: #1951656
    (cherry picked from commit 185201974775bab966f4e5ca3bbdc31b8269fa4c)
    (cherry picked from commit 857df72d3166a8f7e8a8cdfeabb62ad6ead46565)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/866154
Committed: https://opendev.org/openstack/nova/commit/28053917200e3e242148672efda0e1a2b043dc48
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 28053917200e3e242148672efda0e1a2b043dc48
Author: Billy Olsen <email address hidden>
Date: Thu Apr 21 19:42:27 2022 -0700

    Handle mdev devices in libvirt 7.7+

    Libvirt 7.7 changed the mdev device naming to include the parent PCI
    device when listing node devices. The domain, however, will still only
    see the UUID and not see the parent PCI device. Changing the parsing to
    simply drop the PCI identifier is not enough as the device cannot be
    found when attempting to lookup the new ID.

    Modify the Libvirt Driver's _get_mediated_device_information to tolerate
    different formats of the mdev name. This first uses the legacy behavior
    by trying to lookup the device name that is passed in (typically
    mdev_<uuid> format) and if that is not found, iterates the list of mdev
    node devices until the right UUID is found and selects that one.

    Note that the lookup of the mdev device by UUID are needed in order
    to keep the ability to recreate assigned mediated devices on a reboot of
    the compute node.

    Additionally, the libvirt utils parsing method mdev_name2uuid, has
    been updated to tolerate both mdev_<uuid> and mdev_<uuid>_<pciid>
    formats.

    Closes-Bug: 1951656

    Change-Id: Ifed0fa16053228990a6a8df8d4c666521db7e329
    (cherry picked from commit a28b907c4f0dbba6e141a8fbea807e6cb0438977)
    (cherry picked from commit 98d8c9eaa3c415cc234193e6a9115db887751363)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/866155
Committed: https://opendev.org/openstack/nova/commit/fdd4e7dc51ad34d3bc4741e7e40a10e8f3a7d138
Submitter: "Zuul (22348)"
Branch: stable/xena

commit fdd4e7dc51ad34d3bc4741e7e40a10e8f3a7d138
Author: Sylvain Bauza <email address hidden>
Date: Thu Jul 21 18:21:51 2022 +0200

    Reproducer for bug 1951656

    Due to a new mdev naming, we can't parse it.

    Change-Id: I0f785178b132dfef668829558dea9f7e674abadb
    Related-Bug: #1951656
    (cherry picked from commit 185201974775bab966f4e5ca3bbdc31b8269fa4c)
    (cherry picked from commit 857df72d3166a8f7e8a8cdfeabb62ad6ead46565)
    (cherry picked from commit 71aa17a487136be5e938192857721d9119222811)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/866156
Committed: https://opendev.org/openstack/nova/commit/c3800f7863116412e90029381085cd01ebeae220
Submitter: "Zuul (22348)"
Branch: stable/xena

commit c3800f7863116412e90029381085cd01ebeae220
Author: Billy Olsen <email address hidden>
Date: Thu Apr 21 19:42:27 2022 -0700

    Handle mdev devices in libvirt 7.7+

    Libvirt 7.7 changed the mdev device naming to include the parent PCI
    device when listing node devices. The domain, however, will still only
    see the UUID and not see the parent PCI device. Changing the parsing to
    simply drop the PCI identifier is not enough as the device cannot be
    found when attempting to lookup the new ID.

    Modify the Libvirt Driver's _get_mediated_device_information to tolerate
    different formats of the mdev name. This first uses the legacy behavior
    by trying to lookup the device name that is passed in (typically
    mdev_<uuid> format) and if that is not found, iterates the list of mdev
    node devices until the right UUID is found and selects that one.

    Note that the lookup of the mdev device by UUID are needed in order
    to keep the ability to recreate assigned mediated devices on a reboot of
    the compute node.

    Additionally, the libvirt utils parsing method mdev_name2uuid, has
    been updated to tolerate both mdev_<uuid> and mdev_<uuid>_<pciid>
    formats.

    Closes-Bug: 1951656

    Change-Id: Ifed0fa16053228990a6a8df8d4c666521db7e329
    (cherry picked from commit a28b907c4f0dbba6e141a8fbea807e6cb0438977)
    (cherry picked from commit 98d8c9eaa3c415cc234193e6a9115db887751363)
    (cherry picked from commit 28053917200e3e242148672efda0e1a2b043dc48)

[Christian Rohmann (christian-rohmann)]

Are there any plans for new point releases for e.g. Yoga?

See my comment at https://bugs.launchpad.net/nova/+bug/1951656/comments/22

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 24.2.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 25.1.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 26.1.1 release.

[Lars Erik Pedersen (pedersen-larserik)]

Any plans for including this in UCA for yoga?

[Nobuto Murata (nobuto)]

Ubuntu's SRU for Nova 25.1.1 is tracked at
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/2019759

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/866158
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/866157
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.