Lack of proper handling of new firmware architectures

Bug #2122288 reported by Takashi Kajinami
62
This bug affects 10 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Takashi Kajinami
Ubuntu Cloud Archive
Status tracked in Flamingo
Caracal
New
Undecided
Unassigned
Dalmatian
New
Undecided
Unassigned
Epoxy
New
Undecided
Unassigned
Flamingo
New
Undecided
Unassigned
Gazpacho
New
Undecided
Unassigned
nova (Ubuntu)
Status tracked in Resolute
Noble
New
Undecided
Unassigned
Questing
In Progress
Undecided
Marcin Wilk
Resolute
In Progress
Undecided
Marcin Wilk

Bug Description

[ Impact ]
Users reported that when trying to launch a VM from the UEFI-enabled image on Ubuntu 24.04 (Noble) with ovmf package versions 2024.02-2ubuntu0.5 and later, the VM fails to launch because nova doesn't know some of the libvirt-provided firmware features. As a result, they can't run UEFI-booted VMs.

This problem has already been resolved in the upstream code [1]. I have verified that the upstream patch fixes the problem with launching a UEFI-enabled VM on Resolute and Questing.

[1] https://opendev.org/openstack/nova/commit/d2188b9e6b35ba512661711165cef706f8bed693

[ Test Plan ]
1. Deploy the regress-stack on a Resolute VM
(make sure the VM has enough resources to run regress-stack):
sudo apt update && sudo apt upgrade -y
sudo apt install -y dpkg-dev python3-dev python-apt-dev
sudo snap install astral-uv --classic
git clone https://github.com/canonical/regress-stack.git
cd regress-stack
# Python and dependency version pinning
uv python pin 3.13
sed -i 's/requires-python = ".*"/requires-python = ">=3.13"/' pyproject.toml
sed -i 's/^ "python-apt"/# "python-apt"/' pyproject.toml
sed -i 's/^python-apt =/#python-apt = /' pyproject.toml

# Nova and Neutron service name fixes
sed -i 's/ core_utils.restart_service("neutron-server")/ core_utils.restart_service("apache2")/' src/regress_stack/modules/neutron.py
sed -i '/ core_utils.restart_service("neutron-ovn-metadata-agent")/a\ core_utils.restart_service("neutron-periodic-workers")' src/regress_stack/modules/neutron.py
sed -i '/ core_utils.restart_service("neutron-ovn-metadata-agent")/a\ core_utils.restart_service("neutron-rpc-server")' src/regress_stack/modules/neutron.py
sed -i 's/ core_utils.restart_service("nova-api")/ core_utils.restart_service("apache2")/' src/regress_stack/modules/nova.py
uvx pre-commit install
uv venv --system-site-packages --python-preference system
uv sync
uv run py.test
sudo apt install -y crudini ceph-mgr ceph-mon ceph-osd ceph-volume mysql-server keystone apache2 libapache2-mod-wsgi-py3 glance-api ovn-central openvswitch-switch ovn-host placement-api rabbitmq-server cinder-api cinder-scheduler cinder-volume neutron-server neutron-ovn-metadata-agent nova-api nova-conductor nova-scheduler nova-compute nova-spiceproxy spice-html5
sudo uv run regress-stack setup
sudo cp /root/auth.rc ~
sudo chown $(id -u):$(id -g) ~/auth.rc
source ~/auth.rc

# verify OpenStack is up and running:
openstack endpoint list
openstack service list
openstack hypervisor list

2. Create required resources in OpenStack
wget https://cloud-images.ubuntu.com/releases/noble/release/ubuntu-24.04-server-cloudimg-amd64.img
openstack flavor create --ram 2048 --vcpus 1 --disk 20 m1.small
openstack keypair create testkey
openstack network create private
openstack subnet create --network private --ip-version 4 --subnet-range 192.168.21.0/24 --allocation-pool start=192.168.21.2,end=192.168.21.254 private_subnet
openstack image create --container-format bare --disk-format qcow2 --public --file ubuntu-24.04-server-cloudimg-amd64.img noble_uefi
openstack image set --property hw_machine_type=q35 --property hw_firmware_type=uefi noble_uefi

3. Try to create a UEFI-enabled VM:
openstack server create --image noble_uefi --flavor m1.small --key-name testkey --network private noble1

Without the patch, the VM will end up in the ERROR state, /var/log/nova/nova-compute.log will contain the following error messages:
2026-02-12 16:11:34.693 47928 INFO nova.virt.libvirt.host [None req-e741f97a-3201-40fc-9bb5-a76e2d378697 85da01e2dffd4b7d83a4ce1fb1713a39 b71f3390a85142b2aa247e9070bd469b - - default default] UEFI support detected
2026-02-12 16:11:34.694 47928 ERROR nova.compute.manager [None req-e741f97a-3201-40fc-9bb5-a76e2d378697 85da01e2dffd4b7d83a4ce1fb1713a39 b71f3390a85142b2aa247e9070bd469b - - default default] [instance: c5c12f18-4189-4eeb-8784-97e6f77ea8fb] Instance failed to spawn: KeyError: 'nvram-template'

With the patch applied, the VM will be successfully created

[ Where problems could occur ]
The patch modifies the code related/isolated to the UEFI-enabled VMs on KVM/libvirt hypervisors. The code change is small. If it causes a regression, most likely, a UEFI-enabled VM would not launch (but they don't launch already due to this bug). The /var/log/nova/nova-compute.log log should be examined to get a better understanding wof hat went wrong.

[ Other Info ]

The original bug description:

Description
===========

Recent CentOS and Fedora introduced a few new architecture patterns of OVMF firmware.
However current nova does not handle these patterns appropriately and assumes that every firmware is always "statefull" and also "plash" type.

1) mode=stateless
example. 60-edk2-ovmf-x64-amdsev.json in Fedora 40
---
{
    "description": "OVMF with SEV-ES support",
    "interface-types": [
        "uefi"
    ],
    "mapping": {
        "device": "flash",
        "mode": "stateless",
        "executable": {
            "filename": "/usr/share/edk2/ovmf/OVMF.amdsev.fd",
            "format": "raw"
        }
    },
    "targets": [
        {
            "architecture": "x86_64",
            "machines": [
                "pc-q35-*"
            ]
        }
    ],
    "features": [
        "amd-sev",
        "amd-sev-es",
        "amd-sev-snp",
        "verbose-dynamic"
    ],
    "tags": [

    ]
}
---

2) device=memory
example. 60-edk2-ovmf-x64-amdsev.json in CentOS Stream 10

---
{
    "description": "OVMF with SEV-ES support",
    "interface-types": [
        "uefi"
    ],
    "mapping": {
        "device": "memory",
        "filename": "/usr/share/edk2/ovmf/OVMF.amdsev.fd"
    },
    "targets": [
        {
            "architecture": "x86_64",
            "machines": [
                "pc-q35-*"
            ]
        }
    ],
    "features": [
        "amd-sev",
        "amd-sev-es",
        "amd-sev-snp",
        "verbose-dynamic"
    ],
    "tags": [

    ]
}
---

These files should be excluded when detecting the loader files, to avoid KeyError or launch time error.

Tags: patch
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/959985

Changed in nova:
status: New → In Progress
Revision history for this message
Takashi Kajinami (kajinamit) wrote :

I wonder if we should also exclude qcow2 format or should set format='qcow2' if we use these.

description: updated
Revision history for this message
Léo Marché (leomarche) wrote :
Download full text (3.6 KiB)

Bug also affects ubuntu 24.04 (noble) with ovmf package versions 2024.02-2ubuntu0.5 and later with the file '60-edk2-ovmf-x64-amdsev.json'

nova-compute log when spawning a VM:

---
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [None req-4640bbb3-f56f-4e4e-9ea8-9198ec13b417 d62ed6f13e4d4a6382c9336c03e1b8e4 20e37549bd0d4654924e9cc74e686c74 - - default default] [instance: 95409154-ccfa-4094-877a-234583a2f86f] Failed to build and run instance: KeyError: 'nvram-template'
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] Traceback (most recent call last):
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] File "/openstack/venvs/nova-30.1.0/lib/python3.12/site-packages/nova/compute/manager.py", line 2642, in _build_and_run_instance
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] self.driver.spawn(context, instance, image_meta,
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] File "/openstack/venvs/nova-30.1.0/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 4757, in spawn
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] xml = self._get_guest_xml(context, instance, network_info,
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] File "/openstack/venvs/nova-30.1.0/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 7931, in _get_guest_xml
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] conf = self._get_guest_config(instance, network_info, image_meta,
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] File "/openstack/venvs/nova-30.1.0/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 7432, in _get_guest_config
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] self._configure_guest_by_virt_type(guest, instance, image_meta, flavor)
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] File "/openstack/venvs/nova-30.1.0/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 7017, in _configure_guest_by_virt_type
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] self._host.get_loader(
2025-11-27 13:40:31.013 56259 ERROR nova.compute.manager [instance: 95409154-ccfa-4094-877a-234583a2f86f] File "/openstack/venvs/nova-30.1.0/lib/python3.12/site-packages/nova/virt/libvirt/host.py", line 2048, in get_loader
2025-11-27 13:40:3...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by "Takashi Kajinami <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/959988

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Takashi Kajinami <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/959985
Reason: Replaced by https://review.opendev.org/c/openstack/nova/+/968942

Changed in nova:
assignee: nobody → Takashi Kajinami (kajinamit)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/968942
Committed: https://opendev.org/openstack/nova/commit/d2188b9e6b35ba512661711165cef706f8bed693
Submitter: "Zuul (22348)"
Branch: master

commit d2188b9e6b35ba512661711165cef706f8bed693
Author: Takashi Kajinami <email address hidden>
Date: Sat Nov 29 23:31:59 2025 +0900

    libvirt: Skip unsupported firmware types

    Ignore (1) stateless mode firmware and (2) memory device firmware which
    do not include a few core keys such as nvram-template. This is
    a temporal (and backportable) workaround until firmware detection using
    libvirt's internal feature is implemented by [1]

    [1] https://blueprints.launchpad.net/nova/+spec/libvirt-firmware-auto-selection

    Closes-Bug: #2122288
    Change-Id: I99bc36fdd5df816c9ae374db71e4734fb7fc467b
    Signed-off-by: Takashi Kajinami <email address hidden>

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/2025.2)

Fix proposed to branch: stable/2025.2
Review: https://review.opendev.org/c/openstack/nova/+/972276

Revision history for this message
Stefan Pinter (nidrix-glandda) wrote :

will this fix be proposed to 2025.1 as well as it is affected too?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/2025.2)

Reviewed: https://review.opendev.org/c/openstack/nova/+/972276
Committed: https://opendev.org/openstack/nova/commit/53c253db09eef3e91e2518e5862cacd73e860da9
Submitter: "Zuul (22348)"
Branch: stable/2025.2

commit 53c253db09eef3e91e2518e5862cacd73e860da9
Author: Takashi Kajinami <email address hidden>
Date: Sat Nov 29 23:31:59 2025 +0900

    libvirt: Skip unsupported firmware types

    Ignore (1) stateless mode firmware and (2) memory device firmware which
    do not include a few core keys such as nvram-template. This is
    a temporal (and backportable) workaround until firmware detection using
    libvirt's internal feature is implemented by [1]

    [1] https://blueprints.launchpad.net/nova/+spec/libvirt-firmware-auto-selection

    Closes-Bug: #2122288
    Change-Id: I99bc36fdd5df816c9ae374db71e4734fb7fc467b
    Signed-off-by: Takashi Kajinami <email address hidden>
    (cherry picked from commit d2188b9e6b35ba512661711165cef706f8bed693)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/2025.1)

Fix proposed to branch: stable/2025.1
Review: https://review.opendev.org/c/openstack/nova/+/973821

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/2025.1)

Reviewed: https://review.opendev.org/c/openstack/nova/+/973821
Committed: https://opendev.org/openstack/nova/commit/09f52ec4427f3aaec28cedea381e00c65b75e6fa
Submitter: "Zuul (22348)"
Branch: stable/2025.1

commit 09f52ec4427f3aaec28cedea381e00c65b75e6fa
Author: Takashi Kajinami <email address hidden>
Date: Sat Nov 29 23:31:59 2025 +0900

    libvirt: Skip unsupported firmware types

    Ignore (1) stateless mode firmware and (2) memory device firmware which
    do not include a few core keys such as nvram-template. This is
    a temporal (and backportable) workaround until firmware detection using
    libvirt's internal feature is implemented by [1]

    [1] https://blueprints.launchpad.net/nova/+spec/libvirt-firmware-auto-selection

    Closes-Bug: #2122288
    Change-Id: I99bc36fdd5df816c9ae374db71e4734fb7fc467b
    Signed-off-by: Takashi Kajinami <email address hidden>
    (cherry picked from commit d2188b9e6b35ba512661711165cef706f8bed693)
    (cherry picked from commit 53c253db09eef3e91e2518e5862cacd73e860da9)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/2024.2)

Fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/nova/+/974409

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/2024.2)

Reviewed: https://review.opendev.org/c/openstack/nova/+/974409
Committed: https://opendev.org/openstack/nova/commit/4c3fea7fae317e38f7e9ea5967f28dbd43854cdd
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit 4c3fea7fae317e38f7e9ea5967f28dbd43854cdd
Author: Takashi Kajinami <email address hidden>
Date: Sat Nov 29 23:31:59 2025 +0900

    libvirt: Skip unsupported firmware types

    Ignore (1) stateless mode firmware and (2) memory device firmware which
    do not include a few core keys such as nvram-template. This is
    a temporal (and backportable) workaround until firmware detection using
    libvirt's internal feature is implemented by [1]

    [1] https://blueprints.launchpad.net/nova/+spec/libvirt-firmware-auto-selection

    Closes-Bug: #2122288
    Change-Id: I99bc36fdd5df816c9ae374db71e4734fb7fc467b
    Signed-off-by: Takashi Kajinami <email address hidden>
    (cherry picked from commit d2188b9e6b35ba512661711165cef706f8bed693)
    (cherry picked from commit 53c253db09eef3e91e2518e5862cacd73e860da9)
    (cherry picked from commit 09f52ec4427f3aaec28cedea381e00c65b75e6fa)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (unmaintained/2024.1)

Fix proposed to branch: unmaintained/2024.1
Review: https://review.opendev.org/c/openstack/nova/+/976244

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (unmaintained/2024.1)

Reviewed: https://review.opendev.org/c/openstack/nova/+/976244
Committed: https://opendev.org/openstack/nova/commit/10c37270c5d0044e702bef5ca53f4963cc6f14f8
Submitter: "Zuul (22348)"
Branch: unmaintained/2024.1

commit 10c37270c5d0044e702bef5ca53f4963cc6f14f8
Author: Takashi Kajinami <email address hidden>
Date: Sat Nov 29 23:31:59 2025 +0900

    libvirt: Skip unsupported firmware types

    Ignore (1) stateless mode firmware and (2) memory device firmware which
    do not include a few core keys such as nvram-template. This is
    a temporal (and backportable) workaround until firmware detection using
    libvirt's internal feature is implemented by [1]

    [1] https://blueprints.launchpad.net/nova/+spec/libvirt-firmware-auto-selection

    Closes-Bug: #2122288
    Change-Id: I99bc36fdd5df816c9ae374db71e4734fb7fc467b
    Signed-off-by: Takashi Kajinami <email address hidden>
    (cherry picked from commit d2188b9e6b35ba512661711165cef706f8bed693)
    (cherry picked from commit 53c253db09eef3e91e2518e5862cacd73e860da9)
    (cherry picked from commit 09f52ec4427f3aaec28cedea381e00c65b75e6fa)
    (cherry picked from commit 4c3fea7fae317e38f7e9ea5967f28dbd43854cdd)

Marcin Wilk (wilkmarcin)
description: updated
Marcin Wilk (wilkmarcin)
no longer affects: nova (Ubuntu Noble)
no longer affects: nova (Ubuntu Questing)
no longer affects: nova (Ubuntu Resolute)
Marcin Wilk (wilkmarcin)
no longer affects: cloud-archive/dalmatian
no longer affects: cloud-archive/epoxy
no longer affects: cloud-archive/flamingo
Marcin Wilk (wilkmarcin)
Changed in nova (Ubuntu Resolute):
assignee: nobody → Marcin Wilk (wilkmarcin)
status: New → In Progress
Marcin Wilk (wilkmarcin)
Changed in nova (Ubuntu Questing):
status: New → In Progress
Revision history for this message
Marcin Wilk (wilkmarcin) wrote (last edit ):

Added debdiff for Ubuntu Questing
The package built successfully in Launchpad.
I tested the package on the regress-stack Questing, and it solves the problem.

Revision history for this message
Marcin Wilk (wilkmarcin) wrote :

For Ubuntu Resolute, the patch is already in the proposed pocket, package version: 3:32.0.0+git20260128.59a7093-0ubuntu1

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lp2122288.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Marcin Wilk (wilkmarcin)
Changed in nova (Ubuntu Questing):
assignee: nobody → Marcin Wilk (wilkmarcin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/969132
Committed: https://opendev.org/openstack/nova/commit/5841095740fb4e8c5a61e8f7673aa9478f4c3953
Submitter: "Zuul (22348)"
Branch: master

commit 5841095740fb4e8c5a61e8f7673aa9478f4c3953
Author: Takashi Kajinami <email address hidden>
Date: Tue Dec 2 00:36:57 2025 +0900

    libvirt: Use firmware auto-selection by libvirt

    Use the firmware auto-selection feature in libvirt to find the best
    UEFI firmware file according to the requested feature.

    Firmware files may be reselected when a libvirt domain is created from
    scratch, while these are kept during hard-reboot (or live migration
    which preserves the loader/nvram elements filled by libvirt).

    Closes-Bug: #2122296
    Related-Bug: #2122288
    Implements: blueprint libvirt-firmware-auto-selection
    Change-Id: Ie48b020597a1a2fb3280815eec5ba3565e396f9b
    Signed-off-by: Takashi Kajinami <email address hidden>

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.