please provide 4MB firmware builds

[Dimitri John Ledkov]

[ Impact ]

Windows virtual machines running on top of Ubuntu Focal and Bionic hypervisors with the OVMF UEFI bios are facing an issue while installing the KB5012170 update from Microsoft. The windows update cannot proceed to install the KB5012170, because the 2MB OVMF image on the QEMU host no longer has sufficient space to store the contents for the update, which causes the update process to fail. The people running Windows VMs on Ubuntu Focal/QEMU are unable to update their VMs.

There is a known workaround problem, which is using 4MB OVMF images from either Debian or Ubuntu Jammy.

This fix is a backport of 4MB OVMF images from upstream for Focal. This fix also changes the default OVMF image from 2MB to 4MB for all new VMs that will be spawned by a hypervisor using the descriptor interface.

Apart from the fix, the SRU includes autopkgtests for the backported changes as well.

[ Test Plan ]

The test plan basically boils down to the scenario below:

- Spawn a Focal/QEMU host with the container/VM technology of your choice
- Create a Windows 10 VM on QEMU (use existing OVMF images at launch)
- Try to install KB5012170 on VM, and it should fail
- Install the fix to the Focal/QEMU host via PPA
- Launch the VM with new 4M images
- Try to install KB5012170 on VM, and it should succeed

The steps below are a reproduction of the test plan above with LXC as a host environment for Focal/QEMU.

# Environment setup describes QEMU running over
# Focal LXC with X11 forwarding

# Download windows 10 ISO:
# https://www.microsoft.com/en-us/software-download/windows10ISO

# -- host shell --
# Create an LXC profile that enables X11 forwarding
# for containers <See the attached x11.profile file>
$ lxc profile create x11
$ cat x11.profile | lxc profile edit x11
# Create a Focal/QEMU container with X11 forwarding
$ lxc launch ubuntu:focal --profile default --profile x11 test-qemu-host
# Mount folder containing windows iso
$ lxc config device add test-qemu-host winiso disk source=/workspace/os-iso path=/os-iso
$ lxc exec test-qemu-host -- sudo --user ubuntu --login

# -- QEMU container shell --
# Install QEMU & OVMF
$ sudo apt -y update && sudo apt -y install qemu-kvm ovmf
# Create a windows VM
$ mkdir windows-vm && cd windows-vm
$ qemu-img create -f qcow2 windows.img 32G
$ cp /usr/share/OVMF/OVMF_VARS.ms.fd OVMF_VARS.ms.windows.fd
# Launch the VM
$ qemu-system-x86_64 -hda windows.img -m 4096M -smp cores=4 -M q35,smm=on -global ICH9-LPC.disable_s3=1 -global driver=cfi.pflash01,property=secure,value=on -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.ms.fd -drive if=pflash,format=raw,file=./OVMF_VARS.ms.windows.fd -cdrom /os-iso/Win10_22H2_English_x64.iso
# Setup the operating system
# After everything settles:
# Download KB5012170 from Microsoft Update Catalog
# https://www.catalog.update.microsoft.com/Search.aspx?q=KB5012170
# Try to install the KB5012170 update, it should fail
# Shutdown VM
# Install fix to QEMU host from PPA:
sudo add-apt-repository ppa:mustafakemalgilor/lp-1885662-4
sudo apt update
# Re-launch VM with new 4M images:
cp /usr/share/OVMF/OVMF_VARS_4M.ms.fd OVMF_VARS_4M.ms.windows.fd
qemu-system-x86_64 -hda windows.img -m 8192M -smp cores=4 -M q35,smm=on -global ICH9-LPC.disable_s3=1 -global driver=cfi.pflash01,property=secure,value=on -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.ms.fd -drive if=pflash,format=raw,file=./OVMF_VARS_4M.ms.windows.fd
# Re-try to install the KB5012170, it should succeed

[ Where problems could occur ]

This SRU introduces providing an OVMF image with a higher capacity variable store, which will have no impact on existing VMs. The migration of existing VMs will be opt-in.

The new VMs will be using the 4MB OVMF image by default, given that the owning hypervisor is using the descriptor interface.

The user will be able to switch back to the 2MB image in case of regression.

[ Other Info ]

This update does not replace the existing 2MB OVMF image; instead, it provides a new 4MB image. Therefore, the existing VMs should be manually migrated to the 4MB image.

[Original bug description]
-DFD_SIZE_2MB OVMF builds are not longer fit for purpose, as they are too small to apply dbxupdate.

We must provide -DFD_SIZE_4MB and default to them.

We must keep -DFD_SIZE_2MB builds to keep existing VMs working.

[Dimitri John Ledkov (xnox)]

or one can use FD_SIZE_4MB

[Dimitri John Ledkov (xnox)]

This is confirmed as impacting application of large dbxupdates.

[Dimitri John Ledkov (xnox)]

I guess this embargoed until Wednesday 10am PDT.

[dann frazier (dannf)]

What's the security vulnerability? And does QEMU need to know about the size change as well?
Looks like SuSE may have a special 4M flavor for X64:

 https://build.opensuse.org/package/view_file/Virtualization/ovmf/ovmf.spec

[Dimitri John Ledkov (xnox)]

2MB builds no longer suitable for Secureboot, as the nvram space is now too small to remain secure. The next dbxupdate is larger than what fits in NVRAM of the 2MB firmware.

The 4MB and 2MB are incompatible with respect to VARS. The VM boots with 4MB code and "old" VARS, but the efivars state is all wrong. Instead of secureboot on, it's secureboot off.

We must add 4MB flavour, and add it to qemu. Going with similar names as SUSE i guess is helpful.

Ideally somehow order 4MB flavour "first" such that e.g. launching new VMs uses 4MB firmwares.

I wish we could drop 2MB firmwares somehow.

[Dimitri John Ledkov (xnox)]

Updated description.

Also affects LXD, as it too builds 2MB big OVMF firmware. It should switch to 4MB right away, if possible.

[Dimitri John Ledkov (xnox)]

So i think since we can't fix existing firmware sizes, this can just go in as a regular SRU.

[Stéphane Graber (stgraber)]

Is there some way to detect whether a copy of VARS is in 2MB or 4MB format?

If so, then for LXD we'd most likely add detection logic and have LXD reset VARS when detecting a mis-match (making backup, logging info message, ...).

If not, we'll also go down the path of building two copies, at least for a while.

[dann frazier (dannf)]

@Stéphane: I suspect you can just look at the size of the VARS file itself. 2MB VARS files are 256K. 4MB ones are (...does test build...) 528K. I believe these file sizes are static, but would be good to verify.

Bumping from 2MB->4MB apparently bumps our Non-volatile variable storage from 56KB to 256KB:
 https://github.com/tianocore/edk2/blob/master/OvmfPkg/README#L350
If we're already surpassing 56KB, is bumping to just 256KB short sighted? 4MB/256KB seems to be the max upstream supports at the moment, so we likely don't have any other option.

fyi, it looks like ARM/AARCH64 have 256K VarStores already:
$ grep Size: edk2-2020.05/ArmVirtPkg/VarStore.fdf.inc
  # Size: 0x40000 (gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageVariableSize) -

Is there descriptor (debian/descriptors/*.json) syntax for exposing the supported variable size? I don't see anything in QEMU's descriptors in master, which I believe are canonical. I suppose we can just point 40-edk2-x86_64-secure-enrolled.json and 50-edk2-x86_64-secure.json to the larger images and no longer provide descriptors that point at the 2MB images. That's the best way I can think of to prioritize 4MB over 2MB - and should work for libvirt and things built on top of it like OpenStack.

Are we coordinating with other vendors on this? Would obviously be nice to do things consistently.

[dann frazier (dannf)]

It would also be good to have a migration strategy for libvirt-based guests do we know if they are working on one?

[dann frazier (dannf)]

I've got a patch that I'm iterating on locally. @Dimitri: Was the GRUB embargo the only embargo to which this is bound? If so, may I upload test fixes to a public PPA for testing and possibly open up this bug?

[Dimitri John Ledkov (xnox)]

Embargo is lifted, the dbxupdate is now available from uefi.org. One can drop it into /usr/share/secureboot/updates/dbx/ and trigger reinstall of secureboot-db package to observe that dbx update fails to apply on 2MB firmware, but should work on the 4MB firmware.

There are update securebootdb packages available from a PPA too, I will make them available tomorrow.

[dann frazier (dannf)]

I've uploaded a test package for groovy to:
   https://launchpad.net/~dannf/+archive/ubuntu/test
And I'm attaching the debdiff here.

I'm not married to the image names I chose (OVMF_CODE_4M.fd et al), so open to other suggestions.
Also, I have not addressed OVMF_VARS.snakeoil.fd yet. For stable releases, I suppose we need to add a new 4M variant like we do for the other images. But for devel releases, I'd also like to drop the 2M image at the same time.

[Launchpad Janitor (janitor)]

This bug was fixed in the package edk2 - 2020.05-3

---------------
edk2 (2020.05-3) unstable; urgency=medium

  * Provide 4MB OVMF images as the existing 2MB images no longer
    have sufficient variable space for the current Secure Boot Forbidden
    Signature Database. LP: #1885662.
  * Update fw descriptors to reference 4M images instead of their 2M
    counterparts. This will migrate tools that use the descriptor interface
    (like libvirt) over to the 4M images when creating new VMs. Existing 2M
    VMs will require manual migration.
  * Add a 4M snakeoil variable template and drop the 2M version. This will
    break existing snakeoil VMs, but that should be OK for a test/devel
    facility.
  * Increase autopkgtest timeout from 30s to 60s. LP: #1885186.

 -- dann frazier <email address hidden> Wed, 05 Aug 2020 18:33:22 -0600

[Mustafa Kemal Gilor (mustafakemalgilor)]

-- SRU paperwork for Focal --

[ Impact ]

Windows virtual machines running on top of Ubuntu Focal and Bionic hypervisors with the OVMF UEFI bios are facing an issue while installing the KB5012170 update from Microsoft. The windows update cannot proceed to install the KB5012170, because the 2MB OVMF image on the QEMU host no longer has sufficient space to store the contents for the update, which causes the update process to fail. The people running Windows VMs on Ubuntu Focal/QEMU are unable to update their VMs.

There is a known workaround problem, which is using 4MB OVMF images from either Debian or Ubuntu Jammy.

This fix is a backport of 4MB OVMF images from upstream for Focal. This fix also changes the default OVMF image from 2MB to 4MB for all new VMs that will be spawned by a hypervisor using the descriptor interface.

Apart from the fix, the SRU includes a patch that aims to eliminate the flakiness of the autopkgtests by increasing the timeout value, as well as autopkgtests for the backported changes.

[ Test Plan ]

The test plan basically boils down to the scenario below:

- Spawn a Focal/QEMU host with the container/VM technology of your choice
- Create a Windows 10 VM on QEMU (use existing OVMF images at launch)
- Try to install KB5012170 on VM, and it should fail
- Install the fix to the Focal/QEMU host via PPA
- Launch the VM with new 4M images
- Try to install KB5012170 on VM, and it should succeed

The steps below are a reproduction of the test plan above with LXC as a host environment for Focal/QEMU.

# Environment setup describes QEMU running over
# Focal LXC with X11 forwarding

# Download windows 10 ISO:
# https://www.microsoft.com/en-us/software-download/windows10ISO

# -- host shell --
# Create an LXC profile that enables X11 forwarding
# for containers <See the attached x11.profile file>
$ lxc profile create x11
$ cat x11.profile | lxc profile edit x11
# Create a Focal/QEMU container with X11 forwarding
$ lxc launch ubuntu:focal --profile default --profile x11 test-qemu-host
# Mount folder containing windows iso
$ lxc config device add test-qemu-host winiso disk source=/workspace/os-iso path=/os-iso
$ lxc exec test-qemu-host -- sudo --user ubuntu --login

# -- QEMU container shell --
# Install QEMU & OVMF
$ sudo apt -y update && sudo apt -y install qemu-kvm ovmf
# Create a windows VM
$ mkdir windows-vm && cd windows-vm
$ qemu-img create -f qcow2 windows.img 32G
$ cp /usr/share/OVMF/OVMF_VARS.ms.fd OVMF_VARS.ms.windows.fd
# Launch the VM
$ qemu-system-x86_64 -hda windows.img -m 4096M -smp cores=4 -M q35,smm=on -global ICH9-LPC.disable_s3=1 -global driver=cfi.pflash01,property=secure,value=on -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.ms.fd -drive if=pflash,format=raw,file=./OVMF_VARS.ms.windows.fd -cdrom /os-iso/Win10_22H2_English_x64.iso
# Setup the operating system
# After everything settles:
# Download KB5012170 from Microsoft Update Catalog
# https://www.catalog.update.microsoft.com/Search.aspx?q=KB5012170
# Try to install the KB5012170 update, it should fail
# Shutdown VM
# Install fix to QEMU host from PPA:
sudo add-apt-repository ppa:mustafakemalgilor/lp-1885662-4
sudo apt update
# Re-l...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Ehrhardt  (paelzer)]

Hi,
I've seen this fly bug by and wanted to warn (= ask for early testing) about an associated risk.
Changing rom sizes happens sometimes and back in the day of bug 1713490 it was unplanned :-/
This time we at least have the benefit of doing it consciously.

The tricky part is cross release / cross update live migration.
Your guest has memory of a given size allocated for that ROM and that is usually derived from the file size - when migrating to a target that has a different file size it will allocate a different size and fail the migration due to that.
And that is the reason why - usually - one is supposed to keep the size as-is and if changing it use an extra file for the new size. That might explain some of the cases you saw with other Distributors.

I wanted to ask to test the following migrations:
1. same release: pre-upgrade -> post-upgrade
2. same release: post-upgrade -> pre-upgrade (there is no hard guarantee for this, but we try rather hard to keep it working whenever possible)
3. cross release: bionic -> unupdated focal (just to verify your setup works)
4. cross release: bionic -> updated focal
5. cross release: unupdated focal -> jammy (just to verify your setup works)
6. cross release: updated focal -> jammy
7. cross release: bionic -> unupdated focal -> jammy (just to verify your setup works)
8. cross release: bionic -> updated focal -> jammy

Maybe we are lucky and the ram allocation for edk2 roms is smarter than the one for ipxe and this is not an issue. But I wanted to make you aware of the major regression risk and ask you to test this in advance.

#3, #5, #7 are self checks and should always work, but if either of #1, #4, #6, #8 and to some extent if #2 fails - this needs to go back at the drawing board and probably needs a bigger fix with the same design as bug 1713490.

[Mustafa Kemal Gilor (mustafakemalgilor)]

Hi Christian,

Thanks for stopping by and raising this regression possibility to my notice -- greatly appreciated. This is an angle to the topic that I was not aware of.

> And that is the reason why - usually - one is supposed to keep the
> size as-is and if changing it use an extra file for the new size.
> That might explain some of the cases you saw with other
> Distributors.

I think this SRU exactly does this as well. We're not touching the existing OVMF images, instead, we're providing a separate 4MB image. A guest created with the existing 2MB image will continue to use that unless the user decides to switch that guest to 4MB images (manually). But we're indeed changing the default image in the descriptor interface, so subsequently created guests will use the 4MB image instead. There is a possibility for regression if the live migration is using the descriptor interface as well. Otherwise, I'd expect QEMU to use the same OVMF image for the guest, regardless of the series or the edk2 package version.

We should definitely look into the scenarios you have given. I'll test these in the upcoming week, and *hopefully* see the possible impact in more detail.

[Mustafa Kemal Gilor (mustafakemalgilor)]

Hello Christian,

I wrote a test script to run the migrations you suggested in a reproducible manner, (sorry for taking so long, it took me quite a while to deal with all libvirt/QEMU/OVMF quirks)

Here is the summary:

Migration of default "--boot uefi" guests:

focal -> focal(upgraded) [OK]
focal (upgraded) -> focal [FAIL] (see notes)
bionic -> focal [OK]
bionic -> focal (upgraded) [OK]
focal -> jammy [OK]
focal (upgraded) -> jammy [OK]
bionic -> focal -> jammy [OK]
bionic -> focal (upgraded) -> jammy [OK]

Migration of "--boot loader=OVMF_CODE.fd/OVMF_VARS.fd" guests (simulates created guests before the update):

focal -> focal(upgraded) [OK]
focal (upgraded) -> focal [OK]
bionic -> focal [OK]
bionic -> focal (upgraded) [OK]
focal -> jammy [OK]
focal (upgraded) -> jammy [OK]
bionic -> focal -> jammy [OK]
bionic -> focal (upgraded) -> jammy [OK]

Notes:
The reason why focal (upgraded) -> focal migration fails in the default "--boot uefi" scenario is; the upgraded host creates the guest with the new "_4M" images (because fw descriptors are now pointing to the 4M images), and the non-upgraded focal simply doesn't have them, hence the migration fails. VM stays on the source host, untouched.

You can find the test script and test reports in the following attachment.

[Paride Legovini (paride)]

Stumbled on this in triage, so this is somehow a drive-by comment. The testing you prepared looks solid to me. What I'm surprised about is the lack of comments / extra bug reports from users affected by this issue, especially given that this bug is 2.5 years old. I expect Win10 guests on Ubuntu hosts to be a pretty common scenario. Are we sure that the initial problem (Win10 guests failing to install some upgrades) is still happening? Could it be that this was fixed or worker around on the Windows side by lowering the requirements?

[Mustafa Kemal Gilor (mustafakemalgilor)]

Hello Paride,

Thanks for stopping by! Despite the age of this ticket, the bug surfaced on Windows quite recently given that this is an NVRAM space issue and each OS does different things with NVRAM space. The problem most likely going to surface at different times and during different procedures.

AFAIK the problem has no Windows-side workarounds, and according to the upstream issue (https://github.com/tianocore/edk2/discussions/3221?sort=new) most people stumbled upon the problem just switched to the 4MB images. I guess VM UEFI boot is still a niche thing and not widely adopted (yet?). Unaffected people seem to be using legacy BIOS.(https://forum.proxmox.com/threads/kb5012170-fails-to-install-on-win11-host.113561/)

Tomorrow I'll try to deploy a Windows VM again to see if the issue is still the same or not.

[Paride Legovini (paride)]

Thanks! That's all I wanted to check. You're right in that VM UEFI is still uncommon and that's most likely the reason why we're not seeing more bug reports on this.

[Mustafa Kemal Gilor (mustafakemalgilor)]

Hello Paride,

FYI: Today I re-tested the scenario with the latest Windows 10 ISO available, and the issue still persists.

[Timo Aaltonen (tjaalton)]

Hello Dimitri, or anyone else affected,

Accepted edk2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/edk2/0~20191122.bd85bf54-2ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mustafa Kemal Gilor (mustafakemalgilor)]

Hello,

I've run live migration tests with the proposed package, and the results are OK. Only failing migration test is focal-proposed to focal, which is expected and has valid workarounds. The complete execution log is attached to the comment.

Additionally, I re-run the Test plan and am glad to say the problem is gone. The mentioned Windows update is installed successfully.

[Łukasz Zemczak (sil2100)]

@mustafakemalgilor do the migration tests correlate to the Test Case test mentioned in the bug description?

[Mustafa Kemal Gilor (mustafakemalgilor)]

Hi Łukasz;

Yes, but indirectly. Migration tests are done from the hypervisor's perspective whereas the test case is done from the guest's perspective. The issue test case is only concerned with the guest's perspective, but we needed to make sure that the patch does not affect the live migrations as well (as recommended by Christian), so migration tests verify that.

FTR, I've run both (i.e. the test plan and the migration tests), and they both pass.

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for edk2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package edk2 - 0~20191122.bd85bf54-2ubuntu3.4

---------------
edk2 (0~20191122.bd85bf54-2ubuntu3.4) focal; urgency=medium

  [ dann frazier ]
  * Provide 4MB OVMF images: The existing 2MB images no longer
    have sufficient variable space for the current Secure Boot
    Forbidden Signature Database. (LP: #1885662)
    - Convert targets for pre-enrolled variable template images
      into pattern rules. This will be useful for adding additional
      pre-enrolled variable templates.
    - Update fw descriptors to reference 4M images instead of their
      2M counterparts. This will migrate tools that use the descriptor
      interface (like libvirt) over to the 4M images when creating new
      VMs. Existing 2M VMs will require manual migration.
  * Increase autopkgtest timeout from 30s to 60s. (LP: #1885186)

  [ Mustafa Kemal Gilor ]
  * Added autopkg tests for 4MB OVMF images. (LP: #1885662)

 -- Mustafa Kemal GILOR <email address hidden> Tue, 08 Nov 2022 11:40:07 +0300