QEMU/KVM display is garbled when booting from kernel EFI stub due to missing bochs-drm module

[Matthew Ruffell]

BugLink: https://bugs.launchpad.net/bugs/1872863

[Impact]

A recent grub2 SRU, LP #1864533, now forces the kernel to boot via the kernel EFI stub whenever EFI is enabled. This causes problems for QEMU/KVM virtual machines which use the VGA=std video device, as the efifb driver yields an unreadable garbled screen. See the attached image.

The correct framebuffer driver to use in this situation is bochs-drm, and modprobing it from a HWE kernel fixes the issues.

bochs-drm is missing from Bionic since CONFIG_DRM_BOCHS was disabled in LP #1378648 due to bochs-drm causing problems in a PowerKVM machine. This problem appears to be fixed now, and bochs-drm has been re-enabled for Disco and up, in LP #1795857 and has been proven safe.

This has also come up again in LP #1823152, as well as chatter on LP #1795857 to get this enabled on Bionic.

The customer which is experiencing this issue cannot switch to VGA=qxl as a workaround, and must use VGA=std, hence I suggest we re-enable bochs-drm for Bionic.

[Fix]

I noticed on Focal, if you boot, the framebuffer is initially efifb:

[ 0.603716] efifb: probing for efifb
[ 0.603733] efifb: framebuffer at 0xc0000000, using 1876k, total 1875k
[ 0.603735] efifb: mode is 800x600x32, linelength=3200, pages=1
[ 0.603736] efifb: scrolling: redraw
[ 0.603738] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[ 0.604462] Console: switching to colour frame buffer device 100x37
[ 0.605829] fb0: EFI VGA frame buffer device

This soon changes to bochs-drm about a second later:

[ 0.935988] bochs-drm 0000:00:01.0: remove_conflicting_pci_framebuffers: bar 0: 0xc0000000 -> 0xc0ffffff
[ 0.937023] bochs-drm 0000:00:01.0: remove_conflicting_pci_framebuffers: bar 2: 0xc1c8c000 -> 0xc1c8cfff
[ 0.937776] checking generic (c0000000 1d5000) vs hw (c0000000 1000000)
[ 0.937776] fb0: switching to bochsdrmfb from EFI VGA
[ 0.939085] Console: switching to colour dummy device 80x25
[ 0.939117] bochs-drm 0000:00:01.0: vgaarb: deactivate vga console
[ 0.939210] [drm] Found bochs VGA, ID 0xb0c5.
[ 0.939212] [drm] Framebuffer size 16384 kB @ 0xc0000000, mmio @ 0xc1c8c000.
[ 0.941955] lpc_ich 0000:00:1f.0: I/O space for GPIO uninitialized
[ 0.942069] [TTM] Zone kernel: Available graphics memory: 2006780 KiB
[ 0.942081] [TTM] Initializing pool allocator
[ 0.942089] [TTM] Initializing DMA pool allocator
[ 0.943026] virtio_blk virtio2: [vda] 20971520 512-byte logical blocks (10.7 GB/10.0 GiB)
[ 0.944019] [drm] Found EDID data blob.
[ 0.944162] [drm] Initialized bochs-drm 1.0.0 20130925 for 0000:00:01.0 on minor 0
[ 0.944979] fbcon: bochs-drmdrmfb (fb0) is primary device
[ 0.947712] Console: switching to colour frame buffer device 128x48

On bionic, the framebuffer never changes from efifb, since the bochs-drm kernel module is not built, and it is also present on the module banlist in /etc/modprobe.d/blacklist-framebuffer.conf

bochs-drm needs to be enabled to be built in the kernel config, and removed from the module blacklist in kmod.

[Testcase]

Create a new QEMU/KVM virtual machine, I used virt-manager. Before you install the OS, check the box to modify settings before install. In the "Overview" tab, enable EFI by setting the firmware to "UEFI x86_64: /usr/share/OVMF/OVMF_CODE.secboot.fd".

Set the video device to qxl while you install Bionic. Once the install is done, reboot, do a "apt update" and "apt upgrade", to ensure you have grub2 2.02-2ubuntu8.15 installed.

Shut the VM down, and set the video device to VGA. Or VGA=std if you use the QEMU command line.

Start the VM up, and the screen will be garbled. See attached picture.

If you install my test packages, which are available here:

https://launchpad.net/~mruffell/+archive/ubuntu/sf272653-test

Instructions to install (on a bionic system):

1) Enable bionic-proposed
2) sudo apt-get update
3) sudo apt install linux-image-unsigned-4.15.0-96-generic linux-modules-4.15.0-96-generic linux-modules-extra-4.15.0-96-generic linux-headers-4.15.0-96-generic linux-headers-4.15.0-96 libkmod2 kmod
4) sudo reboot
5) uname -rv
4.15.0-96-generic #97+TEST272653v20200409b1-Ubuntu SMP Thu Apr 9 04:09:18 UTC 2020

If you reboot, the screen will be perfectly readable, since the bochs-drm driver will be in use.

[Regression Potential]

We are enabling a display driver on Bionic which was previously disabled, so there will be some virtual machines making the jump to using bochs-drm since it would now be available.

I don't think that this is a bad thing, and in many ways will probably end up improving some user's experience. Although there are two scenarios where a regression could happen:

1 - frequently gamers use GPU passthrough to their kvm based virtual machines, and I have seen them expect, or hardcode their kernel command lines to disable the efifb driver. See https://passthroughpo.st/explaining-csm-efifboff-setting-boot-gpu-manually/ Worst case scenario, gamers will also need to disable bochs-drm as well. Since the PCI device of the GPU is being passed through to the VM, I believe the kernel will make the correct choice and load the GPU drivers, like noverau or nvidia-drm instead.

2 - if the bug with PowerKVM machines isn't actually fixed in Bionic, it will cause a regression on POWER. We just need to test this on POWER and ensure that the bug in LP #1378648 was fixed. If not, then we could just disable bochs-drm on POWER, leaving it enabled for all other archs.

Note: 2 has now been tested and verified working. See comment #7.

If a regression were to occur, it would only affect KVM machines that use the built in SPICE graphics display. The screen would either not show at all or be garbled. bochs-drm is limited only to kvm users, and does not affect other virtualisation platforms like VMware or Virtualbox. Instances will still be accessible via ssh, and work normally.

[Matthew Ruffell (mruffell)]

Attached is a picture of the garbled screen when VGA=std under booting via kernel EFI stub.

[Matthew Ruffell (mruffell)]

Submitted Kernel patches to mailing list:

https://lists.ubuntu.com/archives/kernel-team/2020-April/109027.html
https://lists.ubuntu.com/archives/kernel-team/2020-April/109028.html

[Matthew Ruffell (mruffell)]

Attached is the debdiff for kmod, which removes bochs-drm from the framebuffer blacklist.

[Matthew Ruffell (mruffell)]

I rebuilt the kmod and Linux packages to include the ppc64le and powerpc architectures, so we can test against POWER to ensure that LP #1378648 is no more, and we don't cause any regressions.

I don't have access to any POWER machines though, so I need help testing.

[Matthew Ruffell (mruffell)]

Attached is screenshot proof that bochs-drm functions correctly on 4.15.0-96-generic ppc64el, and does not cause any regressions when bochs-drm is enabled on ppc64el.

[Matthew Ruffell (mruffell)]

Attached is proof the VM is using VGA=std for the ppc64el test.

[Matthew Ruffell (mruffell)]

I have verified that enabling the bochs-drm kernel module on bionic does not cause any regressions for PowerKVM machines when the video device is VGA=std.

TLDR: See Screenshot: https://launchpadlibrarian.net/475658598/Screenshot%20from%202020-04-22%2015-14-13.png

This is in response to the main cause of regression concern, LP #1378648, which got bochs-drm disabled in the first place, on yakkety.

On Christian's advice, I created a ppc64el instance on Canonistack, with the m1.medium flavor and c8c120bd-3d55-4e2d-8cc5-93039a62524f image (ubuntu/ubuntu-bionic-18.04-ppc64el-server-20200407-disk1.img).

From there, I installed the KVM packages and a xfce4 desktop, and got xrdp working. I then installed virt-manager.

I modprobed the kvm_pr kernel module on my instance to enable nested kvm on ppc64el.

From there, I installed Ubuntu 18.04.4 Server ppc64el, using virt-manager. The "Machine Type" was pseries-2.12, and the Video Model was set to "VGA" [1].

[1]: https://launchpadlibrarian.net/475658622/Screenshot%20from%202020-04-22%2015-14-04.png

Upon booting, the system was using the Open Firmware frame buffer:

[ 0.888072] Using unsupported 800x600 vga at 200081000000, depth=32, pitch=3200
[ 0.890480] Console: switching to colour frame buffer device 100x37
[ 0.892181] fb0: Open Firmware frame buffer device on /pci@800000020000000/vga@6

$ cat /proc/fb
0 OFfb vga

Which is what LP #1378648 wanted.

I then enabled my ppa where my test packages were built:

https://launchpad.net/~mruffell/+archive/ubuntu/sf272653-test

and installed new kmod and 4.15.0-96-generic test builds with bochs-drm enabled in the kernel config, and removed from the kmod blacklist.

I then rebooted, and found that system boots successfully, and does not get hung up at the bug in LP #1378648, indicating that it has been fixed before 4.15.

I checked "lsmod" and bochs-drm has been loaded, and looking at dmesg, initially the Open Firmware frame buffer device is used, and later in the boot, bochs-drm is loaded, and takes over the frame buffer.

See screenshot [2] for proof that bochs-drm is running and functions correctly in virt-manager.

[2]: https://launchpadlibrarian.net/475658598/Screenshot%20from%202020-04-22%2015-14-13.png

A complete dmesg from this boot can be found here [3].

[3] https://paste.ubuntu.com/p/Q8V7fKnRzz/

Interesting parts:

[ 0.893046] Using unsupported 800x600 vga at 200081000000, depth=32, pitch=3200
[ 0.895482] Console: switching to colour frame buffer device 100x37
[ 0.897270] fb0: Open Firmware frame buffer device on /pci@800000020000000/vga@6
...
[ 2.041835] checking generic (200081000000 1d4c00) vs hw (200081000000 1000000)
[ 2.041836] fb: switching to bochsdrmfb from OFfb vga
[ 2.042568] Console: switching to colour dummy device 80x25
[ 2.045581] [drm] Found bochs VGA, ID 0xb0c5.
[ 2.046100] [drm] Framebuffer size 16384 kB @ 0x200081000000, mmio @ 0x200082000000.
[ 2.058141] [TTM] Zone kernel: Available graphics memory: 504512 kiB
[ 2.058883] [TTM] Initializing pool allocator
[ 2.059372] [TTM] Initializing DMA pool allocator
[ 2.074890] Console: switching to colour frame buffer device 128x48
[ 2.08...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Matthew Ruffell (mruffell)]

For both tests I used the kmod package from my test ppa, since it is still
in the unapproved upload queue.

Firstly, I installed 4.15.0-99-generic from -updates, and booted my EFI based
KVM machine with VGA=std. The screen was garbled, as the framebuffer used was
efifb.

I then installed 4.15.0-100-generic from -proposed, and rebooted, keeping the
same settings. The screen was perfect and readable. The issue was fixed. I looked
at lsmod, and bochs-drm was loaded.

$ lsmod | grep bochs_drm
bochs_drm 20480 1
ttm 106496 1 bochs_drm
drm_kms_helper 172032 1 bochs_drm
drm 401408 4 drm_kms_helper,bochs_drm,ttm

Dmesg shows:

[ 1.082068] checking generic (c0000000 12c000) vs hw (c0000000 1000000)
[ 1.082069] fb: switching to bochsdrmfb from EFI VGA
[ 1.082468] Console: switching to colour dummy device 80x25
[ 1.083148] [drm] Found bochs VGA, ID 0xb0c5.
[ 1.083151] [drm] Framebuffer size 16384 kB @ 0xc0000000, mmio @ 0xc1c8c000.
[ 1.083386] [TTM] Zone kernel: Available graphics memory: 4026194 kiB
[ 1.083389] [TTM] Zone dma32: Available graphics memory: 2097152 kiB
[ 1.083390] [TTM] Initializing pool allocator
[ 1.083394] [TTM] Initializing DMA pool allocator
[ 1.085265] fbcon: bochsdrmfb (fb0) is primary device
[ 1.087421] Console: switching to colour frame buffer device 128x48
[ 1.091589] bochs-drm 0000:00:01.0: fb0: bochsdrmfb frame buffer device
[ 1.108664] [drm] Initialized bochs-drm 1.0.0 20130925 for 0000:00:01.0 on minor 0

Attached is a screenshot showing a good screen. I am happy to mark this as
verified for the linux package.

[Łukasz Zemczak (sil2100)]

Hello Matthew, or anyone else affected,

Accepted kmod into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/kmod/24-1ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted kmod (24-1ubuntu3.4) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

chrony/3.2-4ubuntu4.4 (armhf)
linux-hwe-5.0/5.0.0-48.52~18.04.1 (arm64)
linux-aws-5.3/5.3.0-1018.20~18.04.1 (arm64)
linux/4.15.0-100.101 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#kmod

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Matthew Ruffell (mruffell)]

I repeated the same SRU verification as before, this time using kmod from -proposed.

I installed a fresh Bionic KVM, with EFI enabled. I upgraded the system to use the latest -updates packages. Linux 4.15.0-99-generic and Kmod 24-1ubuntu3.3 were installed. I shut the machine down, set the VGA device to vga=std, and rebooted. The screen was garbled and unreadable.

I then ssh'd in, enabled -proposed, and installed Linux 4.15.0-100-generic, and kmod 24-1ubuntu3.4. I rebooted, and the display came up great, and was perfectly readable. Attached screenshot as proof.

Relevent details from dmesg:

[ 1.137544] checking generic (c0000000 1d5000) vs hw (c0000000 1000000)
[ 1.137545] fb: switching to bochsdrmfb from EFI VGA
[ 1.137918] Console: switching to colour dummy device 80x25
[ 1.140305] [drm] Found bochs VGA, ID 0xb0c5.
[ 1.140308] [drm] Framebuffer size 16384 kB @ 0xc0000000, mmio @ 0xc1c8c000.
[ 1.140616] [TTM] Zone kernel: Available graphics memory: 2011058 kiB
[ 1.140619] [TTM] Initializing pool allocator
[ 1.140622] [TTM] Initializing DMA pool allocator
[ 1.146542] fbcon: bochsdrmfb (fb0) is primary device
[ 1.149357] Console: switching to colour frame buffer device 128x48
[ 1.151830] bochs-drm 0000:00:01.0: fb0: bochsdrmfb frame buffer device
[ 1.168977] [drm] Initialized bochs-drm 1.0.0 20130925 for 0000:00:01.0 on minor 0

lsmod:

$ lsmod | grep bochs
bochs_drm 20480 1
ttm 106496 1 bochs_drm
drm_kms_helper 172032 1 bochs_drm
drm 401408 4 drm_kms_helper,bochs_drm,ttm

Since the Linux package builds bochs-drm, and the kmod package removes the blacklist, I am happy to mark this verified for both packages.

[Matthew Ruffell (mruffell)]

I went and looked at the autopkgtest regressions, and they aren't actual regressions, just transient network problems and slow build timeouts.

chrony autopkgtest regression on armhf:

Need to get 263 kB of archives.
After this operation, 2,048 B of additional disk space will be used.
Err:1 http://ftpmaster.internal/ubuntu bionic-proposed/main armhf kmod armhf 24-1ubuntu3.4
  Could not connect to ftpmaster.internal:80 (91.189.89.99), connection timed out
Err:2 http://ftpmaster.internal/ubuntu bionic-proposed/main armhf libkmod2 armhf 24-1ubuntu3.4
  Unable to connect to ftpmaster.internal:http:

Network problems for this one.

All the linux packages for arm64:

autopkgtest [15:44:38]: ERROR: timed out on command /tmp/autopkgtest.M7KUyX/build.f2S/src/debian/tests/rebuild
autopkgtest [15:48:48]: test rebuild: -----------------------]
rebuild FAIL timed out

Slow builds for these packages. Maybe the arm64 build host was having a bad day.

I think the autopkgtests just need to be re-run.

[Richard Laager (rlaager)]

I have confirmed that the fix in -proposed fixes the issue for me.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-101.102

---------------
linux (4.15.0-101.102) bionic; urgency=medium

  * bionic/linux: 4.15.0-101.102 -proposed tracker (LP: #1877262)

  * 4.15.0-100.101 breaks userspace builds due to a bug in the headers
    /usr/include/linux/swab.h of linux-libc-dev (LP: #1877123)
    - include/uapi/linux/swab.h: fix userspace breakage, use __BITS_PER_LONG for
      swap

  * bionic snapdragon 4.15 snap failed Certification testing (LP: #1877657)
    - Revert "drm/msm: Use the correct dma_sync calls in msm_gem"
    - Revert "drm/msm: stop abusing dma_map/unmap for cache"

linux (4.15.0-100.101) bionic; urgency=medium

  * bionic/linux: 4.15.0-100.101 -proposed tracker (LP: #1875878)

  * built-using constraints preventing uploads (LP: #1875601)
    - temporarily drop Built-Using data

  * Add debian/rules targets to compile/run kernel selftests (LP: #1874286)
    - [Packaging] add support to compile/run selftests

  * getitimer returns it_value=0 erroneously (LP: #1349028)
    - [Config] CONTEXT_TRACKING_FORCE policy should be unset

  * QEMU/KVM display is garbled when booting from kernel EFI stub due to missing
    bochs-drm module (LP: #1872863)
    - [Config] Enable CONFIG_DRM_BOCHS as module for all archs

  * Backport MPLS patches from 5.3 to 4.15 (LP: #1851446)
    - net/mlx5e: Report netdevice MPLS features
    - net: vlan: Inherit MPLS features from parent device
    - net: bonding: Inherit MPLS features from slave devices
    - net/mlx5e: Move to HW checksumming advertising

  * LIO hanging in iscsit_free_session and iscsit_stop_session (LP: #1871688)
    - scsi: target: remove boilerplate code
    - scsi: target: fix hang when multiple threads try to destroy the same iscsi
      session
    - scsi: target: iscsi: calling iscsit_stop_session() inside
      iscsit_close_session() has no effect

  * Add hw timestamps to received skbs in peak_canfd (LP: #1874124)
    - can: peak_canfd: provide hw timestamps in rx skbs

  * Bionic update: upstream stable patchset 2020-04-23 (LP: #1874502)
    - ARM: dts: sun8i-a83t-tbs-a711: HM5065 doesn't like such a high voltage
    - bus: sunxi-rsb: Return correct data when mixing 16-bit and 8-bit reads
    - net: vxge: fix wrong __VA_ARGS__ usage
    - hinic: fix a bug of waitting for IO stopped
    - hinic: fix wrong para of wait_for_completion_timeout
    - cxgb4/ptp: pass the sign of offset delta in FW CMD
    - qlcnic: Fix bad kzalloc null test
    - i2c: st: fix missing struct parameter description
    - firmware: arm_sdei: fix double-lock on hibernate with shared events
    - null_blk: Fix the null_add_dev() error path
    - null_blk: Handle null_add_dev() failures properly
    - null_blk: fix spurious IO errors after failed past-wp access
    - xhci: bail out early if driver can't accress host in resume
    - x86: Don't let pgprot_modify() change the page encryption bit
    - block: keep bdi->io_pages in sync with max_sectors_kb for stacked devices
    - irqchip/versatile-fpga: Handle chained IRQs properly
    - sched: Avoid scale real weight down to zero
    - selftests/x86/ptrace_syscall_32: Fix no-vDSO segfault
    - PCI/switchtec: Fix init_completio...

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for kmod has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package kmod - 24-1ubuntu3.4

---------------
kmod (24-1ubuntu3.4) bionic; urgency=medium

  * Remove bochs-drm from d/modprobe.d/blacklist-framebuffer.conf
    to fix garbled screen with EFI based QEMU/KVM virtual machines
    which use VGA=std video devices. (LP: #1872863)

 -- Matthew Ruffell <email address hidden> Thu, 09 Apr 2020 16:03:35 +1200