5.15 and 5.16 arm64 kernels fail to boot in VMWare Fusion

[Paul Jolly]

I am using VMWare Fusion Tech Preview on an Apple M1 Max MacBook Pro per:

https://blogs.vmware.com/teamfusion/2021/09/fusion-for-m1-public-tech-preview-now-available.html

I am successfully using the 5.14 kernel series (currently 5.14.21-051421-generic) but note this is now EOL.

However whenever I try to upgrade to either the 5.15 or 5.16 series my VM hangs at boot time after:

EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
EFI stub: ERROR: FIRMWARE BUG: Image BSS overlaps adjacent EFI memory region
EFI stub: Using DIB from configuration table
EFI stub: Exiting boot services...

I see the same "hang" for both 5.15.7 and 5.16-rc4.

FYI this has also been reported by others who note that the problem appears to be specific to Ubuntu:

https://communities.vmware.com/t5/Fusion-for-Apple-Silicon-Tech/Anyone-have-any-luck-with-Ubuntu-5-15-kernels/m-p/2880362

Given that I can't boot using the 5.15.7 and 5.16-rc4 kernels, I can't provide the output from:

cat /proc/version

However I attach the output from lspci.

Please let me know what other information I can provide to help diagnose this issue.

Thanks
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu74
Architecture: arm64
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2021-11-12 (26 days ago)
InstallationMedia: Ubuntu-Server 21.10 "Impish Indri" - Release arm64 (20211013)
Package: linux-image-generic 5.13.0.21.32 [origin: unknown]
PackageArchitecture: arm64
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
Tags: jammy uec-images third-party-packages
Uname: Linux 5.14.21-051421-generic aarch64
UnreportableReason: This does not seem to be an official Ubuntu package. Please retry after updating the indexes of available packages, if that does not work then remove related third party packages and try again.
UpgradeStatus: Upgraded to jammy on 2021-11-15 (23 days ago)
UserGroups: N/A
_MarkForUpload: True
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw----+ 1 root audio 116, 1 Dec 17 11:38 seq
 crw-rw----+ 1 root audio 116, 33 Dec 17 11:38 timer
AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.13.0-21-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu74
Architecture: arm64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/seq: myitcv 1543 F.... pipewire
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2021-11-12 (34 days ago)
InstallationMedia: Ubuntu-Server 21.10 "Impish Indri" - Release arm64 (20211013)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: VMware, Inc. VBSA
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.13.0-21-generic root=UUID=7378d8ea-883e-47b8-b5ab-a92790860f9c ro
ProcVersionSignature: Ubuntu 5.13.0-21.21-generic 5.13.18
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-5.13.0-21-generic N/A
 linux-backports-modules-5.13.0-21-generic N/A
 linux-firmware 1.202
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 5.13.0-21-generic aarch64
UpgradeStatus: Upgraded to jammy on 2021-11-15 (31 days ago)
UserGroups: N/A
_MarkForUpload: True
acpidump:

dmi.bios.date: 12/31/2020
dmi.bios.release: 0.0
dmi.bios.vendor: VMware
dmi.bios.version: VEFI
dmi.board.name: VBSA
dmi.board.vendor: VMware
dmi.board.version: 1
dmi.chassis.type: 1
dmi.chassis.vendor: VMware
dmi.chassis.version: VBSA
dmi.ec.firmware.release: 0.0
dmi.modalias: dmi:bvnVMware:bvrVEFI:bd12/31/2020:br0.0:efr0.0:svnVMware,Inc.:pnVBSA:pvr1:rvnVMware:rnVBSA:rvr1:cvnVMware:ct1:cvrVBSA:sku0000000000000001:
dmi.product.family: VMware
dmi.product.name: VBSA
dmi.product.sku: 0000000000000001
dmi.product.version: 1
dmi.sys.vendor: VMware, Inc.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1953671

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Paul Jolly (myitcv)]

Per my comments in the issue description, I can't run apport-collect because I can't boot the 5.15 or 5.16 kernels. And when I try to run it for 5.14.21 it tells me I'm running a mainline kernel, despite having installed the debs from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.14.21/.

Please let me know if there is any other information I can usefully provide however.

[Chris Guiver (guiverc)]

Without full package/release details, this bug is not likely to get much attention until details have been provided (using apport tools).

Please run the apport-collect using the last kernel that booted; and just note that it was run with the working kernel, and not the kernel the bug report was about.

No Ubuntu release uses the 5.14 or later kernels currently; they are currently still in testing only; and no product details have yet been provided (being populated when ubuntu-bug or apport-collect are run).

[Kai-Heng Feng (kaihengfeng)]

Can you please do a kernel bisection?

[Paul Jolly (myitcv)]

@Chris - understood. Per my most recent message, running apport-collect triggers the following warning:

"It appears you are currently running a mainline kernel. It would be better to report this bug upstream at http://bugzilla.kernel.org/ so that the upstream kernel developers are aware of the issue. If you'd still like to file a bug against the Ubuntu kernel, please boot with an official Ubuntu kernel and re-file."

Excuse my noob status, but what should I therefore do?

I honestly thought that by running a kernel installed from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.14.21/ I would be running an Ubuntu kernel.

Thanks

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

I've tried to provide some information via:

$ sudo apport-collect -p linux-image-generic 1953671

If there is a better command I can run, please let me know.

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

apport information

[Paul Jolly (myitcv)]

I've just updated with the logs from:

apport-collect 1953671

Please note: these are gathered when using the 5.13.0-21-generic kernel.

However, VMWare Fusion requires at least 5.14, so I have been running v5.14.21-051421-generic day-to-day, and only switched to 5.13.0-21 in order to report these diagnostics.

Both v5.15.8 and v5.16-rc5 both hang, so v5.14.21-051421 remains the only kernel I can use at the moment.

[Paul Jolly (myitcv)]

This problem persists with both v5.15.11 and v5.16-rc6

[Michael Roy (mikeroysoft)]

Hi there, VMware here...
We're able to repro this and have the following temporary workaround while we figure out who's code needs adjusting:

We are able to boot by adding:
acpi=force

to: GRUB_CMDLINE_LINUX_DEFAULT
in: /etc/default/grub
Such that the full line reads:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash acpi=force"

Remember to also disable Wayland (which is not presently supported in this Fusion tech preview) in /etc/gdm3/custom.conf by uncommenting WaylandEnable=false.

When it's in the unresponsive state it still boots so ssh works... you can check /proc/iomem and it seems like the PCI memory is already reserved for BOOTFB..

Not working state:
50000000-7fffffff : pcie@0x40000000
  78000000-7fffffff : 0000:00:0f.0
    78000000-782fffff : BOOTFB

Working state: (i.e. with acpi=force)

50000000-7fffffff : PCI Bus 0000:00
  50000000-501fffff : PCI Bus 0000:20
  50200000-503fffff : PCI Bus 0000:21
  50400000-505fffff : PCI Bus 0000:21
  78000000-782fffff : BOOTFB

We have an internal bug filed so I'll update if we have more we can share.

Thanks,
-Michael

[Kai-Heng Feng (kaihengfeng)]

Michael,

Is it possible to perform git bisect to find the offending commit?

[Kai-Heng Feng (kaihengfeng)]

This can be the efifb regression Linus also encountered.
FWIW, the initial efifb runtime PM grabbing is written by me.

[Anders Östling (anders-ostling)]

Any news on this one? I can boot the Ubuntu Jammy desktop using acpi=force and install the desktop.
However, on subsequent boots it hangs with the same error in spite of the changed grub config.

Also, shutdown does not work at all. Ubuntu just hangs silently.

This is on VMware Fusion (latest tech preview) on a Mac Mini 2020 M1.

[Terry Simons (terry-simons)]

I'm not able to boot with the workaround provided in the final release Jammy 22.04 or latest daily Ubuntu Jammy 22.04 ISO using VMWare Technology Preview "Professional Version e.x.p (19431034)."

I've edited the grub config in the Ubuntu installer to include the mentioned acpi=force option, but it doesn't help.

All I get is the previously mentioned issue (but only part of it):

Booting a command list

EFI stub: Error: FIRMWARE BUG: Kernel not aligned on a 64K boundary
EFI stub: Error: FIRMWARE BUG: Image BSS overlaps adjacent EFI memory region

System is a 64GB 2020 MacBook Pro M1 Max.

Any news VMWare?

[Alex Henrie (alexhenrie24)]

I also had this problem with VMWare and a 5.15.12 kernel, although technically the kernel did boot because I could get kernel output and log in over a serial console, it's just that VMWare showed a black screen. The workaround described at https://www.reddit.com/r/VFIO/comments/uud5sx/comment/in5uqtr/ worked for me: I added initcall_blacklist=sysfb_init to the kernel command line (GRUB_CMDLINE_LINUX_DEFAULT) and now it boots to the GUI without any trouble.