Virtualbox Guru meditation on VM start caused by kernel commit in v6.9-rc4

[Dave Odell]

BugLink: https://bugs.launchpad.net/bugs/2073267

SRU Justification:

[Impact]

Commit "randomize_kstack: Improve entropy diffusion" changed the
kernel stack for entropy to 1KiB, limiting the thread kernel stack to
15Kib. This impacts virtualbox 6.1.50 on jammy, that is no longer
maintained upstream. The issue does not persist on version 7.0.20 due to a
code refactoring that probably resulted in less stack usage. Fixing it on
the jammy virtualbox package side is not straightfoward because the fix is
not easy to backport to 6.x and upgrading the jammy package to 7.x breaks
current users machines that run Windows, but not only.
Users need to uninstall the Guest additions drivers, migrate the
virtualbox package to 7.x, boot each VMs and install the Guest additions
drivers in each VM.

This impacts:
1. jammy:linux
2. jammy:linux-hwe-6.8
3. focal:linux-hwe-5.15

[Fix]

Revert commit "randomize_kstack: Improve entropy diffusion"

[Test Plan]

Without this fix, a VM would crash, showing with "VCPU0: Guru
Meditation -2708 (VERR_VMM_SET_JMP_ABORTED_RESUME)".
After the kernel upgrade, all VMs should run with no problem.

[Where problems could occur]
This may have an impact on security. The commit is a fix to improve the
stack entropy.

Original description:

It worked yesterday, but today I get a Guru Meditation trying to start some of my virtual machines. This shows up in VBox.log as "VCPU0: Guru Meditation -2708 (VERR_VMM_SET_JMP_ABORTED_RESUME)". I suspect this may have started due to a Linux kernel upgrade I installed this morning.

A fresh VM with no disk shows the issue. Sometimes turning off the I/O APIC makes the issue go away, sometimes not. Turning off nested paging sometimes lets VirtualBox make a little bit of progress w.r.t. booting VMs, but that usually still crashes before the VM finishes starting.

This may be related to this bug reported on the VirtualBox forums: https://forums.virtualbox.org/viewtopic.php?t=111889&sid=5cd33c0872a03b689e7e9f84d850f538

https://forums.virtualbox.org/viewtopic.php?t=111918

Ubuntu is 22.04.4 LTS, kernel is 5.15.0-116-generic, VirtualBox is 6.1.50-dfsg-1~ubuntu1.22.04.1.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ange des Ténèbres (angedestenebresk)]

Hey,

I have exactly the same issue on Linux Mint 21.3 (based on Ubuntu 22.04) with the kernel 5.15.0-116 and Virtualbox installed from Ubuntu repo (version 6.1.50-dfsg-1~ubuntu1.22.04.1).

After trying to go back to previous kernel version 5.15.0-113, all my VM worked again. So, it seems the last kernel version for 5.15 causes that.

[Arrigo Marchiori (ardovm)]

Same problem here, on several machines, all Ubuntu 22.04 after updating the kernel to version 5.15.0-116.

All hosts and guests are x86_64.

[berni123 (bernd-bernd--zimmermann)]

I can confirm this bug an add the following information:

Kernel 5.15.0-116-generic: VBox 6.1.50 is not working at all

Kernel 5.15.0-112-generic and 5.15.0-113-generic: VBox 6.1.50 is working with boot workaround amg_iommu=off due to the bug in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2068738

Kernel 5.15.0-107-generic: VBox 6.1.50 is working without problems

It looks like, that another kernel patch between -112/-113 and -116 causes the issue with VBox 6.1.50

Regards,
Bernd

[Andy (montagu)]

Here is a ticket for the same issue on virtualbox.org, for what it's worth:
https://www.virtualbox.org/ticket/22115
But support for VirtualBox 6.1 ended in January this year so we can't expect any fix to be forthcoming from Oracle.

[Xavier Tannier (xtannier)]

I have the same problem, solved by uninstalling the version from Ubuntu repo, and installing instead the version 7 from virtualbox web site:
https://www.virtualbox.org/wiki/Linux_Downloads

[Pudding (little-pudding)]

Same story for me with Linux Mint 21.3 Cinnamon and VirtualBox 6.1 on Thinkpad T495 (AMD Ryzen):

5.15.0-116 fails

5.15.0-113 fails (doesn't even manage to boot Linux Mint, why did they push this to updates?!)

5.15.0-112 fails (manages to get a line of text out on the console before the VM crashes)

5.15.0-107 works perfectly fine

[Alan Knepper (al-knepper)]

I've had all sorts of issues with Kernels after -107. First my keyboard was unresponsive after a sleep/suspend. Then it was not being able to suspend (it would just immediately wake).

Adding entries in grub fixed the first issue for while and got fixed about 112 or 113. Now they toss this issue in.

I've been trying quite a few other operating systems, installing them to a spare 40G chunk of HD. I'm surprised at how many have my original suspend/keyboard issues. I've never made records as to which Kernels are involved, sure might have been useful huh?!!

I have also discovered that Kernel 113 or 6.5x fix Virtualbox. I found the partial fix by booting Zorin OS (my current testing OS on that 40G hd) and found that all of my VM images worked in that OS. I was glad to find out that all the vdi files were still valid.

I think I'm going to wait for the dust to settle, as I can always backup to Kernel 113 as long as I don't do a autoremove command.

[William Veliz (wilardo)]

I have the same error with Linux Mint 21.2 Cinnamon with kernel 5.15.0-116-generic and VirtualBox installed from Ubuntu repo (version 6.1.50-dfsg-1~ubuntu1.22.04.1).

[Alan Knepper (al-knepper)]

Upgrading to virtualbox 7.0.20 fixed my system. I'm back on 5.15.0-116 kernel.

[Andre Juanel (ajuanel)]

Same issue here with Ubuntu Mate 22.04. It worked with kernel 5.15.0-112.

The log file says:
ERROR [COM]: aRC=NS_ERROR_INVALID_ARG (0x80070057) aIID={4680b2de-8690-11e9-b83d-5719e53cf1de} aComponent={DisplayWrap} aText={Argument aWidth is invalid (must be aWidth != 0 && aWidth <= 32767)}, preserve=false aResultDetail=0

Waiting for a kernel fix.

[Michael Plotkin (plotkin)]

Just tested the "focal-proposed" kernel version 5.15.0-118 (with "Pre-released updates" enabled) on Ubuntu 20.04 with VBox 6.1.50. Unfortunately the VM didn't work at all with exactly the same symptoms as with "stable" kernel version 5.15.0-116. Was unable to find any workaround so I had to downgrade the kernel version to 5.15.0-113 and to disable further kernel upgrades until a working fix is available.

[Schwobi (hemaha)]

Same issue on Ubuntu 20.04 (kernel version 5.15.0-116-generic; virtualbox version 6.1.50).

Considering the Linux world was hit with a few major security flaws in the last few months, I strongly advice against the use of older kernel versions.

I am not sure whether this should be fixed in kernel development, as it is the VirtualBox application that is out of date for all but the latest LTS releases of Ubuntu. What would be the use in changing the kernel if the application itself is outdated and needs an overdue update ASAP ..?
VirtualBox version 6.# had reached end of life by January 2024; it is no longer maintained by Oracle. And VirtualBox version 7.#, as provided by source (Oracle), appears to work with current linux kernels according to reports in the user forum: https://forums.virtualbox.org/viewtopic.php?t=111925

Thus, I add my voice to the chorus of pleas, requesting to finally switch to VirtualBox version 7.# for all Ubuntu releases (at least 20.04 to 22.04 LTS).

Edit: I suspect Ubuntu 24.04 to be unaffected, as it is running VirtualBox version 7.0.# and Linux kernel version 6.8.#. Can someone confirm that?

[Vasya Pupkin (shadowlmd)]

> finally switch to VirtualBox version 7

While it's totally fine, I do not agree that it should be a fix for this problem. Kernel upgrades within the same minor version should not break existing software. This is clearly a bug (major change) in the kernel package and this is where it should be addressed.

[Andy (montagu)]

At the moment it's unclear if anyone who can do anything about this is even aware of it. On any list of bugs I look at, this one is buried under hundreds of others that are sometimes a decade or more old. I'm not sure how anyone would become aware of it.

Reading about bug handling processes on wiki.ubuntu.com, I found this:

----
In order to set bug statuses to 'Triaged' you need to be a member of the Ubuntu Bug Control team. If you are not a Bug Control member, and you find a bug you believe is ready to be marked as Triaged, then please join the #ubuntu-bugs channel on irc.freenode.net and provide a link to the report. A Bug Control member can then examine the report and, if it is ready, mark it as Triaged for you.
----

If anyone affected is on IRC, perhaps it would be worth going onto that channel and asking if this bug report is ready to be triaged.

[Robie Basak (racb)]

Somebody appeared on IRC asking for help with this bug but disappeared again before I could reply.

It's clear that there is an issue, but I don't see enough information here to isolate it to specific versions of packages that Ubuntu shipped. People seem to be reporting versions either not published by Ubuntu or not reporting complete package names and versions that Ubuntu did ship, and posts from people not using Ubuntu (Mint is not Ubuntu and makes its own changes in ways that Ubuntu developers cannot support) also do not help Ubuntu developers isolate affected versions.

Of all the "me too" reports in the comments above, I can't find a single report that is not using software from a third party source that states both the package name and package versions of the kernel and virtualbox that no longer work and previous package name and package versions of the kernel and virtualbox that do work.

If you'd like to help, I suggest reporting precisely which package names and versions are affected on an Ubuntu system with no software from third party sources installed, as well as confirming that reverting the kernel does work around the issue and the package names and versions that you used to do that. The entire package version is important - cutting off half of it removes valuable information.

Note that I'm not a kernel developer for Ubuntu and am not in a position to fix the bug. I'm just trying to help gather information that would be useful to a kernel developer.

[Vasya Pupkin (shadowlmd)]

The problem appeared in the package linux-image-5.15.0-116-generic in all Ubuntu releases that ship VirtualBox version 6.x, particular version of VirtualBox package doesn't matter. Rolling back to linux-image-5.15.0-113-generic fixes the issue.

[Vasya Pupkin (shadowlmd)]

Also, I reported this bug: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.15/+bug/2073691 against specific package version and no third party software.

[Schwobi (hemaha)]

In support of Vasya's observation, the issue appeared for me on Ubuntu 20.04.6 LTS on Ubuntu-shipped VirtualBox
package: virtualbox (version: 6.1.50-dfsg-1~ubuntu1.20.04.1)
following a kernel update from
[working kernel] package: linux-image-5.15.0-113-generic (version: 5.15.0-113.123~20.04.1)
to
[affected kernel] package: linux-image-5.15.0-116-generic (version: 5.15.0-116.126~20.04.1).

Reverting back to the previous kernel version ( 5.15.0-113.123~20.04.1) does indeed make the error go away.

> Quoting Robie: "posts from people not using Ubuntu (Mint is not Ubuntu and makes its own changes in ways that Ubuntu developers cannot support)"

Please note that Linux Mint users are affected because their packages are built from Ubuntu repositories. The same linux kernel update affected their VirtualBox version 6.x. See: https://forums.linuxmint.com/viewtopic.php?t=424497
Thus, a fix in the Ubuntu-shipped kernel will also solve the problem for Linux Mint users.

> Quoting Robie: "not using software from a third party source"

The discussed installation of the newer version 7.# of VirtualBox from Oracle (i.e. source code provider for Ubuntu-shipped packages) is a workaround for those who are not willing to revert back to linux-image-5.15.0-113-generic and blacklist newer kernel versions until a fix in kernel is available.

[Schwobi (hemaha)]

Specified the affected package / kernel branch for clarity.

[Andy (montagu)]

I have the issue on Kubuntu 22.04 LTS.

VirtualBox package: virtualbox
version: 6.1.50-dfsg-1~ubuntu1.22.04.1 (jammy-updates)

Kernel package: linux-image-5.15.0-116-generic
version: 5.15.0-116.126 (jammy-updates)

The above combination has the problem. VirtualBox cannot start VMs. This started when the update to the -116 kernel dropped last week.

Reverting to the kernel package below makes VirtualBox work again.

Kernel package: linux-image-5.15.0-113-generic
version: 5.15.0-113.123 (jammy-updates)

Thanks for coming to help, Robie. If there's any other information we should provide, please tell us.

[David Streicher (davidstreicher123)]

For what its worth:

I have the same issue with Ubuntu 22.04.4 LTS (Desktop).

Working kernel (i.e. VM in Virtualbox 6.1 boots): Linux 5.15.0-1061-gke
Broken kernel (i.e. VM no longer boots): Linux 5.15.0-1062-gke

Workaround: Installed Virtualbox 7 from the Oracle PPA repository and migrated VM with some teething problems.

[Kevin Brubeck Unhammer (unhammer)]

Upgrading to virtualbox 7 let the vm start booting, but windows crashes on boot with a message about an issue with the driver vboxwddm.sys , I'm guessing because guest additions installed from ubuntu package conflict with those from the ppa? So that workaround doesn't solve all cases either.

[David Streicher (davidstreicher123)]

@Kevin you need to remove (uninstall) the guest additions from the VM before upgrading to virtualbox 7.
(This is true when migrating any VM between major virtualbox versions - even if both work correctly with the same kernel version on the Host).

Shortly:

1. Boot Windows Guest VM with Virtualbox 6.1 (and an older Linux kernel to get around the original issue).

2. Remove (uninstall) the Guest additions. (As a sanity check, it may be worth ensuring that your guest VM can boot without the Guest additions installed; if not, adjust the client VM config as required).

3. Uninstall virtualbox 6.1 and the virtualbox 6.1 extension pack

4. Install virtualbox 7 (and the virutalbox 7 extension pack if required)

5. Boot the Windows Guest VM in virtualbox 7
Note: Setting the paravirtualization interface to the "default" option in virtualbox 7 may result in the guest VM perpetually booting into repair mode. Set the paravirtualization explicitely to "Hyper-V" (what Windows actually requires) or "KVM" (this option worked in my case - haven't explicitely tested Hyper-V yet)

6. Install the virtualbox 7 guest additions in your guest VM again.

Hope this helps.

[Andy (montagu)]

I think that pretty clearly shows that upgrading the VirtualBox version to 7 in the Ubuntu repos for 20.04 and 22.04 is not going to be the solution.

[Shai Berger (shai-ikko)]

FWIW I had a situation a lot like that reported by @unhammer in #24, but (maybe because I am using a physical disk for the Windows, maybe because it's a different Windows version) I didn't have clear error messages about drivers. However, the fix suggested by @davidstreicher123 in #25 worked: Boot into Windows (in my case, as a main OS), uninstall Guest Additions, go back to Ubuntu (where VB7 is installed) and then boot Windows again as a VM under the new VirtualBox.

[Andy Townsend (someoneelseosm)]

With regard to "but I don't see enough information here to isolate it to specific versions of packages that Ubuntu shipped" above, perhaps this will help:

On an Ubuntu 22.04 LTS system, "apt search virtualbox | grep installed" finds:

libvirt-bin/now 4.0.0-1ubuntu8.21 amd64 [installed,local]
libvirt-clients/jammy-updates,jammy-security,now 8.0.0-1ubuntu7.10 amd64 [installed,automatic]
libvirt-daemon/jammy-updates,jammy-security,now 8.0.0-1ubuntu7.10 amd64 [installed,automatic]
libvirt-daemon-config-network/jammy-updates,jammy-updates,jammy-security,jammy-security,now 8.0.0-1ubuntu7.10 all [installed,automatic]
libvirt-daemon-config-nwfilter/jammy-updates,jammy-updates,jammy-security,jammy-security,now 8.0.0-1ubuntu7.10 all [installed,automatic]
libvirt-daemon-driver-qemu/jammy-updates,jammy-security,now 8.0.0-1ubuntu7.10 amd64 [installed,automatic]
libvirt-daemon-system/jammy-updates,jammy-security,now 8.0.0-1ubuntu7.10 amd64 [installed,automatic]
libvirt-daemon-system-systemd/jammy-updates,jammy-updates,jammy-security,jammy-security,now 8.0.0-1ubuntu7.10 all [installed,automatic]
libvirt0/jammy-updates,jammy-security,now 8.0.0-1ubuntu7.10 amd64 [installed,automatic]
virtualbox/jammy-updates,now 6.1.50-dfsg-1~ubuntu1.22.04.1 amd64 [installed]
virtualbox-dkms/jammy-updates,now 6.1.50-dfsg-1~ubuntu1.22.04.1 amd64 [installed]
virtualbox-qt/jammy-updates,now 6.1.50-dfsg-1~ubuntu1.22.04.1 amd64 [installed]

"apt search generic | grep 116 | grep installed" finds:

linux-generic/jammy-updates,jammy-security,now 5.15.0.116.116 amd64 [installed]
linux-headers-5.15.0-116-generic/jammy-updates,jammy-security,now 5.15.0-116.126 amd64 [installed,automatic]
linux-headers-generic/jammy-updates,jammy-security,now 5.15.0.116.116 amd64 [installed]
linux-image-5.15.0-116-generic/jammy-updates,jammy-security,now 5.15.0-116.126 amd64 [installed,automatic]
linux-image-generic/jammy-updates,jammy-security,now 5.15.0.116.116 amd64 [installed,automatic]
linux-modules-5.15.0-116-generic/jammy-updates,jammy-security,now 5.15.0-116.126 amd64 [installed,automatic]
linux-modules-extra-5.15.0-116-generic/jammy-updates,jammy-security,now 5.15.0-116.126 amd64 [installed,automatic]

"apt search generic | grep 113 | grep installed" finds:

linux-headers-5.15.0-113-generic/jammy-updates,jammy-security,now 5.15.0-113.123 amd64 [installed,automatic]
linux-image-5.15.0-113-generic/jammy-updates,jammy-security,now 5.15.0-113.123 amd64 [installed,automatic]
linux-modules-5.15.0-113-generic/jammy-updates,jammy-security,now 5.15.0-113.123 amd64 [installed,automatic]
linux-modules-extra-5.15.0-113-generic/jammy-updates,jammy-security,now 5.15.0-113.123 amd64 [installed,automatic]

When booted off the "5.15.0-113-generic" kernel Virtualbox works OK. When booted off the "5.15.0-116-generic" one it does not.

[Tom Robinson (mariner10)]

I have the same setup as Andy (comment #22).

Ubuntu 22.04.4 LTS

VirtualBox is version installed from Ubuntu repo:
virtualbox/jammy-updates,now 6.1.50-dfsg-1~ubuntu1.22.04.1 amd64 [installed]

Problem with no longer being able to start any guest OS in VirtualBox first appeared after system update to kernel package:
linux-image-5.15.0-116-generic/jammy-updates,jammy-security,now 5.15.0-116.126 amd64 [installed,automatic]

Choosing to boot using previous kernel package allows VirtualBox to start guest OSs again without issue. Previous kernel package is:
linux-image-5.15.0-113-generic/jammy-updates,jammy-security,now 5.15.0-113.123 amd64 [installed,automatic]

[Kevin Brubeck Unhammer (unhammer)]

Does this mean the uninstall-upgrade-reinstall dance has to be done anyway on any major upgrade of virtualbox (e.g. when upgrading Ubuntu past jammy)?

[David Streicher (davidstreicher123)]

@Kevin yes. It really depends on the extent of changes between major versions of virtualbox as to whether it will _always_ be required, but it is at least necessary in the 6.1 -> 7 migration case.

AFAICT Oracle never released an official VM migration guide with the release of virtualbox 7, but based on the bug reports on Stack Overflow, the Virtualbox threads, and my own experience, I think its safe to say that the Windows Guest Additions driver in 6.1 is definitely not compatible with Virtualbox 7.

[Schwobi (hemaha)]

Don't think this will be fixed any time soon. Has someone a way to get in touch with the upstream kernel developers..?

[Keith Z-G (keithzg)]

Can sadly but unsurprisingly confirm that the new 5.15.0-117 kernel doesn't resolve this on 22.04.

[Graham Jenkins (grahjenk)]

Yes, the problem's still there for me with kernel 5.15.0-117 on 22.04.

[Vasya Pupkin (shadowlmd)]

This bug is reported against wrong package. Is it possible to change it somehow? Maybe that's the reason this bug has so little attention.

[Schwobi (hemaha)]

> Quoting comment 36: "This bug is reported against wrong package."

The bug report is currently filed against the packages "linux-signed-hwe-5.15 (Ubuntu)" and "virtualbox (Ubuntu)". Would you mind to elaborate on why you think that this is not appropriate?

[Vasya Pupkin (shadowlmd)]

Primary package for this bug should be linux-signed-hwe-5.15. VirtualBox is affected, but it's not the root cause.

I am judging by the bug URL, currently it contains "virtualbox", so it may look like a VirtualBox bug. If that's not a big deal and Linux kernel maintainers are properly notified about this bug, then it's ok.

[Andy (montagu)]

I think the question is what is the correct kernel package to list it against. "linux-signed-hwe-5.15" is a hardware enablement kernel package but the issue is not specific to that build of the kernel.

[Manuel Bestle (firewiremb)]

In the Dublicate Bug 2074095 one of the Kernel Developers asked für help to debug. See comment #3.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2074095/comments/3

I am using Linux Mint so I can't help him. Maybe someone from this thread is able to provide him the nessesary informations.

[Schwobi (hemaha)]

Note that the URL is irrelevant. The bug is listed for all indicated packages; see: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.15/+bug/2073267

To my understanding, the HWE stack is the default for kernel installations in Ubuntu. Its is upstream of any Ubuntu-signed kernel images of the branch 5.15. By installing "linux-signed-hwe-5.15", one's system will automatically update and install always the latest kernel of the branch 5.15.
It is also the only package indicated if one searches for the query "5.15.0-*-generic" in the Target/Distribution/Package field on the top of the bug report.

[Schwobi (hemaha)]

> Quoting comment #40: "In the Dublicate Bug 2074095 one of the Kernel Developers asked für help to debug. See comment #3."

Adjusting the apport-collect command for this bug report corresponds to:
apport-collect 2073267

The crash happens in any guest virtual machine (not only Ubuntu), which becomes unresponsive and barely generates any usable log output. Thus, the command can not be run on the guest machine.
Are we supposed to run the apport on the host immediately after the guest has crashed..?

[Schwobi (hemaha)]

apport-collect after virtualbox crashed on Ubuntu 20.04.4 LTS with kernel version 5.15.0-116
and graphics driver nvidia-driver-535 (proprietary, tested):

ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.27
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 20.04
InstallationDate: Installed on 2022-03-26 (857 days ago)
InstallationMedia: Ubuntu 20.04.4 LTS "Focal Fossa" - Release amd64 (20220223)
NonfreeKernelModules: nvidia_modeset nvidia
Package: virtualbox 6.1.50-dfsg-1~ubuntu1.20.04.1
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 5.15.0-116.126~20.04.1-generic 5.15.158
Tags: focal
Uname: Linux 5.15.0-116-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

As the crash of the virtual machine might be graphics driver - related, I tried the nouveau driver.
Alas, I get the same issue.

apport-collect after virtualbox crashed on Ubuntu 20.04.4 LTS with kernel version 5.15.0-116
and graphics driver x-server-xorg-video-nouveau:

ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.27
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 20.04
InstallationDate: Installed on 2022-03-26 (857 days ago)
InstallationMedia: Ubuntu 20.04.4 LTS "Focal Fossa" - Release amd64 (20220223)
NonfreeKernelModules: nvidia_drm nvidia_modeset nvidia
Package: virtualbox 6.1.50-dfsg-1~ubuntu1.20.04.1
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 5.15.0-116.126~20.04.1-generic 5.15.158
Tags: focal
Uname: Linux 5.15.0-116-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

apport information

[Schwobi (hemaha)]

I would love to test the recent kernel of linux-image-5.15.0-117-generic (version: 5.15.0-117.127~20.04.1 (focal updates).
But unfortunately I am not even getting to the graphical user interface (i.e. login screen) with that kernel version. And in the TTY that I _can_ access, there doesn't seem to be an option to report the bug with the "apport-bug linux-image-5.15.0-117-generic" command, as this fails due to the inability to open a web browser (Edit: Found the "apport-cli" command to work with the "--save" flag and will open a separate bug report for this issue).
More and more stuff seems to be breaking for me with the recent kernel versions...

[AaronMa (mapengyu)]

Hi all,

1, This issue is caused by the following 2 commits in upstream kernel:
9c573cd31343 randomize_kstack: Improve entropy diffusion
ef40d28f17bd randomize_kstack: Remove non-functional per-arch entropy filtering

But only virtualbox 6.1.50 will crash.
VirtualBox 6.1.x is no longer supported! Which is anounced by virtualbox.org.

So it's a bit hard to push kernel changes for an unsupported application.
I will file an upstream bug to try.

2, virtualbox 7.0.20 from virtualbox.org will work well on all the kernel versions.

Please use virtualbox 7.0.20 if you can.

[Gustavo Homem (Personal) (gustavo-pa)]

Confirmed to be a problem with kernel 5.15.117 installed via unattended-upgrades. 5.15.116 does not work. Virtualbox previously worked. Trying to revert to 5.15.107.

This is on Kubuntu 22.04.

[Schwobi (hemaha)]

> Quoting @mapengyu: "VirtualBox 6.1.x is no longer supported! Which is announced by virtualbox.org. So it's a bit hard to push kernel changes for an unsupported application."

That is what I was alluding to in comment #14.
Thus, the real question might be if there is a way to facilitate/expedite the port of virtualbox version 7.0.x into the official repositories of focal/jammy (Ubuntu 20.04 / 22.04) which are affected due to an outdated version of aforementioned application..? I am asking this since both LTS releases are _supposed_ to supported for a few more years.

[AaronMa (mapengyu)]

Upstream bug:

https://bugzilla.kernel.org/show_bug.cgi?id=219107

[AaronMa (mapengyu)]

Virtualbox official deb package for Ubuntu 22.04:

https://download.virtualbox.org/virtualbox/7.0.20/virtualbox-7.0_7.0.20-163906~Ubuntu~jammy_amd64.deb

[Andy (montagu)]

Putting Virtualbox 7 into the Ubuntu LTS repos is problematic too, since as per comment #25, that change will also break people's VMs.

5.15 is a LTS kernel version. I think it is reasonable to expect it to continue to support the applications that ran on it at the time it went into LTS, even if those applications are themselves out of support and would not work on the latest kernel. LTS is supposed to be stable, not breaking combinations of things that worked yesterday, at least not without a very good reason.

Good luck with the upstream bug, @mapengyu, and thanks for your efforts. It's great to know that the root cause has been isolated, and the possibility of a solution is in sight.

[Andy (montagu)]

Though I have to concede, if a change to stack randomization broke Virtualbox (and only Virtualbox), then it suggests perhaps Virtualbox was doing something it shouldn't, and just happened to get away with it all this time.

[Gustavo Homem (Personal) (gustavo-pa)]

5.15.107 did not boot not sure why. Temporary mitigation:

```
sudo apt-get install --install-recommends linux-generic-hwe-22.04
sudo update-grub2
```

System boots and virtualbox works.

[Gustavo Homem (Personal) (gustavo-pa)]

Ideally a patch number update to the 5.15.X kernel line should not break installed software like Virtualbox. Even if upgrading the kernel or upgrading virtualbox is a workaround, this situation is undesired within an LTS release.

The main reason for staying inside the Ubuntu repositories (for python3, docker, virtualbox, etc...) is the LTS maintenance of the existing package set.

Thanks everybody for the efforts.

[Matthew Ruffell (mruffell)]

Hello everyone,

Lets get this problem solved.

Now, doing a major version update to virtualbox in Jammy from 6.1.50-dfsg-1~ubuntu1.20.04.1 to 7.0.20 is against Ubuntu SRU policy, and would also come with the consequence that it would break all existing VMs that use guest additions, as guest additions would have to be removed from VMs, virtualbox upgraded, and then guest additions installed again.

So no, doing a major upgrade to virtualbox is the wrong approach.

We have two options:
1) Locate the change in the kernel, then either revert it, or find a fix for it.
2) Locate the fix in virtualbox 7.0.20 and backport if possible to 6.1.50.

For the moment, let's just investigate 1).

I read AaronMa's comment with the upstream bug:

https://bugzilla.kernel.org/show_bug.cgi?id=219107

Now, only the first commit got backported to the jammy 5.15 kernel:

commit 507882a9d37affee0903c0ebb3d1544f4bfba024 ubuntu-jammy
Author: Kees Cook <email address hidden>
Date: Sat Mar 9 12:24:48 2024 -0800
Subject: randomize_kstack: Improve entropy diffusion
Link: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/?id=507882a9d37affee0903c0ebb3d1544f4bfba024

This landed in:

$ git describe --contains 507882a9d37affee0903c0ebb3d1544f4bfba024
Ubuntu-5.15.0-115.125~210

which is when we started seeing the issues.

I have reverted this commit, and built test kernels for the more technically
inclined people to try:

https://launchpad.net/~mruffell/+archive/ubuntu/lp2073267-test

I just uploaded them. Please wait three hours from this message for them to have finished building.
There are both focal HWE and jammy kernels in this ppa.

Please note this package is NOT SUPPORTED by Canonical, and is for TESTING
PURPOSES ONLY. ONLY Install in a dedicated test environment.

Instructions to Install (On a focal or jammy system):
1) sudo add-apt-repository ppa:mruffell/lp2073267-test
2) sudo apt update
3) sudo apt install linux-image-unsigned-5.15.0-117-generic linux-modules-5.15.0-117-generic linux-modules-extra-5.15.0-117-generic linux-headers-5.15.0-117-generic
4) sudo reboot
5) uname -rv
Look for +TEST2073267v20240731b1

If you are asked to abort removal of the currently running kernel, say no.

Does it fix virtualbox 6.1.50? Please let me know.

In the meantime, I will try and look through the virtualbox subversion repository to find where virtualbox fixed this to see if backporting a fix for virtualbox would be a better route to take.

https://www.virtualbox.org/browser/vbox/trunk

Thanks,
Matthew

[Gianfranco Costamagna (costamagnagianfranco)]

@Matthew Ruffell (mruffell)
Debian and Ubuntu virtualbox maintainer here.

Some questions:
1) can you please try a ppa build with also a cherry-pick of ef40d28f17bd randomize_kstack: Remove non-functional per-arch entropy filtering
instead of reverting?

If the issue is fixed with another cherry-pick and not by a revert it would be better

2)In the meantime, I will try and look through the virtualbox subversion repository to find where virtualbox fixed this to see if backporting a fix for virtualbox would be a better route to take.

no need to do it, upstream rewritten all the code (for different reasons) and the bug is just not there anymore.

Quoting upstream
"7.0 is unaffected because of some code rewrite (without intending to avoid any such problem) but that's a widespread change which we won't make for 6.1."

[Matthew Ruffell (mruffell)]

Hi Gianfranco,

Great! Let's work this out together.

I have strong doubts that:

commit ef40d28f17bd384d7e0b630c7d83f108a526351b
Author: Kees Cook <email address hidden>
Date: Wed Jun 19 14:47:15 2024 -0700
Subject: randomize_kstack: Remove non-functional per-arch entropy filtering
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ef40d28f17bd384d7e0b630c7d83f108a526351b

will fix this issue. If you read the commit log and the code, it clearly removes all per-arch entropy values, and just sticks with a universal 1KiB of space for all architectures.

The root cause of the issue is that "randomize_kstack: Improve entropy diffusion" changed the amd64 kernel stack consumption for randomisation, from 0.25KiB to a full 1KiB of space. The per thread kernel stacks are only 16KiB in size, so we went from VirtualBox having 15.75KiB of stack space down to 15KiB. VirtualBox must have been really pushing the limit and needing that extra 0.75KiB of space, since without it, we panic.

They probably made some architectural changes in 7.0.x that reduce the total kernel thread stack consumption, and now fall under the 15KiB limit that "randomize_kstack: Improve entropy diffusion" imposes.

Anyway, I still made you a test kernel. It is based on 5.15.0-117-generic + "randomize_kstack: Remove non-functional per-arch entropy filtering".

This test kernel is for Gianfranco Costamagna ONLY! Other users, please, try my other test kernel in the above comment instead.

Gianfranco, the kernel will be ready in 3 hours from this message. They are still building.

Please note this package is NOT SUPPORTED by Canonical, and is for TESTING
PURPOSES ONLY. ONLY Install in a dedicated test environment.

Instructions to Install (On a focal or jammy system):
1) sudo add-apt-repository ppa:mruffell/lp2073267-test-2
2) sudo apt update
3) sudo apt install linux-image-unsigned-5.15.0-117-generic linux-modules-5.15.0-117-generic linux-modules-extra-5.15.0-117-generic linux-headers-5.15.0-117-generic
4) sudo reboot
5) uname -rv
Look for +TEST2073267v20240731b2

Anyway, I think we either need to figure out how to get the virtualbox kernel module stack consumption down, or we revert "randomize_kstack: Improve entropy diffusion" for focal, focal HWE, jammy, jammy HWE (but not noble).

 virtualbox | 6.1.6-dfsg-1 | focal/multiverse | source, amd64
 virtualbox | 6.1.32-dfsg-1build1 | jammy/multiverse | source, amd64
 virtualbox | 6.1.50-dfsg-1~ubuntu1.20.04.1 | focal-security/multiverse | source, amd64
 virtualbox | 6.1.50-dfsg-1~ubuntu1.20.04.1 | focal-updates/multiverse | source, amd64
 virtualbox | 6.1.50-dfsg-1~ubuntu1.22.04.1 | jammy-updates/multiverse | source, amd64
 virtualbox | 6.1.50-dfsg-1~ubuntu1.22.04.2 | jammy-proposed/multiverse | source, amd64
 virtualbox | 7.0.16-dfsg-2 | noble/multiverse | source, amd64
 virtualbox | 7.0.16-dfsg-2ubuntu1 | noble-updates/multiverse | source, amd64
 virtualbox | 7.0.20-dfsg-1 | oracular/multiverse | source, amd64

Thanks,
Matthew

[Keith Z-G (keithzg)]

I can happily report that installing 5.15.0-117-generic #127+TEST2073267v20240731b1-Ubuntu from the PPA cited in comment #74 here allows VirtualBox 6.1.50-dfsg-1~ubuntu1.22.04.1 to run VMs just fine on my test Jammy system!

[Matthew Ruffell (mruffell)]

Hi everyone,

Keith, thanks for testing! It is fantastic news that the test kernel fixes the issue, and our theory is correct, and we are on the right track.

I also tested both test kernels myself, and can confirm that reverting "randomize_kstack: Improve entropy diffusion" fixes the issue, and that "randomize_kstack: Remove non-functional per-arch entropy filtering" has no improvement at all, and still has the issue.

So, I think the way forward is to do the unpopular thing, which is to revert "randomize_kstack: Improve entropy diffusion".

We should never have changed something so fundamental as the kernel thread stack size on a stable kernel. We can do that on the development release sure, but not on LTS kernels.

I will have a talk with the Kernel team about it, but I will begin preparing the patches and writing a SRU template.

Thanks,
Matthew

[AaronMa (mapengyu)]

@mruffell

Thanks for re-verified my comment in #65 and provide the test kernel.

The solutions are under Kernel team's discussion.

Reverting is an option, but not the best. Actually, it's not a kernel issue.

We are still working on it. Please wait a moment.

If you have more insight in virtualbox fixes please let us know.

[Matthew Ruffell (mruffell)]

Hi everyone,

I see that Greg KH just assigned CVE-2024-35918 to "randomize_kstack: Improve entropy diffusion".

I suppose that means that we cannot revert it now.

https://lore.kernel.org/linux-cve-announce/2024073029-clerk-trophy-b84c@gregkh/T/

This is going to take some time.

Thanks,
Matthew

[David Streicher (davidstreicher123)]

@mruffell Perhaps I'm missing something, but the kernel discussion thread you linked specifically ends with "The CVE has now been rejected, thanks for the review!".

Surely that means that the Ubuntu team can revert the commit then? (unless you are of the opinion that the security vulnerability the commit addresses is serious enough that the change has to be kept regardless of CVE status?)

[Matthew Ruffell (mruffell)]

Hi David,

Yes, it does indeed seem that the CVE has now been rejected.

https://lore.kernel.org/linux-cve-announce/2024073029-clerk-trophy-b84c@gregkh/

https://nvd.nist.gov/vuln/detail/CVE-2024-35918
https://www.cve.org/CVERecord/?id=CVE-2024-35918

Maybe we can revert it after all!

I will have a talk with Aaron and the Kernel Team about how we should move forward.

Thanks,
Matthew

[Roxana Nicolescu (roxanan)]

As seen in the modified description, we have decided to revert the commit "randomize_kstack: Improve entropy diffusion" for now. The updated kernel should be available in ~2 weeks.
This is against our policy, differing from upstream makes the maintenance of kernels harder, but the fix from virtualbox 7.x is not easy to backport, and upgrading to 7.x for jammy will break a bunch of people's setups.
This is a temporary fix. We will look into either finding something on the kernel side upstream or fixing it on the virtualbox side.

[Schwobi (hemaha)]

@ Stefan Bader (smb)

The "affects"/"status" changes erroneously suggest that only jammy would would affected.
Please note that this also affects the focal repositories!

Big thanks to all the developers that have gotten involved and proposed a reasonable fix in a timely manner!

[Kees Cook (kees)]

Anyone affected by this should be able to boot with "randomize_kstack_offset=off" on the kernel command line to disable the offset randomization. No need to upgrade anything nor revert anything. :P

[David Streicher (davidstreicher123)]

@kees Thanks for jumping on this thread!

From a technical perspective I agree with you, but I would reason that it's not really a change one would want end-users of the Ubuntu LTS kernel to implement.

It comes back to the reasoning mentioned by @Andy in #70 and @Gustav in #73: a kernel update on a LTS kernel really shouldn't break previously running software, and I would add that a normal package update via apt shouldn't require the user to have to tinker with the kernel boot parameters before getting a working system again.

Unless there is some way for the package upgrade to automatically alter the boot parameters? (Although even if possible, this is probably not desired behaviour from a package upgrade either)

[Andy (montagu)]

I tried the workaround suggested by @kees, and it works for me on the latest jammy kernel (now 5.15.0-117). With randomize_kstack_offset not specified on the kernel command line, VMs fail to start with the guru meditation error. With randomize_kstack_offset=off on the kernel command line of the host machine, guest VMs start successfully.

This seems like a valuable workaround for any affected users who need other kernel fixes that were made in -116 or -117.

If I've understood the discussions correctly, the downside of this approach is that it disables kernel stack offset randomization completely; whereas reverting to the 5.15.0-113 kernel will retain that security safety net, albeit with a reduced level of entropy in the randomization.

[Andy (montagu)]

Something interesting I came across when looking up the documentation for the randomize_kstack_offset parameter: apparently Virtualbox had a similar problem when this feature was introduced in the 5.13 kernel, and they made a change to fix it in Virtualbox 6.1.24. I wonder what they did?

https://www.virtualbox.org/ticket/20452

[Andy (montagu)]

This Virtualbox ticket contains a diff that looks like it might be the change they did to fix compatibility with RANDOMIZE_KSTACK_OFFSET in the 5.13 kernel.
https://www.virtualbox.org/ticket/20456

Edited to add: found links to the relevant SVN changesets...
https://www.virtualbox.org/changeset/90173/vbox
https://www.virtualbox.org/changeset/90177/vbox

[Kees Cook (kees)]

Right, yeah, I don't mean that universally disabling the kstack randomization is the permanent solution. It should be possible to add a patch to Ubuntu's kernel to restore the prior bit width to deal with Virtualbox.

[Gianfranco Costamagna (costamagnagianfranco)]

@montagy
The change you mention is part of virtualbox since 6.1.26
commit d0c8c1d9763108e99a60822c3441881cf0007e24 (tag: upstream/6.1.26-dfsg)
Author: Gianfranco Costamagna <email address hidden>
Date: Thu Jul 29 11:18:35 2021 +0200

    New upstream version 6.1.26-dfsg

diff --git a/src/VBox/VMM/VMMR0/VMMR0JmpA-amd64.asm b/src/VBox/VMM/VMMR0/VMMR0JmpA-amd64.asm
index 9d3304175..e1ab30b72 100644
--- a/src/VBox/VMM/VMMR0/VMMR0JmpA-amd64.asm
+++ b/src/VBox/VMM/VMMR0/VMMR0JmpA-amd64.asm
@@ -32,10 +32,17 @@
 %define STACK_PADDING 0eeeeeeeeeeeeeeeeh

 ;; Workaround for linux 4.6 fast/slow syscall stack depth difference.
+;; Update: This got worse with linux 5.13 and CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT.
+;; The x86 arch_exit_to_user_mode_prepare code limits the offset to 255,
+;; while the generic limit is 1023. See bugref:10064 for details.
 %ifdef VMM_R0_SWITCH_STACK
  %define STACK_FUZZ_SIZE 0
 %else
- %define STACK_FUZZ_SIZE 128
+ %ifdef RT_OS_LINUX
+ %define STACK_FUZZ_SIZE 384
+ %else
+ %define STACK_FUZZ_SIZE 128
+ %endif
 %endif

So definitely not the issue

[Andy (montagu)]

That's right, it was a fix for a Virtualbox error that arose when kernel stack randomization was first introduced in the 5.13 kernel.
My speculation was that it might have been a similar issue, and if so, its fix might give a clue to a way to fix the current issue in Virtualbox.

One might naively observe that when the available kernel stack space was reduced by 0.25kb in kernel 5.13, Virtualbox was fixed by adding 256 to STACK_FUZZ_SIZE. Now kernel 5.15.0-116 has reduced the available stack space by another 0.75kb; could Virtualbox be fixed by adding another 768 to STACK_FUZZ_SIZE? I don't know enough about what's going on under the hood to know if that's even plausible, or what undesirable side effects it might have.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/5.15.0-119.129 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux' to 'verification-done-jammy-linux'. If the problem still exists, change the tag 'verification-needed-jammy-linux' to 'verification-failed-jammy-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Schwobi (hemaha)]

Is there a reason why a fix has not yet been committed for focal (Ubuntu 20.04), which is likewise affected..?

[Matthew Ruffell (mruffell)]

Performing verification for Jammy.

I deployed a fresh baremetal server running Jammy in the Server Lab, installed
ubuntu-desktop and rebooted.

The kernel is 5.15.0-117-generic from updates:
5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024

$ sudo apt-cache policy virtualbox
virtualbox:
  Installed: 6.1.50-dfsg-1~ubuntu1.22.04.1

I downloaded a Ubuntu server iso from cdimage, and made a new virtualbox VM.

On starting, I get "Guru Meditation" and in the logs I see

00:00:02.191995 emR3Debug: rc=VERR_VMM_SET_JMP_ABORTED_RESUME

I then enable proposed2 and installed 5.15.0-119-generic:

5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024

The virtualbox VM started up normally. Happy to mark verified for jammy.

[Arrigo Marchiori (ardovm)]

Sorry but I cannot replicate the test here.

After enabling the jammy-proposed apt sources, I got kernel 5.15.0.118, not .119:

# apt-get install linux-generic/jammy-proposed
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
linux-generic is already the newest version (5.15.0.118.118).

And the VMs still do not start.

There seem to be VirtualBox packages in -proposed as well. Version is 6.1.50-dfsg-1~ubuntu1.22.04.3.
Shall I install them too?

Thank you in advance.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matthew Ruffell (mruffell)]

Hi Arrigo,

Kernel SRU cycles are a little tricky... the Kernel Team make 150 kernels twice
a month, so not everything goes in the distro -proposed pocket.

This one is a respin, and hasn't made its way to the main -proposed pocket yet,
and because its a part of the security SRU cycle, it gets placed in the Kernel
Teams -proposed2 pocket:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/proposed2

The Kernel Team also have a -proposed pocket, which is just like the distro
-proposed, but only has the primary kernel SRU cycle packages in it:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/proposed

anyway, -119 is in -proposed2 if you want to try it.

The virtualbox package in -proposed is the same as the one in -updates, and
fixes a DKMS build issue with the 6.8 HWE kernel.

Hope that clears some things up.

Matthew

[Arrigo Marchiori (ardovm)]

I confirm that kernel 5.15.0-119 from the proposed2 PPA fixes this issue on Jammy.

Thank you, Matthew, for your kind explanation, and to all the developers who worked on this!

[Schwobi (hemaha)]

Wanting to test whether the fix had also made it into the focal-proposed (Ubuntu 20.04) kernel version -119, despite the absence of any indication of that in the bug status description, I tried to install the proposed kernel, but I can't:

# trying to install from proposed2 (also tested proposed, which didn't make any difference)
:~$ sudo apt-get install linux-hwe-5.15.0-119/focal-proposed2
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-hwe-5.15.0-119
E: Couldn't find any package by glob 'linux-hwe-5.15.0-119'
E: Couldn't find any package by regex 'linux-hwe-5.15.0-119'

# tried to be more generic...
:~$ sudo apt-get install linux-generic/focal-proposed2
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Release 'focal-proposed2' for 'linux-generic' was not found

# getting paranoid, thus confirming that the proposed-repository was indeed enabled
:~$ cat /etc/apt/sources.list | grep -i proposed
deb http://de.archive.ubuntu.com/ubuntu/ focal-proposed main universe restricted multiverse

# but alas, no such package(s) can be found by apt
:~$ sudo apt-cache search --names-only "linux.*\-119" | grep -e 'image' -e 'hwe'
<empty return>

And yet, the ubuntu/proposed2 archive seems to contain the packages of interest:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/proposed2?field.series_filter=focal

What am I missing..?

[Ian Warrilow (umisterw)]

linux-hwe-5.15.0-119 does seem to be in the focal-proposed2 repos now.
Added proposed2 to a test system and 119 was installed on update.
Couldnt test virtualbox though as not installed on the system

[Ian Warrilow (umisterw)]

Thought I may as well install Virtualbox and test it whilst I was here.
Booted to kernel 117, Installed VB 6.1.50 from repos. Created VM with no disc, started it, got the Guru-mediation error.
Rebooted to 119 kernel, VM started just fine.

I can confirm kernel 119 fixes the problem for focal.

[Detlef Bracker (supermann-granada)]

kernel 118 - the problem is not fixed !!!
I hope 119 fix this this long time problem!

Please understand: when this problems longer time exists, then linux (ejemplo Mint) are not better as Windows !!! So you go backwards! This cant been!

I can only use the kernal 106/107 on 3 machines with all different problems and this over a time of 2-3 months! Before was all good!

[Schwobi (hemaha)]

@ Detlef Bracker:

I understand your frustration.
But, it should be acknowledged that the developers' have promptly responded and committed the appropriate changes in a reasonable amount of time.
Kernel version 5.15.0-119 should fix the issue and be available in the official repositories (presumable also in downstream Linux Mint) within the next few weeks.
Please be just a bit more patient.

[Andy (montagu)]

@Detlef:
There are a number of workarounds to address this problem while waiting for the long-term fix in the -119 kernel.
You can change your grub configuration to either boot an older kernel by default, or to add randomize_kstack_offset=off to the kernel command line (see comment #85).
Alternatively, use apt to uninstall the recent kernel and revert to the -113 kernel.
All 3 of these options work equally well so it's just a question of which one you find most convenient.

Note that this specific bug has been precisely traced to a change introduced in the 5.15.0-116 kernel, so any problems you've had with any earlier kernel versions are a different issue.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-hwe-6.8/6.8.0-41.41~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-hwe-6.8' to 'verification-done-jammy-linux-hwe-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-hwe-6.8' to 'verification-failed-jammy-linux-hwe-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ian Warrilow (umisterw)]

Confirmed Linux-hwe-6.8.0.41 fixes the problem in jammy

[Peter Sejerman (p-sejerman)]

Tested the following combinations with VirtualBox 6.1.50_Ubuntu r161033.

Ubuntu 20.04:
linux-image-5.15.0-117-generic -> Guru Meditation Error
linux-image-5.15.0-118-generic -> Guru Meditation Error
linux-image-5.15.0-119-generic -> successful boot of VM

Ubuntu 22.04:
linux-image-5.15.0-118-generic -> Guru Meditation Error on VM-start
linux-image-5.15.0-119-generic -> successful boot of VM

linux-image-6.8-40-generic -> Guru Meditation Error
linux-image-6.8-41-generic (linux-generic-hwe-22.04) -> successful boot of VM

[Janne Pulkkinen (matoking)]

Tested the following:

* linux-generic-hwe-22.04 6.8.0-40 crashes on VM boot with the VERR_VMM_SET_JMP_ABORTED_RESUME error
* linux-generic-hwe-22.04 6.8.0-41 boots VM successfully

[Thomas Schönemann (tschoenemann)]

Confirmed the fix solves the bug in focal, jammy, jammy-hwe

Sorry, i can't edit tags and add "verification-done-jammy-linux-hwe-6.8" - i could not figure out how to:(

[Kopa Rebu (koparebu)]

> Sorry, i can't edit tags and add "verification-done-jammy-linux-hwe-6.8" - i could not figure out how to:(

I guess I managed to.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.15.0-119.129

---------------
linux (5.15.0-119.129) jammy; urgency=medium

  * jammy/linux: 5.15.0-119.129 -proposed tracker (LP: #2075665)

  * Virtualbox Guru meditation on VM start caused by kernel commit in v6.9-rc4
    (LP: #2073267)
    - SAUCE: Revert "randomize_kstack: Improve entropy diffusion"

  * CVE-2024-26921
    - inet: inet_defrag: prevent sk release while still in use

  * Jammy update: v5.15.162 upstream stable release (LP: #2073765) //
    CVE-2024-39484
    - mmc: davinci: Don't strip remove function when driver is builtin

  * Jammy update: v5.15.162 upstream stable release (LP: #2073765)
    - mmc: davinci_mmc: Convert to platform remove callback returning void

  * CVE-2024-39292
    - um: Add winch to winch_handlers before registering winch IRQ

  * CVE-2024-36901
    - ipv6: prevent NULL dereference in ip6_output()

  * CVE-2024-26830
    - i40e: Do not allow untrusted VF to remove administratively set MAC

  * CVE-2024-26680
    - net: atlantic: Fix DMA mapping for PTP hwts ring

  * CVE-2023-52760
    - gfs2: Fix slab-use-after-free in gfs2_qd_dealloc

  * CVE-2023-52629
    - sh: push-switch: Reorder cleanup operations to avoid use-after-free bug

 -- Manuel Diewald <email address hidden> Fri, 02 Aug 2024 16:15:36 +0200

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gke/5.15.0-1065.71 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-gke' to 'verification-done-jammy-linux-gke'. If the problem still exists, change the tag 'verification-needed-jammy-linux-gke' to 'verification-failed-jammy-linux-gke'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gkeop/5.15.0-1051.58 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-gkeop' to 'verification-done-jammy-linux-gkeop'. If the problem still exists, change the tag 'verification-needed-jammy-linux-gkeop' to 'verification-failed-jammy-linux-gkeop'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-ibm/5.15.0-1061.64 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-ibm' to 'verification-done-jammy-linux-ibm'. If the problem still exists, change the tag 'verification-needed-jammy-linux-ibm' to 'verification-failed-jammy-linux-ibm'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-ibm-gt/5.15.0-1071.74 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-ibm-gt' to 'verification-done-jammy-linux-ibm-gt'. If the problem still exists, change the tag 'verification-needed-jammy-linux-ibm-gt' to 'verification-failed-jammy-linux-ibm-gt'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-realtime/5.15.0-1069.77 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-realtime' to 'verification-done-jammy-linux-realtime'. If the problem still exists, change the tag 'verification-needed-jammy-linux-realtime' to 'verification-failed-jammy-linux-realtime'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-intel-iotg/5.15.0-1063.69 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-intel-iotg' to 'verification-done-jammy-linux-intel-iotg'. If the problem still exists, change the tag 'verification-needed-jammy-linux-intel-iotg' to 'verification-failed-jammy-linux-intel-iotg'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-kvm/5.15.0-1065.70 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-kvm' to 'verification-done-jammy-linux-kvm'. If the problem still exists, change the tag 'verification-needed-jammy-linux-kvm' to 'verification-failed-jammy-linux-kvm'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-lowlatency/5.15.0-119.129 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-lowlatency' to 'verification-done-jammy-linux-lowlatency'. If the problem still exists, change the tag 'verification-needed-jammy-linux-lowlatency' to 'verification-failed-jammy-linux-lowlatency'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-raspi/5.15.0-1061.64 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-raspi' to 'verification-done-jammy-linux-raspi'. If the problem still exists, change the tag 'verification-needed-jammy-linux-raspi' to 'verification-failed-jammy-linux-raspi'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia/5.15.0-1063.64 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia' to 'verification-done-jammy-linux-nvidia'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia' to 'verification-failed-jammy-linux-nvidia'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oracle/5.15.0-1066.72 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-oracle' to 'verification-done-jammy-linux-oracle'. If the problem still exists, change the tag 'verification-needed-jammy-linux-oracle' to 'verification-failed-jammy-linux-oracle'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-hwe-5.15/5.15.0-119.129~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-hwe-5.15' to 'verification-done-focal-linux-hwe-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-hwe-5.15' to 'verification-failed-focal-linux-hwe-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oracle-5.15/5.15.0-1066.72~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-oracle-5.15' to 'verification-done-focal-linux-oracle-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-oracle-5.15' to 'verification-failed-focal-linux-oracle-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-bluefield/5.15.0-1051.53 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-bluefield' to 'verification-done-jammy-linux-bluefield'. If the problem still exists, change the tag 'verification-needed-jammy-linux-bluefield' to 'verification-failed-jammy-linux-bluefield'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Torsten Bronger (bronger)]

Apparently I need Linux-hwe-6.8.0.41 for jammy. I have no experience with this fixing process, so what remaining waiting time have I to assume? I need the VM urgently for my work and have to make a decision which option to take (e.g. installing VB 7).

It’s okay if it still needs time, I just need to know.

[Joner Cyrre Worm (jonerworm-6)]

VMs starting OK again. Thanks for the work!

~$ dpkg -l linux-generic*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=======================-======================-============-=========================================
ii linux-generic-hwe-20.04 5.15.0.119.129~20.04.1 amd64 Complete Generic Linux kernel and headers

~$ dpkg -l virtualbox
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=============================-============-===========================================
ii virtualbox 6.1.38-dfsg-3~ubuntu1.20.04.1 amd64 x86 virtualization solution - base binaries

~$ uname -r
5.15.0-119-generic

~$ lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal

[Michael Plotkin (plotkin)]

Just tested VM 6.1.50 with Ubuntu kernel 5.15.0-119-generic.
Confirmed that it starts and works normally as expected according to the 119 kernel changelog.

Thank you very much to everybody who helped in resolving this problem.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra/5.15.0-1028.28 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-tegra' to 'verification-done-jammy-linux-nvidia-tegra'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-tegra' to 'verification-failed-jammy-linux-nvidia-tegra'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra-igx/5.15.0-1016.16 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-tegra-igx' to 'verification-done-jammy-linux-nvidia-tegra-igx'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-tegra-igx' to 'verification-failed-jammy-linux-nvidia-tegra-igx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Carlos Cristian Rios Donoso (ccrios)]

Same bug in Ubuntu 22.04.4, to install virtualbox-7 fixed the problem.
Thanks

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-mtk/5.15.0-1034.39 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-done-jammy-linux-mtk'. If the problem still exists, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-failed-jammy-linux-mtk'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!