System freezes when starting Xorg after installing linux-image-4.13.0-32-generic

[Sandra Ray]

My laptop (Lenovo T440) freezes when starting Xorg (lightdm) only after installing the latest ubuntu kernel for 17.10, linux-image-4.13.0-32-generic.

Am running Xorg instead of MIR due to MIR crashing when I initially upgraded from 17.04 to 17.10, but that's a different problem for a different day. It's been running with Xorg stably for at least 2-3 months.

If I boot with the previous kernel version on my /boot (linux-image-4.13.0-25-generic), everything works fine.

If I boot with the latest mainline kernel from the Ubuntu mainline repo (4.15.0), everything works.

Using grub's option to run "Recovery Mode" for the 4.13.0-32 kernel, I am able to use the CLI and access the encrypted root filesystem, which is how I generated the apport repo that's attached. It's only when telling it to continue Normal booting that it freezes, which is why I suspect the problem of being Xorg.

When the system freezes, the fan starts to spin up. I cannot switch virtual consoles using CTRL-ALT-F{1-7}. CTRL-ALT-DEL pressed repeatedly doesn't do anything. To reboot, I have to hold the power button until the system powers off.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1746418

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[SkySurfer (mueller-g)]

I was able to gather the output by disabling plymoth screen:

[ 61.359631] NMI watchdog: Watchdog detected hard LOCKUP on cpu 4
[ 64.063967] watchdog: BUG: soft lockup - CPU6 stuck for 22s! [snap-exec:6105]
[ 64.063968] watchdog: BUG: soft lockup - CPU7 stuck for 22s! [systemd:6412]
[ 80.059966] watchdog: BUG: soft lockup - CPU0 stuck for 22s! [kworker/0::0:3]
[ 84.823028] NMI watchdog: Watchdog detected hard LOCKUP on cpu 5

[Joseph Salisbury (jsalisbury)]

I'd like to perform a "Reverse" bisect to figure out what commit fixes this bug. We need to identify the last kernel version that had the bug, and the first kernel version that fixed the bug.

Can you test the following kernels and report back:

v4.14 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14/
v4.15-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc1/
v4.15-rc4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc4/

You don't have to test every kernel, just up until the first kernel that does not have the bug.

Thanks in advance!

[Sandra Ray (sandraray)]

I can confirm that v4.14 final works.

Also, I made an obvious blunder in the initial report: I had to disable *Wayland* after upgrading to 17.10, not Mir. That must've been a crossover from an alternate universe where Ubuntu went with Mir :)

[Dominik George (natureshadow)]

I can confirm the issue on a Lenovo T470s. However, it runs 16.04 LTS, but with the same kernel package.

Disabling lightdm (and thus startup of X does not fix the issue here, but the rest is identical.

[Dominik George (natureshadow)]

I have to add: We have quite a few laptops of this kind that show the issue, but one doesn't. It is the same hardware and has the same firmware versions.

[Dominik George (natureshadow)]

Another update: 4.4.0-112 generic from 16.04 shows the same issue. I do not know how to mark that in Launchpad.

[Dominik George (natureshadow)]

I verified again that it is not X. The hang also occurs when booting into multi-user.target and even after purging X.org.

I felt adventorous and did the following:

I booted into rescue mode. Then I let a shell loop start every service in /lib/systemd/system with 1 sec delay. All service came up without issues and I got the system to the same state as multi-user.target would.

[Joseph Salisbury (jsalisbury)]

@Sandra Ray, can you give 4.14-rc1 a try:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14-rc1/

[SkySurfer (mueller-g)]

I can confirm that 4.14-rc1 is working fine on W540

[Sandra Ray (sandraray)]

4.14-rc1 boots ok!

Also, now that Dominik mentions it, I'm not 100% that the hang is happening in X - it could be in any of a number of startup programs.

[Dominik George (natureshadow)]

4.14-rc1 also works here.

Any updates?

[Ingemar Fällman (mrsmurf)]

Could it be the same issue as #1745349? It looks like it is the Spectre fix that causes the issue.
The problem goes away for me on the 4.4.0-112 kernel with the "noibpb" kernel argument on boot.

[FaberfoX (faberfox)]

I've been having this same issue on ubuntu mate 17.10 on a thinkpad T450s since kernel 4.13.0-25, and was booting with previous 4.13.0-21. Just tried #14 suggestion (adding noibpb) and it fixed it here.

[Joseph Salisbury (jsalisbury)]

We now know the fix is in v4.14-rc1. Can folks affected by this bug confirm that v4.13 final is bad? If it is, I'll start a reverse bisect between 4.13 final and v4.14-rc1.

4.13 final is available from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.13/

[SkySurfer (mueller-g)]

I have tested 4.13 final from your link and it boots up.

[Joseph Salisbury (jsalisbury)]

Thanks for testing. That could mean that a SAUCE patch caused this regression. I review the git log. If nothing sticks out, we can perform a regular kernel bisect of the Ubuntu kernels.

[Joseph Salisbury (jsalisbury)]

Before starting the bisect, can you see if the latest Artful kernel still has the bug? It is in the updates as kernel version 4.13.0-36.

[Sandra Ray (sandraray)]

4.13.0-36 hangs the system.

Adding "noibpb" to the kernel command line fixes the problem for me too.

[Joseph Salisbury (jsalisbury)]

I built an Artful test kernel with a revert of commit: 8578993

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1746418

Can you test this kernel and see if it resolves this bug?

Note, to test this kernel, you need to install both the linux-image and linux-image-extra .deb packages.

Thanks in advance!

[Lars Ehrhardt (lehrhardt)]

Hi there,

we tried this new kernel - our test machine still locked up during the boot process while starting services.

[Sandra Ray (sandraray)]

@jsalisbury, sorry - your test kernel still hangs (though it continues to boot with "noibpb" passed as a kernel flag).

Also - do you know if https could be enabled for kernel.ubuntu.com? While it's unlikely to happen, I wouldn't want some evil man in the middle to corrupt those beautiful kernel bits.

[Joseph Salisbury (jsalisbury)]

I built a Artful test kernel with a revert of commit 96d520d. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1746418

Can you test this kernel and see if it resolves this bug?

Note, to test this kernel, you need to install both the linux-image and linux-image-extra .deb packages.

[Sandra Ray (sandraray)]

@jsalisbury: 96d520d boots!

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the proposed kernel and post back if it resolves this bug?
See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed.

Thank you in advance!

[Sandra Ray (sandraray)]

@jsalisbury : linux-image version 4.13.0-38.43 from artful-proposed still hangs. :(

[Joseph Salisbury (jsalisbury)]

This bug is probably a duplicate of bug 1745349 in Xenial.

[Sandra Ray (sandraray)]

For anyone looking for a temporary/unsupported workaround on Thinkpads without disabling the meltdown/spectre protections via noibpb, the mainline kernels (4.14 and 4.15) seem to function.

You can find more information on running them at https://wiki.ubuntu.com/Kernel/MainlineBuilds

[kolya (mar-kolya)]

Looks like I've got this one too on Lenovo P50 with E3-1505M. I've created #1759997 since I was not able to trace it immediately to this bug.

The problem is that I had a fully working system until 3.20180312.0 microcode update arrived - which made my system essentially unbootable. This was an unpleasant surprise and may be a wide spread problem since this microcode is now 'main stream'.

[Gerard Dethier (g-dethier)]

Hi, I am also affected by this bug but not on a Lenovo or Thinkpad laptop. Mine is an Asus Zenbook UX301L running Ubuntu 17.10. Latest working kernel version is 4.13.0-36-generic. Patches 37 and 38 cause the freeze on boot. Adding option noibpb solves the issue (at least with 4.13.0-38-generic).

[Joseph Salisbury (jsalisbury)]

Can those affected by this bug, please test the following kernel:
https://launchpad.net/ubuntu/+source/linux/4.15.0-15.16/+build/14530348

To test the kernel, install both the linux-image and linux-image-extra .deb packages.

[Sandra Ray (sandraray)]

@jsalisbury, it boots!

[Lars Ehrhardt (lehrhardt)]

We can also confirm, that kernel image 4.15.0-15.16 boots on one of our affected machines.

[Sandra Ray (sandraray)]

I was able to successfully boot the latest 17.10 kernel: linux-image-4.13.0-39-generic

This issue is resolved.

Thanks!