Linux 4.8 as in 16.10 hangs during boot

[Dylan Borg]

I have just upgraded to 16.10 and restarted my machine. The new 4.8 kernel seemed to take ages to boot and appears to have hanged. I have managed to boot the system with the older 4.4 kernels from 16.04 just fine without any problems from systemd etc.. Please fix this so I can stop selecting the kernel manually on each boot.
---
ApportVersion: 2.20.3-0ubuntu8
Architecture: amd64
CurrentDesktop: Unity
DistroRelease: Ubuntu 16.10
HibernationDevice: RESUME=UUID=1147110e-7ff9-41b0-a446-ac19e62a85c2
InstallationDate: Installed on 2016-02-19 (236 days ago)
InstallationMedia: Ubuntu 15.10 "Wily Werewolf" - Release amd64 (20151021)
IwConfig:
 lo no wireless extensions.

 enp4s0 no wireless extensions.

 eno1 no wireless extensions.
MachineType: ASUS All Series
NonfreeKernelModules: nvidia_modeset nvidia
Package: linux (not installed)
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-43-generic root=UUID=c4c28e9f-4a06-4a3d-b8c3-05297050ebca ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 4.4.0-43.63-generic 4.4.21
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-43-generic N/A
 linux-backports-modules-4.4.0-43-generic N/A
 linux-firmware 1.161
RfKill:
 0: hci0: Bluetooth
  Soft blocked: no
  Hard blocked: no
Tags: yakkety
Uname: Linux 4.4.0-43-generic x86_64
UpgradeStatus: Upgraded to yakkety on 2016-10-13 (0 days ago)
UserGroups: adm audio cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 05/04/2016
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 3101
dmi.board.asset.tag: Default string
dmi.board.name: SABERTOOTH X99
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3101:bd05/04/2016:svnASUS:pnAllSeries:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnSABERTOOTHX99:rvrRev1.xx:cvnDefaultstring:ct3:cvrDefaultstring:
dmi.product.name: All Series
dmi.product.version: System Version
dmi.sys.vendor: ASUS

[Dylan Borg (borgdylan)]

Please let me know if I need to collect any system info etc. since I am using an NVIDIA gpu with a manually installed proprietary driver.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1633172

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Dylan Borg (borgdylan)]

I will run the command. Do note that this will be done using the 4.4 kernel, not the 4.8 kernel which I could not boot as stated in the bug report.

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Dylan Borg (borgdylan)]

apport information

[Armin Kunaschik (armin-kunaschik)]

Same thing here in a VMWare virtual machine.
It sometimes starts into X11 (xfce) and hangs after a view seconds.

[Dylan Borg (borgdylan)]

I tried the latest 4.8.1 kernel in the Canonical Kernel Team PPA. That kernel makes no progress w.r.t. this issue. Also using 'nomodeset' on the kernel command line has no effect.

[dino99 (9d9)]

Side note:

That issue has been met during testing; and has been fixed by the latest apparmor package from yakkety-proposed.
Check you really have it upgraded (2.10.95-4ubuntu5.1) and reboot with the 4.8 kernel, it should be fine.

[Dylan Borg (borgdylan)]

I do have that version of apparmor, and did not have any update for apparmor since the upgrade. Do note that I upgraded once the official update prompt was issued. I did not use any beta/alpha version of 16.10.

[Dylan Borg (borgdylan)]

Also note that I have the proposed branch enabled.

[Dylan Borg (borgdylan)]

I tried to see the boot log by removing "quiet splash" from the command line as well as setting the gfxmode to "text". Nothing showed up on the black screen apart from a pulsating "_".

[Joseph Salisbury (jsalisbury)]

Can you test the following two kernels? If 4.7 is good and 4.8-rc1 is bad, we can bisect between those two:

4.7 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.7/
4.8-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8-rc1/

[Joseph Salisbury (jsalisbury)]

If both those kernels are good, we can test later 4.8 release candidates. If they are both bad, we can test some of the 4.7 release candidates.

[Dylan Borg (borgdylan)]

Are those links to apt repositories or do I have to use dpkg manually?

[Dylan Borg (borgdylan)]

After looking at those links, I realise that those debs need manual installation. That is no problem for me and I will be trying them ASAP.

[Armin Kunaschik (armin-kunaschik)]

Installing apparmor 2.10.95-4ubuntu5.1 did not change then problem for me.
Kernel 4.8rc1 hangs too.
Kernel 4.7 worked once for a few minutes and crashed the second time.
See attachment for console output (screenshot from VMWare only)

I remember I saw a similar message once with kernel 4.8.0-22, but unfortunately didn't save it.

[Grief (iamgrief)]

@jsalisbury I can confirm that even 4.7.2 is good, I can boot with it without the subject issue.

Adding yakkety-proposal repo has lead to installation of update for libapparmor package, but that didn't fix the issue.

[Armin Kunaschik (armin-kunaschik)]

I did some more reboots and got me a screenshot from 4.8rc1 crash.
I switch to the first console (Ctrl-Alt-F1) as soon as the graphical login screen appears.
I'll try to catch one for kernel 4.8.0-22. So far there is no output. The cursor at the login prompt simply disappears and the VM hangs.

[Grief (iamgrief)]

@jsalisbury - the issue is CONFIG_HARDENED_USERCOPY_PAGESPAN which is enabled in all kernels in ubuntu kernel ppa starting from 4.8 while it has to be disabled according to this issue: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1627198

I am a little bit lost. Where can I find the repo with the kernel compiled with that option disabled?

[Armin Kunaschik (armin-kunaschik)]

@borgdylan writes in bug 1627198 that he has an i7 6900K. My VM runs on an i5 6600 (in 32 bit mode).
Can this be a Skylake issue?

[Grief (iamgrief)]

I am on i5 but I don't think this is relevant. I am rebuilding the kernel CONFIG_HARDENED_USERCOPY_PAGESPAN, it's pretty slow in VM though...

[Dylan Borg (borgdylan)]

I have a Broadwell-E CPU not Skylar.

[Dylan Borg (borgdylan)]

*Skylake

[Dylan Borg (borgdylan)]

CONFIG_HARDENED_USERCOPY_PAGESPAN is disabled in the Kernel PPA builds of the 4.7.1 kernel. That kernel does not boot for me. So in jmy case setting that to no will not fix it.

[Dylan Borg (borgdylan)]

I meant 4.8.1

[dino99 (9d9)]

It should be usefull to know about the error(s); maybe you can switch to a console (alt + F?) and read the journal to grab these info:

cat /var/log/syslog | less

and also try booting with some kernel boot option, like nomodeset or acpi=off ...
https://help.ubuntu.com/community/BootOptions

[Dylan Borg (borgdylan)]

@jsalisbury Both 4.7 and 4.8-rc1 failed to boot.

[Dylan Borg (borgdylan)]

Also 4.6.7 fails to boot, trying 4.6 next.

[Dylan Borg (borgdylan)]

4.6 and 4.5.7 also fail. Trying 4.5 then giving up.

[Dylan Borg (borgdylan)]

4.5 failed as well. nomodeset will not work since NVIDIA have a non-modesetting driver as well.

[Dylan Borg (borgdylan)]

Side-Note: The NVIDIA driver cannot be installed for 4.8 when running on 4.4 as there is some header issue with a function being missing in DRM.

[Dylan Borg (borgdylan)]

I have patched the NVIDIA driver which now builds and installs for 4.8.0. The boot issue is not fixed even if passing nomodeset while booting.

[Grief (iamgrief)]

@borgdylan How then this can be explained?
> cat /boot/config-4.8.1-040801-generic|grep CONFIG_HARDENED_USERCOPY_PAGESPAN
CONFIG_HARDENED_USERCOPY_PAGESPAN=y

[Dylan Borg (borgdylan)]

Does that kernel boot and is that from 4.8.1 kernel in the Canonical PPA? They say they turned it off in the changelog.

[Dylan Borg (borgdylan)]

In the kernel I have (4.8.1) it is really not set. In the 4.8.0 kernel it is also no set. CONFIG_HARDENED_USERCOPY is set to yes in both cases.

[Dylan Borg (borgdylan)]

Those are from the PPA of Ubuntu kernels, not mainline kernels.

[Grief (iamgrief)]

@borgdylan: Ok, I took the kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8.1/ there are both "ppa" and "mainline" word in the url, that confuses me :) Please tell me where I can grab the kernel with that option disabled?

[Grief (iamgrief)]

Nevermind. I can confirm that with that option (CONFIG_HARDENED_USERCOPY_PAGESPAN) DISABLED it boots well. I recompiled the kernel by myself and the only change was commenting out that option. It works well both in VMware and on bare metal.

[Dylan Borg (borgdylan)]

I managed to boot! Steps are below:
Install kernel 4.8.8-25-generic from https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+packages
Boot into the latest 4.4.0 kernel left over from Ubuntu 16.04
Get the 370.28 driver from NVIDIA (prior drivers lack support for Linux 4.7/4.8)
Install it using `sudo sh <path-to-run-file> -k 4.8.0-25-generic`
Reboot your machine!

[Dylan Borg (borgdylan)]

I meant 4.8.0-25-generic

[Dylan Borg (borgdylan)]

This procedure will not work withe 4.8.0-22-generic that comes with 16.10 by default.

[Dylan Borg (borgdylan)]

The procedure I outlined is only needed if you have an NVIDIA GPU on your system. My case is special since this is a Broadwell-E system i.e. the CPU has no Intel HD Graphics so the NVIDIA GPU is the only GPU attached to the system.

[Armin Kunaschik (armin-kunaschik)]

kernel 4.8.0-25 doesn't work here either. See attached crash message.

I tried booting from the 16.10 desktop install media with the "try" option.
Everything runs fine with the install kernel. What's the difference?

[Grief (iamgrief)]

@armin-kunaschik: Could you please try this kernel? https://www.dropbox.com/sh/pt2vs6jp7c8rqzr/AAA7zm-UUBq4hX4OC-1_qSzUa This is 4.8.1 with disabled CONFIG_HARDENED_USERCOPY_PAGESPAN. Install it with
>sudo dpkg -i *.deb

[Armin Kunaschik (armin-kunaschik)]

@iamgrief: Sorry, I can't, my Test VM is i386.

[Dylan Borg (borgdylan)]

I amusing amd64 here. the kernnel I pointed to has that setting disabled.

[Grief (iamgrief)]

@armin-kunaschik: is it a desktop? I mean I guess that kernel defines is it x32 or x64 and if you install dependent packages for kernel modules, some libraries etc, it would be working as x64. I suppose these dependencies would be automatically installed at the moment of installation of x64 kernel... Maybe you would have to change some settings in VM though...

Anyway, you can compile x32 kernel by yourself, just disable this option

[Armin Kunaschik (armin-kunaschik)]

@iamgrief: "Converting" i386 into amd64 is a real mess (such is written on the Internet).
The various reinstall options in the setup are not changing the behavior, when just some
debs are reinstalled. The VM continues crashing.
The re-installation with formatting the disk works fine.
I also made some clean installs, i386 and amd64 are both running without problems.

So I suspect an upgrade issue... maybe only with i386 installations? Maybe only VMs?

So far I have no clue where to look for a reason. I'll keep the faulty VM... if somebody is interested... but since it's just a test VM, I'm fine with the "solution" to install a new one.

[PlantDaddy (plantdaddy)]

I think I am hitting this as well. Upgraded from 16.04, to 16.10. The kernel that is installed is
4.8.0-26-generic

I also have an nVidia GPU. Some strange behavior I noticed, however. If I boot into the recovery kernel for 4.8.0-26, and choose 'Fix broken packages', it hangs at a certain point (don't recall off-hand where).

But if I throw sigint/^c/ctrl+c, the normal boot process starts and eventually drops me into multi-user mode in xfce and I can work normally, and running 4.8.0-26. So this is a possible work around?

Lenovo ThinkStation C30

:~$ hwinfo --short
cpu:
                       Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz, 2500 MHz
                       Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz, 2500 MHz
                       Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz, 2500 MHz
                       Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz, 2500 MHz
keyboard:
                       IBM NetVista Full Width Keyboard
mouse:
                       Lenovo USB Optical Mouse
graphics card:
                       nVidia GK107GL [Quadro K600]
sound:
                       nVidia GK107 HDMI Audio Controller
                       Intel C600/X79 series chipset High Definition Audio Controller
                       Microsoft LifeChat LX-3000 Headset
storage:
                       Intel C604/X79 series chipset 4-Port SATA/SAS Storage Control Unit
                       Intel C600/X79 series chipset 6-Port SATA AHCI Controller
                       Intel C600/X79 series chipset IDE-r Controller
network:
  eth0 Intel 82579LM Gigabit Network Connection
  enp6s0 Intel Gigabit CT Desktop Adapter
network interface:
  vmnet1 Ethernet network interface
  enp6s0 Ethernet network interface
  vmnet8 Ethernet network interface
  lo Loopback network interface
  eth0 Ethernet network interface
disk:
  /dev/ram11 Disk
  /dev/ram2 Disk
  /dev/ram0 Disk
  /dev/ram9 Disk
  /dev/ram7 Disk
  /dev/ram14 Disk
  /dev/ram5 Disk
  /dev/ram12 Disk
  /dev/ram3 Disk
  /dev/ram10 Disk
  /dev/ram1 Disk
  /dev/sda SEAGATE ST3300657SS
  /dev/ram8 Disk
  /dev/ram15 Disk
  /dev/ram6 Disk
  /dev/ram13 Disk
  /dev/ram4 Disk
partition:
  /dev/sda1 Partition
  /dev/sda2 Partition
  /dev/sda5 Partition
cdrom:
  /dev/sr0 MATSHITA DVD-RAM SW830
usb controller:
                       Renesas uPD720201 USB 3.0 Host Controller
                       Intel C600/X79 series chipset USB2 Enhanced Host Controller #2
                       Intel C600/X79 series chipset USB2 Enhanced Host Controller #1
bios:
                       BIOS
bridge:
                       Intel 82801 PCI Bridge
                       Intel Xeon E7 v2/Xeon E5 v2/Core i7 DMI2
                       Intel Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a
                       Intel Xeon E7 v2/Xeon E5 v2/Core i7 PCI Expre...

[Dylan Borg (borgdylan)]

I notice that you have a Xeon E5. Is this related to -E/-EP CPUS?

[PlantDaddy (plantdaddy)]

To be honest, I am not sure. This is the first kernel related issue I've had with Ubuntu. Is it worth trying the steps your proposed in comment #51 ?

[Dylan Borg (borgdylan)]

Yes make sure your kernel is up to date by booting into 4.4 and install as I had indicated the nvidia driver 370.23.

[PlantDaddy (plantdaddy)]

I couldn't find the 4.8.0-25-generic kernel in that PPA, however. Can I just install the driver through 4.4, then boot back into 4.8.0-26-generic? Or is installing 4.8.0-25-generic a mandatory prerequisite?

[Dylan Borg (borgdylan)]

PPa is no longer needed. Use latest kernel from main archives

[PlantDaddy (plantdaddy)]

That driver made it so I could no longer start X. So neither kernel would fully boot. Time was of the essence, so I just started fresh from Xenial until this bug is fixed.

[Vitaliy (b-daemonlion)]

I'm unsure that I have same issue but seems that yes.
I'm unable to boot latest kernel after upgrade to 16.10, It's just hangs with no disk activity, all latest updates where installed using 4.4.0-59 (it boots properly) but no luck.
I also tried latest nouveau, latest available nvidia driver from driver manager and latest available nvidia driver from nvidia site (375.26) but still unable to boot 4.8 kernel :(
Can anyone help or maybe additional info needed ?

[Dylan Borg (borgdylan)]

Try uninstalling the intel-microcode package. It did the trick for me.

[Vitaliy (b-daemonlion)]

Dylan Borg (borgdylan), thank you for suggestion!

I have AMD platform so no package "intel-microcode" installed, I found and removed "amd64-microcode" package but still have the same behavior...

[Vitaliy (b-daemonlion)]

Also recovery mode of latest kernel boots little more (at least it post some messages before hang), please look 2017_02_03.JPG attached.
After this lines cursor just blinks in lower left corner...

[Vitaliy (b-daemonlion)]

Update: I've tried to pass some boot options to kernel, from all of them that I found 2 helped:

acpi=off - Boots but wrong screen resolution.
acpi=noirq - Works and screen resolution correct.

So at the moment I'm staying with acpi=noirq and latest kernel, I hope this problem will be fixed...

[Kaustav Dey Biswas (kaustav)]

Purging broadcom wifi drivers (dkms) worked for me

[mujjingun (mujjingun)]

Nothing of the above works for me..

[dino99 (9d9)]

That report is against the 4.8 kernel, which is not a supported kernel since a while. Since your system(s) has been upgraded many times, and it should be good to report about right actual config, instead of adding comment on that one.

So closing that report.