BUG: soft lockup - CPU#2 stuck for 22s! [systemd-udevd:546]

[Alin Andrei]

I'm using Ubuntu 14.04 Trusty Tahr 64 bit and my laptop (Dell XPS L702x using Nvidia Optimus) can't boot starting with kernel 3.13 (I've tried all three updates: 3.13.0-1-generic, 3.13.0-2-generic and 3.13.0-3-generic). When booting, I get this error: "BUG: soft lockup - CPU#2 stuck for 22s! [systemd-udevd:546]"

Booting the old 3.12 (3.12.0-8-generic) kernel works.

Here's the relevant part from syslog:

Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136613] BUG: soft lockup - CPU#2 stuck for 22s! [systemd-udevd:546]
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136638] Modules linked in: parport_pc ppdev rfcomm bnep uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core btusb videodev bluetooth snd_hda_codec_hdmi snd_hda_codec_realtek binfmt_misc joydev ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 nvidia(POF) xt_conntrack ip6table_filter dell_wmi sparse_keymap ip6_tables nf_conntrack_netbios_ns arc4 nf_conntrack_broadcast snd_hda_intel nf_nat_ftp nf_nat snd_hda_codec nf_conntrack_ftp nf_conntrack intel_rapl iwldvm iptable_filter snd_hwdep x86_pkg_temp_thermal dell_laptop ip_tables intel_powerclamp dcdbas snd_pcm mac80211 x_tables kvm_intel snd_page_alloc kvm snd_seq_midi snd_seq_midi_event snd_rawmidi i915 snd_seq crct10dif_pclmul crc32_pclmul iwlwifi ghash_clmulni_intel snd_seq_device aesni_intel snd_timer aes_x86_64 lrw gf128mul cfg80211 glue_helper ablk_helper drm_kms_helper snd cryptd mei_me drm mei i2c_algo_bit psmouse soundcore wmi lpc_ich serio_raw mac_hid video coretemp lp parport hid_generic usbhid hid microcode ahci r8169 libahci mii
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136678] CPU: 2 PID: 546 Comm: systemd-udevd Tainted: PF W O 3.13.0-3-generic #18-Ubuntu
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136679] Hardware name: Dell Inc. Dell System XPS L702X/0XN71K, BIOS A18 06/18/2012
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136681] task: ffff880423d5e000 ti: ffff880036b4a000 task.ti: ffff880036b4a000
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136682] RIP: 0010:[<ffffffff810d98c8>] [<ffffffff810d98c8>] smp_call_function_many+0x268/0x2d0
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136687] RSP: 0018:ffff880036b4bb78 EFLAGS: 00000202
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136688] RAX: 0000000000000006 RBX: ffff880036b4bb28 RCX: ffff88043e5981d0
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136689] RDX: 0000000000000006 RSI: 0000000000000100 RDI: 0000000000000000
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136690] RBP: ffff880036b4bbc8 R08: ffff88043e495348 R09: 0000000000000004
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136691] R10: ffff88043e495348 R11: 29312e312e312820 R12: ffffffff81362dd4
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136692] R13: ffff880036b4bb40 R14: ffffffff81ec810d R15: ffffffff81a5ef2d
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136693] FS: 00007f192e8ca880(0000) GS:ffff88043e480000(0000) knlGS:0000000000000000
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136695] CR2: 0000000000f76068 CR3: 0000000036935000 CR4: 00000000000407e0
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136697] Stack:
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136697] ffff88043e495368 0000000000015300 0000000000000000 ffffffff81059da0
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136700] 0000010000000001 000000000000011b ffffffff81059da0 0000000000000000
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136702] ffff88043e5d08a8 ffff88043e5d0808 ffff880036b4bbf0 ffffffff810d9a8d
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136704] Call Trace:
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136708] [<ffffffff81059da0>] ? flush_tlb_func+0xc0/0xc0
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136710] [<ffffffff81059da0>] ? flush_tlb_func+0xc0/0xc0
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136712] [<ffffffff810d9a8d>] on_each_cpu+0x2d/0x60
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136715] [<ffffffff8105a1fa>] flush_tlb_kernel_range+0x4a/0x70
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136719] [<ffffffff81182af5>] __purge_vmap_area_lazy+0x335/0x430
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136722] [<ffffffff81182d52>] vm_unmap_aliases+0x162/0x180
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136724] [<ffffffff81056427>] change_page_attr_set_clr+0xb7/0x470
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136726] [<ffffffff8171134c>] ? notifier_call_chain+0x4c/0x70
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136728] [<ffffffff81056823>] set_memory_x+0x43/0x50
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136731] [<ffffffff810db40b>] unset_module_init_ro_nx+0x6b/0x80
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136734] [<ffffffff810dfa9b>] load_module+0x139b/0x1b40
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136736] [<ffffffff810db460>] ? store_uevent+0x40/0x40
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136739] [<ffffffff810e03b6>] SyS_finit_module+0x86/0xb0
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136742] [<ffffffff81715a3f>] tracesys+0xe1/0xe6
Jan 15 14:27:05 ubuntu-desktop kernel: [ 36.136743] Code: d2 e8 5d 00 29 00 3b 05 9f 7e c4 00 89 c2 0f 8d 25 fe ff ff 48 98 49 8b 4d 00 48 03 0c c5 c0 f8 d1 81 f6 41 20 01 74 cb 0f 1f 00 <f3> 90 f6 41 20 01 75 f8 eb be 0f b6 4d d0 48 8b 55 c0 44 89 ef
Jan 15 14:27:24 ubuntu-desktop NetworkManager[1269]: <warn> (eth0): DHCPv4 request timed out.
Jan 15 14:27:27 ubuntu-desktop NetworkManager[1269]: <warn> (eth0): DHCP client pid 1343 didn't exit, will kill it.
Jan 15 14:27:32 ubuntu-desktop kernel: [ 62.429213] nf_conntrack: automatic helper assignment is deprecated and it will be removed soon. Use the iptables CT target to attach helpers instead.
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105866] BUG: soft lockup - CPU#2 stuck for 22s! [systemd-udevd:546]
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105893] Modules linked in: parport_pc ppdev rfcomm bnep uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core btusb videodev bluetooth snd_hda_codec_hdmi snd_hda_codec_realtek binfmt_misc joydev ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 nvidia(POF) xt_conntrack ip6table_filter dell_wmi sparse_keymap ip6_tables nf_conntrack_netbios_ns arc4 nf_conntrack_broadcast snd_hda_intel nf_nat_ftp nf_nat snd_hda_codec nf_conntrack_ftp nf_conntrack intel_rapl iwldvm iptable_filter snd_hwdep x86_pkg_temp_thermal dell_laptop ip_tables intel_powerclamp dcdbas snd_pcm mac80211 x_tables kvm_intel snd_page_alloc kvm snd_seq_midi snd_seq_midi_event snd_rawmidi i915 snd_seq crct10dif_pclmul crc32_pclmul iwlwifi ghash_clmulni_intel snd_seq_device aesni_intel snd_timer aes_x86_64 lrw gf128mul cfg80211 glue_helper ablk_helper drm_kms_helper snd cryptd mei_me drm mei i2c_algo_bit psmouse soundcore wmi lpc_ich serio_raw mac_hid video coretemp lp parport hid_generic usbhid hid microcode ahci r8169 libahci mii
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105945] CPU: 2 PID: 546 Comm: systemd-udevd Tainted: PF W O 3.13.0-3-generic #18-Ubuntu
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105947] Hardware name: Dell Inc. Dell System XPS L702X/0XN71K, BIOS A18 06/18/2012
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105948] task: ffff880423d5e000 ti: ffff880036b4a000 task.ti: ffff880036b4a000
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105950] RIP: 0010:[<ffffffff810d98ce>] [<ffffffff810d98ce>] smp_call_function_many+0x26e/0x2d0
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105955] RSP: 0018:ffff880036b4bb78 EFLAGS: 00000202
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105956] RAX: 0000000000000006 RBX: ffff880036b4bb28 RCX: ffff88043e5981d0
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105957] RDX: 0000000000000006 RSI: 0000000000000100 RDI: 0000000000000000
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105958] RBP: ffff880036b4bbc8 R08: ffff88043e495348 R09: 0000000000000004
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105959] R10: ffff88043e495348 R11: 29312e312e312820 R12: ffffffff81362dd4
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105960] R13: ffff880036b4bb40 R14: ffffffff81ec810d R15: ffffffff81a5ef2d
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105961] FS: 00007f192e8ca880(0000) GS:ffff88043e480000(0000) knlGS:0000000000000000
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105963] CR2: 0000000000f76068 CR3: 0000000036935000 CR4: 00000000000407e0
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105964] Stack:
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105966] ffff88043e495368 0000000000015300 0000000000000000 ffffffff81059da0
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105968] 0000010000000001 000000000000011b ffffffff81059da0 0000000000000000
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105970] ffff88043e5d08a8 ffff88043e5d0808 ffff880036b4bbf0 ffffffff810d9a8d
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105972] Call Trace:
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105978] [<ffffffff81059da0>] ? flush_tlb_func+0xc0/0xc0
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105980] [<ffffffff81059da0>] ? flush_tlb_func+0xc0/0xc0
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105982] [<ffffffff810d9a8d>] on_each_cpu+0x2d/0x60
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105984] [<ffffffff8105a1fa>] flush_tlb_kernel_range+0x4a/0x70
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105988] [<ffffffff81182af5>] __purge_vmap_area_lazy+0x335/0x430
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105991] [<ffffffff81182d52>] vm_unmap_aliases+0x162/0x180
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105993] [<ffffffff81056427>] change_page_attr_set_clr+0xb7/0x470
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105997] [<ffffffff8171134c>] ? notifier_call_chain+0x4c/0x70
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.105999] [<ffffffff81056823>] set_memory_x+0x43/0x50
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.106002] [<ffffffff810db40b>] unset_module_init_ro_nx+0x6b/0x80
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.106005] [<ffffffff810dfa9b>] load_module+0x139b/0x1b40
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.106007] [<ffffffff810db460>] ? store_uevent+0x40/0x40
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.106010] [<ffffffff810e03b6>] SyS_finit_module+0x86/0xb0
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.106013] [<ffffffff81715a3f>] tracesys+0xe1/0xe6
Jan 15 14:27:33 ubuntu-desktop kernel: [ 64.106014] Code: 3b 05 9f 7e c4 00 89 c2 0f 8d 25 fe ff ff 48 98 49 8b 4d 00 48 03 0c c5 c0 f8 d1 81 f6 41 20 01 74 cb 0f 1f 00 f3 90 f6 41 20 01 <75> f8 eb be 0f b6 4d d0 48 8b 55 c0 44 89 ef 48 8b 75 c8 e8 ea
Jan 15 14:27:37 ubuntu-desktop kernel: [ 68.069508] INFO: rcu_sched detected stalls on CPUs/tasks: { 6} (detected by 4, t=15002 jiffies, g=357, c=356, q=0)
Jan 15 14:27:37 ubuntu-desktop kernel: [ 68.069560] sending NMI to all CPUs:
Jan 15 14:27:37 ubuntu-desktop kernel: [ 68.069564] NMI backtrace for cpu 6
Jan 15 14:27:37 ubuntu-desktop kernel: [ 68.069568] CPU: 6 PID: 102 Comm: irq/23-ehci_hcd Tainted: PF W O 3.13.0-3-generic #18-Ubuntu

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-headers-3.13.0-3-generic 3.13.0-3.18
ProcVersionSignature: Ubuntu 3.12.0-8.16-generic 3.12.6
Uname: Linux 3.12.0-8-generic x86_64
ApportVersion: 2.13.1-0ubuntu1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: andrei 5038 F.... pulseaudio
CurrentDesktop: Unity
Date: Wed Jan 15 14:51:37 2014
HibernationDevice: RESUME=UUID=92e5858f-a853-468f-b8fe-c95d3a2f4242
InstallationDate: Installed on 2012-12-30 (381 days ago)
InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
MachineType: Dell Inc. Dell System XPS L702X
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.12.0-8-generic root=UUID=1cd2fc4f-7d99-4c7a-8ea7-6f9a2d4e5960 ro quiet splash threadirqs pcie_aspm=force i915.i915_enable_rc6=1
RelatedPackageVersions:
 linux-restricted-modules-3.12.0-8-generic N/A
 linux-backports-modules-3.12.0-8-generic N/A
 linux-firmware 1.121
SourcePackage: linux
StagingDrivers: zram
UpgradeStatus: Upgraded to trusty on 2013-01-12 (367 days ago)
dmi.bios.date: 06/18/2012
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A18
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: 0XN71K
dmi.board.vendor: Dell Inc.
dmi.board.version: FAB1
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.chassis.version: 0.1
dmi.modalias: dmi:bvnDellInc.:bvrA18:bd06/18/2012:svnDellInc.:pnDellSystemXPSL702X:pvr:rvnDellInc.:rn0XN71K:rvrFAB1:cvnDellInc.:ct8:cvr0.1:
dmi.product.name: Dell System XPS L702X
dmi.sys.vendor: Dell Inc.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

I'd like to perform a bisect to figure out what commit caused this regression. We need to identify the earliest kernel where the issue started happening as well as the latest kernel that did not have this issue.

Can you test the following kernels and report back? We are looking for the first kernel version that exhibits this bug:

v3.12 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.12-trusty/
If v3.12 final does not exhibit the bug, we should move on to testing some of the v3.13 release candidates.

v3.13-rc3: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc3-trusty/

If v3.13-rc3 does not exhibit the bug then test v3.13-rc6:
v3.13-rc6: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc6-trusty/

If v3.13-rc3 does exhibit the bug then test v3.13-rc2:
v3.13-rc2: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc2-trusty

You don't have to test every kernel, just up until the kernel that first has this bug.

Thanks in advance!

[Joseph Salisbury (jsalisbury)]

Also, can you follow the "Boot options" instructions on the following wiki to enable additional output on boot:

https://wiki.ubuntu.com/DebuggingKernelBoot

As mentioned on the wiki, it would be great if you can attach a log file which may have captured any messages you see. If you are unable to capture a log file, a digital photo will work just as well. As a last resort you can even copy messages down by hand.

[Alin Andrei (nilarimogard)]

Hello,

The last working kernel was 3.12.0-8-generic and the first for which the bug started was 3.13.0-1-generic. I'll also try what you said and post the results.

[Alin Andrei (nilarimogard)]

I've tried the kernels you mentioned and:

- v3.12 final worked
- v3.13-rc3 didn't work

The problem is that the bug behavior changed after a couple of reboots. I was trying to take a photo of the error messages but I couldn't get a clear photo and after a few reboots, the error wasn't displayed anymore and the laptop booted until the login screen (LightDM), then it freezes. Once I was able to log in to the Unity desktop and only then the laptop froze.

So basically now the laptop freezes when the login screen shows up when using v3.13-rc3 as well as when using the initial kernels that caused issues (3.13.0-1-generic, 3.13.0-2-generic and 3.13.0-3-generic).

Unfortunately, that means I couldn't take a photo of the old error but I'm attaching the syslog, maybe you can find something helpful there. Let me know how else I can help...

[Joseph Salisbury (jsalisbury)]

Can you give v3.13-rc2 a try:
v3.13-rc2: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc2-trusty

If 3.13-rc2 also has the bug, can you test v3.13-rc1 to see if it also has the bug?
v3.13-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc1-trusty

[Alin Andrei (nilarimogard)]

Both v3.13-rc2 and v3.13-rc1 didn't work. This time I was able to make photos of the errors. I've attached 2 photos to the bug report...

[Joseph Salisbury (jsalisbury)]

Thanks for testing. It sounds like the bug was introduced in 3.13-rc1. I'll start a bisect between v3.12 final and v3.13-rc1.

The v3.13-rc8 kernel is now out. Can you also give that kernel a test to see if it still contains the bug:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc8-trusty/

[Alin Andrei (nilarimogard)]

Thanks!

Unfortunately v3.13-rc8 doesn't work either...

[Alin Andrei (nilarimogard)]

I found out what was causing it: starting with v3.13-rc1, having "threadirqs" under "GRUB_CMDLINE_LINUX_DEFAULT" in the /etc/default/grub configuration file causes the behaviour I mentioned above.

After removing "threadirqs" from /etc/default/grub, the bug doesn't occur any more. I guess this is still a bug in the kernel, right? Because Linux 3.12.0-8 from the Ubuntu repositories works with "threadirqs"...

[Joseph Salisbury (jsalisbury)]

Could be a similar issue reported upstream:
https://lkml.org/lkml/2013/12/31/144

[Joseph Salisbury (jsalisbury)]

I started a kernel bisect between v3.12 final and v3.13-rc1. The kernel bisect will require testing of about 10 - 12 test kernels.

I built the first test kernel, up to the following commit:
5cbb3d216e2041700231bcfc383ee5f8b7fc8b74

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1269404

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[Alin Andrei (nilarimogard)]

The packages you provided do not install successfully. Here's the output (I've tried a couple of time and the result was the same):

sudo dpkg -i *.deb
(Reading database ... 721241 files and directories currently installed.)
Preparing to unpack linux-headers-3.12.0-031200-generic_3.12.0-031200.201401211605_amd64.deb ...
Unpacking linux-headers-3.12.0-031200-generic (3.12.0-031200.201401211605) over (3.12.0-031200.201401211605) ...
Preparing to unpack linux-image-3.12.0-031200-generic_3.12.0-031200.201401211605_amd64.deb ...
Examining /etc/kernel/preinst.d/
run-parts: executing /etc/kernel/preinst.d/intel-microcode 3.12.0-031200-generic /boot/vmlinuz-3.12.0-031200-generic
Done.
Unpacking linux-image-3.12.0-031200-generic (3.12.0-031200.201401211605) over (3.12.0-031200.201401211605) ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.12.0-031200-generic /boot/vmlinuz-3.12.0-031200-generic
run-parts: executing /etc/kernel/postrm.d/zz-extlinux 3.12.0-031200-generic /boot/vmlinuz-3.12.0-031200-generic
P: Checking for EXTLINUX directory... found.
P: Writing config for /boot/vmlinuz-3.13.0-5-generic...
P: Writing config for /boot/vmlinuz-3.13.0-4-generic...
P: Writing config for /boot/vmlinuz-3.12.0-8-generic...
P: Writing config for /boot/vmlinuz-3.12.0-031200-generic...
P: Installing debian theme... done.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.12.0-031200-generic /boot/vmlinuz-3.12.0-031200-generic
dpkg: dependency problems prevent configuration of linux-headers-3.12.0-031200-generic:
 linux-headers-3.12.0-031200-generic depends on linux-headers-3.12.0-031200; however:
  Package linux-headers-3.12.0-031200 is not installed.

dpkg: error processing package linux-headers-3.12.0-031200-generic (--install):
 dependency problems - leaving unconfigured
Setting up linux-image-3.12.0-031200-generic (3.12.0-031200.201401211605) ...
Running depmod.
update-initramfs: deferring update (hook will be called later)
Not updating initrd symbolic links since we are being updated/reinstalled
(3.12.0-031200.201401211605 was configured last, according to dpkg)
Not updating image symbolic links since we are being updated/reinstalled
(3.12.0-031200.201401211605 was configured last, according to dpkg)
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 3.12.0-031200-generic /boot/vmlinuz-3.12.0-031200-generic
run-parts: executing /etc/kernel/postinst.d/dkms 3.12.0-031200-generic /boot/vmlinuz-3.12.0-031200-generic
Error! Could not locate dkms.conf file.
File: does not exist.
ERROR (dkms apport): kernel package linux-headers-3.12.0-031200-generic is not supported
Error! Bad return status for module build on kernel: 3.12.0-031200-generic (x86_64)
Consult /var/lib/dkms/bbswitch/0.8/build/make.log for more information.
ERROR (dkms apport): kernel package linux-headers-3.12.0-031200-generic is not supported
Error! Bad return status for module build on kernel: 3.12.0-031200-generic (x86_64)
Consult /var/lib/dkms/nvidia-331/331.38/build/make.log for more information.
ERROR (dkms apport): kernel package linux-headers-3.12.0-031200-generic is not supported
Error! Bad re...

[Joseph Salisbury (jsalisbury)]

Does the linux-image package install? That is the only one you should need to test. No need to install the headers package.

[Alin Andrei (nilarimogard)]

Yes, it was installed successfully. But unfortunately it doesn't work.

[Joseph Salisbury (jsalisbury)]

By "Doesn't work" do you mean it exhibits the bug, or there is some other issue?

During the bisect, there will be several kernels to test. Some will exhibit this bug and some will not. I update the bisect with which kernels have the bug and which do not, and eventually we will locate the commit that introduced the regression.

[Alin Andrei (nilarimogard)]

Yes, I meant that the bug is still present with this kernel.

[Joseph Salisbury (jsalisbury)]

Thanks for the update.

I built the next test kernel, up to the following commit:
f9efbce6334844c7f8b9b9459f6d7a6fbc2928e0

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1269404

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[Alin Andrei (nilarimogard)]

Unfortunately the bug still occurs with the kernel from your latest comment.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
0b1e73ed225d8f7aeef96b74147215ca8b990dce

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1269404

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[Alin Andrei (nilarimogard)]

Unfortunately the laptop froze with this kernel too, so the bug still occurs.

[Joseph Salisbury (jsalisbury)]

Thanks for testing. During the bisect, we will have some kernels that have the bug and some that don't, which is to be expected. Eventually, we will narrow down to which commit introduced the bug.

I'll update the bisect and build the next kernel. I'll post a link shortly.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
2cb85c0845e5826b789d755c97e13e8d5dc85368

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1269404

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[Alin Andrei (nilarimogard)]

This kernel worked, the bug didn't occur.

[Alin Andrei (nilarimogard)]

Well, I spoke too soon.

With this kernel, the laptop doesn't freeze on boot. But the laptop freezes when I click the restart button under Unity, it happened every time. It doesn't occur with the "sudo reboot" command though.

[Alin Andrei (nilarimogard)]

I investigated further and something is really messed up with this last kernel. Every time I switch to a tty, the laptop freezes and the tty displays a huge list of error messages. And this happens even without "threadirqs". I'm attaching photos of the errors.

[Alin Andrei (nilarimogard)]

I forgot something: those errors occur when switching to a tty AFTER logging in to Unity. If I get to the login screen and then I switch to a tty, the laptop doesn't freeze when switching to the tty and no errors are displayed in the tty. Maybe that's somehow relevant.

[Joseph Salisbury (jsalisbury)]

Is it the same bug when the laptop freezes in comment #28? Do you see the "BUG: soft lockup" message?

[Alin Andrei (nilarimogard)]

No, I didn't see any "soft lockup" messages.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
1071ec7bc2dabd0a9d12a1ae5570f4fd3ba944ca

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1269404

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[Alin Andrei (nilarimogard)]

The bug still occurs, this one freezes too.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
5328f35b1584a9849ffe46afa42018946aa43851

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1269404

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[Alin Andrei (nilarimogard)]

Like the others, this one freezes too when using "threadirqs". So the bug still occurs.

[Joel (ubu6tu)]

I get this too. I thought the error messages were just random (my suspicion was multithreading and IRQs going wild after peeking into the kernel source), but I recognize them here! And, I don\t recognize it on #1275116 so maybe that is not a duplicate of this bug?

Running on 13.10 usb/persistent/installer.
Kernel: Linux ubuntu 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Note that I don\t have a hard crash. While I get errors on bootup and the installer warns me that\s somethings\s wrong, I can still use the OS. For example, I\m writing this from Firefox on the affected machine and things work smoothly.

[perfran (perfran)]

Hi,
I still have this soft lockup bug with kernel 3.13.0-17 (no matter if it's the lowlatency flavour or not, hence not a duplicate of bug #1275116, which seems resolved now).
Joel and Alin, do you still have this bug?
However in my case, this threadirqs is not present in my /etc/default/grub, so that might not be the same bug.
Another clue: I run a dual graphics hardware (sony vaio Z12 laptop, nvidia+intel). The bug does not occur when I set the BIOS to "static" in order to force the nvidia card to run. So it might be related to the intel graphics driver.

Should I open a new bug, or is there any known fix or workaround?

Thanks!

[Joel (ubu6tu)]

How can I test if this still affects me?

I don't currently have Ubuntu installed on the machine, but I can install it or run something from USB (probably easiest as the installer runs ubuntu and also has this bug). I guess I have to have select an experimental kernel somehow?

[David Cohen (dacohen)]

I've got this same issue on Intel Merrifield based device. I managed to workaround with the attached hackish patch.
I commented on lkml here:
http://www.spinics.net/lists/kernel/msg1716965.html

[Joseph Salisbury (jsalisbury)]

Can folks affected by this bug see if the issue still exists with the latest Trusty kernel and updates?

[perfran (perfran)]

It does not occur anymore for me (kernel 3.13.0-21). I couldn't get from which update the issue disappeared but now everything's fine. Thanks !

[David Cohen (dacohen)]

What was the last known kernel affected by this bug? I would like to bisect it and find what patch fixed it.

[TK (biotech)]

I'm currently experiencing this bug on a Dell PowerEdge R430 server, clean install of 14.04.2 (kernel 3.16).

[David Cohen (dacohen)]

You can take a look at this kernel commit. It is the fix for kernel 3.14 stable tree and has the info about the fix applied to Linux master branch too:

commit 4c03d4699182312ed42257834b915492af16022a
Author: Peter Zijlstra <email address hidden>
Date: Wed Apr 9 16:24:47 2014 +0200

    x86,preempt: Fix preemption for i386

    Many people reported preemption/reschedule problems with i386 kernels
    for .13 and .14. After Michele bisected this to a combination of

      3e8e42c69bb ("sched: Revert need_resched() to look at TIF_NEED_RESCHED")
      ded79754754 ("irq: Force hardirq exit's softirq processing on its own stack")

    it finally dawned on me that i386's current_thread_info() was to
    blame.

    When we are on interrupt/exception stacks, we fail to observe the
    right TIF_NEED_RESCHED bit and therefore the PREEMPT_NEED_RESCHED
    folding malfunctions.

    Current upstream fixes this by making i386 behave the same as x86_64
    already did:

      2432e1364bbe ("x86: Nuke the supervisor_stack field in i386 thread_info")
      b807902a88c4 ("x86: Nuke GET_THREAD_INFO_WITH_ESP() macro for i386")
      0788aa6a23cb ("x86: Prepare removal of previous_esp from i386 thread_info structure")
      198d208df437 ("x86: Keep thread_info on thread stack in x86_32")

    However, that is far too much to stuff into -stable. Therefore I
    propose we merge the below patch which uses task_thread_info(current)
    for tif_need_resched() instead of the ESP based current_thread_info().

    This makes sure we always observe the one true TIF_NEED_RESCHED bit
    and things will work as expected again.

[Yorwos81 (yorwos81)]

this just hapenned to me to guest os in virtualbox after cloning , ubuntu host , kali linux guest
kernel in uname -a says
guest 3.18.0-kali3-amd64 #1 SMP Debian 3.18.6-1~kali2

[Yorwos81 (yorwos81)]

after shutdown though it didnt repeat itself it booted normally