Kernel OOPS in 3.13.0-51 due to NULL pointer after kernel upgrade

[Roman Fiedler]

After update, desktop and production machines, both using i386 and amd64 kernels fail.

The crash may only occur after loading of syscall audit rules, hence boot up to initrd is still possible.

Login is still possible but many integral functions fail, e.g. sending signals to processes. Also sudo is not working any more

[ 64.447838] BUG: unable to handle kernel NULL pointer dereference at 000000000000068f
[ 64.448715] IP: [<ffffffff8136cb80>] strlen+0x0/0x30
[ 64.449533] PGD 0
[ 64.450315] Oops: 0000 [#15] SMP
[ 64.451069] Modules linked in: pci_stub vboxpci(OX) vboxnetadp(OX) vboxnetflt(OX) vboxdrv(OX) nf_conntrack_netlink snd_hda_codec_hdmi xt_multiport arc4 xt_hashlimit ipt_REJECT xt_tcpudp xt_NFLOG nfnetlink_log nfnetlink xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle ip_tables x_tables dell_wmi sparse_keymap ppdev dell_laptop dcdbas cdc_mbim cdc_wdm cdc_ncm btusb bluetooth usbnet mii uvcvideo iwlmvm videobuf2_vmalloc videobuf2_memops videobuf2_core mac80211 videodev snd_hda_codec_realtek intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm joydev serio_raw i915 iwlwifi drm_kms_helper snd_hda_intel snd_hda_codec snd_hwdep snd_pcm cfg80211 snd_page_alloc lpc_ich mei_me mei shpchp drm i2c_algo_bit snd_timer snd soundcore wmi parport_pc parport video mac_hid dm_crypt hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse ahci libahci sdhci_pci sdhci e1000e ptp pps_core
[ 64.456682] CPU: 1 PID: 1506 Comm: sudo Tainted: G D OX 3.13.0-51-generic #84-Ubuntu
[ 64.457483] Hardware name: Dell Inc.
[ 64.458281] task: ffff880037251800 ti: ffff8800368c4000 task.ti: ffff8800368c4000
[ 64.459095] RIP: 0010:[<ffffffff8136cb80>] [<ffffffff8136cb80>] strlen+0x0/0x30
[ 64.459901] RSP: 0018:ffff8800368c5df0 EFLAGS: 00010286
[ 64.460707] RAX: ffff8800368c5d40 RBX: 000000000000068f RCX: 0000000000000000
[ 64.461522] RDX: 0000000000000037 RSI: 000000000000068f RDI: 000000000000068f
[ 64.462331] RBP: ffff8800368c5e08 R08: 0000000000000000 R09: 000000000000fffe
[ 64.463141] R10: 0000000000000000 R11: ffff8800368c5c06 R12: ffff8800379c6680
[ 64.463951] R13: 0000000000000000 R14: ffff8802121a8000 R15: 0000000000000000
[ 64.464753] FS: 00007fa9c5d73840(0000) GS:ffff88021ea80000(0000) knlGS:0000000000000000
[ 64.465558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 64.466367] CR2: 000000000000068f CR3: 00000000379b8000 CR4: 00000000001407e0
[ 64.467189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 64.468003] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 64.468803] Stack:
[ 64.469596] ffffffff810f7fda ffff8800379c6680 ffff8802121a8060 ffff8800368c5e78
[ 64.470410] ffffffff810f9581 0000000000000246 ffffffff81c55740 ffff8800368c5e60
[ 64.471217] 0000000000000246 ffff8800368c5ef0 ffff8800368c5e60 ffffffff810f6b93
[ 64.472024] Call Trace:
[ 64.472829] [<ffffffff810f7fda>] ? audit_log_untrustedstring+0x1a/0x30
[ 64.473643] [<ffffffff810f9581>] audit_log_name+0x281/0x320
[ 64.474454] [<ffffffff810f6b93>] ? audit_buffer_free+0x73/0xa0
[ 64.475267] [<ffffffff810fbe37>] audit_log_exit+0x3d7/0xb90
[ 64.476079] [<ffffffff810fe5bf>] __audit_syscall_exit+0x27f/0x2e0
[ 64.476889] [<ffffffff817331e4>] sysret_audit+0x17/0x21
[ 64.477680] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 <80> 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80
[ 64.479363] RIP [<ffffffff8136cb80>] strlen+0x0/0x30
[ 64.480169] RSP <ffff8800368c5df0>
[ 64.480975] CR2: 000000000000068f
[ 64.481784] ---[ end trace 267143d269f88f24 ]---

# cat /proc/version_signature > version.log

failed, but should have reported latest Ubuntu Trusty kernel

# lspci -v

Completely different systems (vmware guest, vbox guest, various hardware). Quite likely not related

$ lsb_release -rd
Description: Ubuntu 14.04.2 LTS
Release: 14.04

$ apt-cache policy linux-image-3.13.0-51-generic
linux-image-3.13.0-51-generic:
  Installed: 3.13.0-51.84
  Candidate: 3.13.0-51.84
  Version table:
 *** 3.13.0-51.84 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main i386 Packages
        500 http://archive.ubuntu.com/ubuntu/ trusty-security/main i386 Packages
        100 /var/lib/dpkg/status

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1451360

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Roman Fiedler (roman-fiedler-deactivatedaccount)]

Isn't apport for reporting of application crashes?

Apart from that sudo fails also to get privileges to access logs. Network, X and many other command line utilities also failing, so I'm not sure how apport could succeed.

Also syslog is crashed, the dmesg log was created manually since dmesg and filesystem write was OK, but sync/unmount/shutdown all fail.

So at the moment I can only try to create file-piped log fragments on a corrupted filesystem and extract them after repair using another kernel.

No log file information apart from that currently available.

[Chris J Arges (arges)]

Roman,
Marked this as a duplicate. We've already identified a fix and it is in the -proposed kernel. (3.13.0-52.85)

[Roman Fiedler (roman-fiedler-deactivatedaccount)]

The fix from 3.13.0-52.85 changes behavior (now SSH login working also after first OOPS) but fix might be incomplete or there are two very similar bugs.

New trace when using proposed:

[ 31.077681] BUG: unable to handle kernel paging request at fffc0000
[ 31.078064] IP: [<c12ff732>] strlen+0x12/0x20
[ 31.078382] *pdpt = 0000000001aa1001 *pde = 0000000001aa8067 *pte = 0000000000000000
[ 31.078996] Oops: 0000 [#10] SMP
[ 31.079287] Modules linked in: nf_conntrack_netlink dm_crypt xt_hashlimit ppdev ipt_REJECT xt_tcpudp xt_NFLOG nfnetlink_log nfnetlink xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle ip_tables x_tables serio_raw parport_pc i2c_piix4 parport mac_hid ahci psmouse libahci pata_acpi
[ 31.081304] CPU: 0 PID: 1155 Comm: lesspipe Tainted: G D 3.13.0-52-generic #85-Ubuntu
[ 31.081304] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 31.081304] task: df195b00 ti: dbfe6000 task.ti: dbfe6000
[ 31.081304] EIP: 0060:[<c12ff732>] EFLAGS: 00010246 CPU: 0
[ 31.081304] EIP is at strlen+0x12/0x20
[ 31.081304] EAX: 00000000 EBX: fffc0000 ECX: ffffffff EDX: fffc0000
[ 31.081304] ESI: dab4a9c0 EDI: fffc0000 EBP: dbfe7e94 ESP: dbfe7e90
[ 31.081304] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 31.081304] CR0: 8005003b CR2: fffc0000 CR3: 1bacc000 CR4: 000006f0
[ 31.081304] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 31.081304] DR6: fffe0ff0 DR7: 00000400
[ 31.081304] Stack:
[ 31.081304] 00000000 dbfe7ea4 c10d4ad5 dab4a9c0 dc3580e0 dbfe7eec c10d5f0c dab4a9c0
[ 31.081304] c1845731 00000002 00000008 00000001 000081ed 00000000 00000000 00000000
[ 31.081304] 00000000 dc358000 00000000 dbfe7eec dc3580e0 dc3581d4 00000003 dbfe7f78
[ 31.081304] Call Trace:
[ 31.081304] [<c10d4ad5>] audit_log_untrustedstring+0x15/0x30
[ 31.081304] [<c10d5f0c>] audit_log_name+0x24c/0x2d0
[ 31.081304] [<c10d830b>] audit_log_exit+0x3ab/0xb30
[ 31.081304] [<c10d9424>] ? audit_filter_rules.isra.5+0x6d4/0xd30
[ 31.081304] [<c10d9afc>] ? audit_filter_syscall+0x7c/0xc0
[ 31.081304] [<c10da5cf>] __audit_syscall_exit+0x21f/0x270
[ 31.081304] [<c101a095>] syscall_trace_leave+0x65/0xb0
[ 31.081304] [<c1180f06>] ? SyS_execve+0x36/0x50
[ 31.081304] [<c165e15e>] syscall_exit_work+0x1a/0x1f
[ 31.081304] Code: f7 be 01 00 00 00 89 f0 48 5e 5d c3 8d b6 00 00 00 00 8d bc 27 00 00 00 00 55 89 e5 57 3e 8d 74 26 00 b9 ff ff ff ff 89 c7 31 c0 <f2> ae b8 fe ff ff ff 29 c8 5f 5d c3 66 90 55 89 e5 57 3e 8d 74
[ 31.081304] EIP: [<c12ff732>] strlen+0x12/0x20 SS:ESP 0068:dbfe7e90
[ 31.081304] CR2: 00000000fffc0000
[ 31.081304] ---[ end trace ff334322094381f6 ]---

[Roman Fiedler (roman-fiedler-deactivatedaccount)]

Sorry, was too fast. The patch works with some audit rulesets, but not all.

And here is also the new OOPS for amd64:

[ 424.803949] BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
[ 424.804735] IP: [<ffffffff8136cbb0>] strlen+0x0/0x30
[ 424.804735] PGD 3905c067 PUD 3b49b067 PMD 0
[ 424.804735] Oops: 0000 [#1] SMP
[ 424.804735] Modules linked in: dm_crypt xt_hashlimit xt_LOG ipt_REJECT xt_tcpudp xt_NFLOG nfnetlink_log nfnetlink xt_conntrack ppdev iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle ip_tables x_tables serio_raw parport_pc i2c_piix4 parport mac_hid psmouse ahci libahci pata_acpi
[ 424.804735] CPU: 0 PID: 1343 Comm: augenrules Not tainted 3.13.0-52-generic #85-Ubuntu
[ 424.804735] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 424.804735] task: ffff8800374b8000 ti: ffff880039206000 task.ti: ffff880039206000
[ 424.804735] RIP: 0010:[<ffffffff8136cbb0>] [<ffffffff8136cbb0>] strlen+0x0/0x30
[ 424.804735] RSP: 0018:ffff880039207d90 EFLAGS: 00010286
[ 424.804735] RAX: ffff880039207d88 RBX: 0000000000000030 RCX: 0000000000000000
[ 424.804735] RDX: 0000000000000036 RSI: 0000000000000030 RDI: 0000000000000030
[ 424.804735] RBP: ffff880039207da8 R08: 0000000000000000 R09: 000000000000fffe
[ 424.804735] R10: 0000000000000000 R11: ffff880039207ba6 R12: ffff8800396ad0c0
[ 424.804735] R13: 0000000000000000 R14: ffff8800392f0c00 R15: 0000000000000001
[ 424.804735] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[ 424.804735] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 424.804735] CR2: 0000000000000030 CR3: 000000003751b000 CR4: 00000000000006f0
[ 424.804735] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 424.804735] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 424.804735] Stack:
[ 424.804735] ffffffff810f7fda ffff8800396ad0c0 ffff8800392f0cc0 ffff880039207e18
[ 424.804735] ffffffff810f9581 0000000000000000 ffffffff00000000 ffff880000000000
[ 424.804735] 0000000000000000 ffff880039207e90 ffff880039207e00 ffffffff810f6b93
[ 424.804735] Call Trace:
[ 424.804735] [<ffffffff810f7fda>] ? audit_log_untrustedstring+0x1a/0x30
[ 424.804735] [<ffffffff810f9581>] audit_log_name+0x281/0x320
[ 424.804735] [<ffffffff810f6b93>] ? audit_buffer_free+0x73/0xa0
[ 424.804735] [<ffffffff810fbe37>] audit_log_exit+0x3d7/0xb90
[ 424.804735] [<ffffffff810fe5bf>] __audit_syscall_exit+0x27f/0x2e0
[ 424.804735] [<ffffffff810212c2>] syscall_trace_leave+0xb2/0x110
[ 424.804735] [<ffffffff8173339f>] int_check_syscall_exit_work+0x34/0x3d
[ 424.804735] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 <80> 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80
[ 424.804735] RIP [<ffffffff8136cbb0>] strlen+0x0/0x30
[ 424.804735] RSP <ffff880039207d90>
[ 424.804735] CR2: 0000000000000030
[ 424.833327] ---[ end trace b570aac2eeb41772 ]---