BUG: unable to handle kernel paging request at 00007f29ec101010

[Simon Déziel]

Got this bug/oops while running with the linux-image-4.4.0-113-generic (4.4.0-113.136) kernel from -proposed:

BUG: unable to handle kernel paging request at 00007f29ec101010
IP: [<ffffffff97413ad5>] csum_and_copy_from_iter+0x55/0x4c0
PGD 800002fcef6f9067 PUD 2fd1615e067 PMD 2fcef77e067 PTE 800002fd0009f867
Oops: 0001 [#1] SMP
Modules linked in: binfmt_misc nf_log_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_
 fjes wmi
CPU: 39 PID: 27347 Comm: fping Not tainted 4.4.0-113-generic #136-Ubuntu
Hardware name: Dell Inc. PowerEdge R830/0VVT0H, BIOS 1.3.4 11/09/2016
task: ffff88bd1125e200 ti: ffff897d02850000 task.ti: ffff897d02850000
RIP: 0010:[<ffffffff97413ad5>] [<ffffffff97413ad5>] csum_and_copy_from_iter+0x55/0x4c0
RSP: 0018:ffff897d02853a18 EFLAGS: 00010246
RAX: 0000000000180000 RBX: 0000000000000006 RCX: ffff897d02853e98
RDX: ffff897d02853a94 RSI: 0000000000000040 RDI: ffff8afcfba4ea24
RBP: ffff897d02853a80 R08: 0000000000000000 R09: ffff8afcfba4ea24
R10: ffff8afcfba4ea24 R11: ffff8afcfba4ea00 R12: ffff897d02853e98
R13: 0000000000000000 R14: 1ee730fc3e9fb34c R15: 00007f29ec101008
FS: 00007f29ec282700(0000) GS:ffff8afd7ec40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f29ec101010 CR3: 000002fce5462000 CR4: 0000000000360670
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
 ffff88bd7ec07340 ffffffff9772d1d7 ffff8afcf02d8f00 ffff897d02853aaf
 ffff897d02853a94 00000000000001c0 00000000ffffffff 1ee730fc3e9fb34c
 0000000000000040 ffff8afcf02d8f00 0000000000000000 ffff897d02853d30
Call Trace:
 [<ffffffff9772d1d7>] ? __alloc_skb+0x87/0x1f0
 [<ffffffff97782cb6>] ip_generic_getfrag+0x56/0xe0
 [<ffffffff977abc0f>] raw_getfrag+0xaf/0x100
 [<ffffffff9778450a>] __ip_append_data.isra.45+0x98a/0xb90
 [<ffffffff977abb60>] ? raw_recvmsg+0x1c0/0x1c0
 [<ffffffff977abb60>] ? raw_recvmsg+0x1c0/0x1c0
 [<ffffffff9778478a>] ip_append_data.part.46+0x7a/0xe0
 [<ffffffff97785474>] ip_append_data+0x34/0x40
 [<ffffffff977ac8a4>] raw_sendmsg+0x724/0xc00
 [<ffffffff973a4ea0>] ? aa_sk_perm+0x70/0x210
 [<ffffffff973a5761>] ? aa_sock_msg_perm+0x61/0x150
 [<ffffffff977bc91b>] inet_sendmsg+0x6b/0xa0
 [<ffffffff97723b5e>] sock_sendmsg+0x3e/0x50
 [<ffffffff97724151>] SYSC_sendto+0x101/0x190
 [<ffffffff971b19fb>] ? vm_mmap_pgoff+0xbb/0xe0
 [<ffffffff9706c764>] ? __do_page_fault+0x1b4/0x400
 [<ffffffff97724c7e>] SyS_sendto+0xe/0x10
 [<ffffffff9784df9f>] entry_SYSCALL_64_fastpath+0x1c/0x93
Code: f3 48 0f 47 de 48 85 db 0f 84 8b 01 00 00 8b 02 49 89 f9 49 89 cc 4c 8b 71 08 89 45 c4 8b 01 a8 04 0f 85 79 01 00 00 4c 8b 79 18 <4d> 8b 6f 08 4d 29 f5 49 39 dd 4c 0f 47 eb a8 02 0f 85 36 02 00
RIP [<ffffffff97413ad5>] csum_and_copy_from_iter+0x55/0x4c0
 RSP <ffff897d02853a18>
CR2: 00007f29ec101010
---[ end trace 48102008d03cb384 ]---

This is something new with the -proposed kernel.

Additional information:

$ lsb_release -rd
Description: Ubuntu 16.04.3 LTS
Release: 16.04

$ apt-cache policy linux-image-4.4.0-113-generic
linux-image-4.4.0-113-generic:
  Installed: 4.4.0-113.136
  Candidate: 4.4.0-113.136
  Version table:
 *** 4.4.0-113.136 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-113-generic 4.4.0-113.136
ProcVersionSignature: Ubuntu 4.4.0-113.136-generic 4.4.98
Uname: Linux 4.4.0-113-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Feb 9 10:45 seq
 crw-rw---- 1 root audio 116, 33 Feb 9 10:45 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.15
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: [Errno 2] No such file or directory: 'fuser'
Date: Fri Feb 9 13:41:20 2018
HibernationDevice: RESUME=/dev/mapper/vghost-swap
InstallationDate: Installed on 2017-04-01 (314 days ago)
InstallationMedia: Ubuntu-Server 16.04.2 LTS "Xenial Xerus" - Release amd64 (20170215.8)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
 Bus 002 Device 002: ID 8087:8002 Intel Corp.
 Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub
 Bus 001 Device 002: ID 8087:800a Intel Corp.
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Dell Inc. PowerEdge R830
PciMultimedia:

ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-113-generic.efi.signed root=/dev/mapper/vghost-root ro panic=300 kaslr vsyscall=none
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-113-generic N/A
 linux-backports-modules-4.4.0-113-generic N/A
 linux-firmware 1.157.15
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 11/09/2016
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.3.4
dmi.board.name: 0VVT0H
dmi.board.vendor: Dell Inc.
dmi.board.version: A01
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.3.4:bd11/09/2016:svnDellInc.:pnPowerEdgeR830:pvr:rvnDellInc.:rn0VVT0H:rvrA01:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R830
dmi.sys.vendor: Dell Inc.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Simon Déziel (sdeziel)]

I just hit a similar looking bug/oops why another machine (a laptop this time) also running the -proposed kernel:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000009
IP: [<ffffffffb1413ad5>] csum_and_copy_from_iter+0x55/0x4c0
PGD 0
Oops: 0000 [#1] SMP
Modules linked in: veth xt_CHECKSUM iptable_mangle xt_comment ctr ccm ec_sys bridge stp llc nf_log_ipv6 ip6table_filter ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_nat_ipv6 ip6_tables nf_log_ip
 ghash_clmulni_intel snd psmouse ahci soundcore r8169 mei_me cfg80211 rtsx_pci mii input_leds libahci mei media vhost_net vhost macvtap macvlan kvm_intel i2c_hid kvm intel_lpss_acpi intel_lpss irqbypass a
CPU: 1 PID: 9573 Comm: dnsmasq Tainted: P W O 4.4.0-113-generic #136-Ubuntu
Hardware name: System76 Lemur/Lemur, BIOS 5.12 02/17/2017
task: ffff88078a910000 ti: ffff88083699c000 task.ti: ffff88083699c000
RIP: 0010:[<ffffffffb1413ad5>] [<ffffffffb1413ad5>] csum_and_copy_from_iter+0x55/0x4c0
RSP: 0018:ffff88083699fa18 EFLAGS: 00010246
RAX: 00000000b1729fd0 RBX: 000000000000001c RCX: ffff88083699fe98
RDX: ffff88083699fa94 RSI: 000000000000001c RDI: ffff88077a0b8824
RBP: ffff88083699fa80 R08: 0000000000000000 R09: ffff88077a0b8824
R10: ffff88077a0b8824 R11: ffff88077a0b8800 R12: ffff88083699fe98
R13: 0000000000000000 R14: 00ffffffb1ea6920 R15: 0000000000000001
FS: 00007f8de7b35880(0000) GS:ffff88086ec80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000009 CR3: 000000078af40000 CR4: 0000000000360670
Stack:
 ffff88084e001600 ffffffffb172d1d7 ffff880776b48600 ffff88083699faaf
 ffff88083699fa94 00000000000001c0 00000000ffffffff 09348cbfe217bf5c
 000000000000001c ffff880776b48600 0000000000000000 ffff88083699fd30
Call Trace:
 [<ffffffffb172d1d7>] ? __alloc_skb+0x87/0x1f0
 [<ffffffffb1782cb6>] ip_generic_getfrag+0x56/0xe0
 [<ffffffffb17abc0f>] raw_getfrag+0xaf/0x100
 [<ffffffffb178450a>] __ip_append_data.isra.45+0x98a/0xb90
 [<ffffffffb17abb60>] ? raw_recvmsg+0x1c0/0x1c0
 [<ffffffffb17abb60>] ? raw_recvmsg+0x1c0/0x1c0
 [<ffffffffb178478a>] ip_append_data.part.46+0x7a/0xe0
 [<ffffffffb1785474>] ip_append_data+0x34/0x40
 [<ffffffffb17ac8a4>] raw_sendmsg+0x724/0xc00
 [<ffffffffb13a4ea0>] ? aa_sk_perm+0x70/0x210
 [<ffffffffb13a5761>] ? aa_sock_msg_perm+0x61/0x150
 [<ffffffffb17bc91b>] inet_sendmsg+0x6b/0xa0
 [<ffffffffb1723b5e>] sock_sendmsg+0x3e/0x50
 [<ffffffffb1724151>] SYSC_sendto+0x101/0x190
 [<ffffffffb1729fd0>] ? sock_setsockopt+0x180/0x830
 [<ffffffffb1397072>] ? apparmor_socket_setsockopt+0x22/0x30
 [<ffffffffb1724c7e>] SyS_sendto+0xe/0x10
 [<ffffffffb184df9f>] entry_SYSCALL_64_fastpath+0x1c/0x93
Code: f3 48 0f 47 de 48 85 db 0f 84 8b 01 00 00 8b 02 49 89 f9 49 89 cc 4c 8b 71 08 89 45 c4 8b 01 a8 04 0f 85 79 01 00 00 4c 8b 79 18 <4d> 8b 6f 08 4d 29 f5 49 39 dd 4c 0f 47 eb a8 02 0f 85 36 02 00
RIP [<ffffffffb1413ad5>] csum_and_copy_from_iter+0x55/0x4c0
 RSP <ffff88083699fa18>
CR2: 0000000000000009
---[ end trace f6995f3da4973edf ]---

I'm not sure if those 2 are related or not.

[Simon Déziel (sdeziel)]

The other bug in comment #1 is a different bug (LP: #1748671) so please ignore the comment.

[Joseph Salisbury (jsalisbury)]

Do you have a way to reproduce this bug? If so, we can bisect down to the commit that introduced it.

[Simon Déziel (sdeziel)]

Unfortunately, no and this only occurred once. I'll be testing with 4.4.0-116.140 that includes a fix related to raw_sendmsg so hopefully this will correct the problem.

[Simon Déziel (sdeziel)]

I found a way to trigger the issue with 4.4.0-113.136 and am glad to report that the -proposed kernel 4.4.0-116.140 fixes the issue, thanks!

[Maksim Chumakov (aeromaks)]

Hi,

I've encountered very similar issue on one of our GCE boxes with kernel 4.4.0-111-generic

Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.129348] BUG: unable to handle kernel paging request at ffff87ff364673b0
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.136723] IP: [<ffffffff811e0e4d>] kmem_cache_alloc_trace+0x7d/0x220
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.143470] PGD 0
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.145708] Oops: 0000 [#1] SMP
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.149268] Modules linked in: tcp_diag inet_diag xt_comment xt_tcpudp nf_conntrack_ip
v4 nf_defrag_ipv4 xt_conntrack xt_CT nf_conntrack xt_multiport iptable_raw ip6table_filter ip6_tables iptable_filter ip_tables x_table
s dm_crypt ppdev parport_pc parport 8250_fintek pvpanic input_leds i2c_piix4 mac_hid serio_raw crct10dif_pclmul crc32_pclmul ghash_clm
ulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse virtio_scsi
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.193555] CPU: 2 PID: 27781 Comm: sshd Not tainted 4.4.0-111-generic #134~14.04.1-Ub
untu
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.201928] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Go
ogle 01/01/2011
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.211254] task: ffff8800ba585400 ti: ffff8800b9f8c000 task.ti: ffff8800b9f8c000
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.218834] RIP: 0010:[<ffffffff811e0e4d>] [<ffffffff811e0e4d>] kmem_cache_alloc_trac
e+0x7d/0x220
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.228010] RSP: 0018:ffff8800b9f8faf0 EFLAGS: 00010286
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.233419] RAX: 0000000000000000 RBX: 00000000024000c0 RCX: 0000000000079622
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.240653] RDX: 0000000000079621 RSI: 00000000024000c0 RDI: ffff88020c403d00
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.247922] RBP: ffff8800b9f8fb30 R08: 000000000001a580 R09: ffff88020c403d00
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.255153] R10: ffffffff814dfdcc R11: ffffea0000d79200 R12: 00000000024000c0
Feb 27 11:55:10 userverlua-gce-sc-97 kernel: [54055.262386] R13: ffff8800ab077400 R14: ffff87ff364673b0 R15: ffff88020c403d00
Feb 27 14:21:32 userverlua-gce-sc-97 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="1194" x-info="http://www.rsyslog.c
om"] start
Feb 27 14:21:32 userverlua-gce-sc-97 rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try
http://www.rsyslog.com/e/2307 ]

There is no stack trace in syslog and no any crash reports in /var/crash&. The issue happened only once on Feb 27 at 11:55 UTC and till Feb 27 14:21 the box was in hanged state totally unresponsive.
So at 14:21 it was cold restarted and after restart the box works fine at the moment.

I'm interested if my issue actually related to this bug. If some one could help with this I will really appreciate.