unkillable process (kernel NULL pointer dereference)

[Vyacheslav]

If process that listens socket on any port crashes (segmentation fault) it becomes unkillable.
Kill command does not kill this process.
Port that listen crashed process never be freed.

journalctl shows error:

Feb 13 13:28:09 vbun04 kernel: socktest[1359]: segfault at 21 ip 000055ec3a6bf11e sp 00007ffd88899fb0 error 6 in socktest[55ec3a6bf000+1000]
Feb 13 13:28:09 vbun04 kernel: Code: 04 24 02 00 17 70 89 c5 48 b8 30 30 30 30 30 30 30 30 89 ef 48 89 44 24 08 e8 8e ff ff ff be 0a 00 00 00 89 ef e8 72 ff ff ff <c7> 04 25 21 00 00 00 21 00 00 00 48 8b 44 24 18
Feb 13 13:28:09 vbun04 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000020
Feb 13 13:28:09 vbun04 kernel: #PF: supervisor read access in kernel mode
Feb 13 13:28:09 vbun04 kernel: #PF: error_code(0x0000) - not-present page
Feb 13 13:28:09 vbun04 kernel: PGD 0 P4D 0
Feb 13 13:28:09 vbun04 kernel: Oops: 0000 [#1] SMP NOPTI
Feb 13 13:28:09 vbun04 kernel: CPU: 1 PID: 1359 Comm: socktest Tainted: G OE 5.3.0-29-generic #31-Ubuntu
Feb 13 13:28:09 vbun04 kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Feb 13 13:28:09 vbun04 kernel: RIP: 0010:do_coredump+0x536/0xb30
Feb 13 13:28:09 vbun04 kernel: Code: 00 48 8b bd 18 ff ff ff 48 85 ff 74 05 e8 c2 47 fa ff 65 48 8b 04 25 c0 6b 01 00 48 8b 00 48 8b 7d a0 a8 04 0f 85 65 05 00 00 <48> 8b 57 20 0f b7 02 66 25 00 f0 66 3d 00 80 0f
Feb 13 13:28:09 vbun04 kernel: RSP: 0000:ffffb464c2c5fca8 EFLAGS: 00010246
Feb 13 13:28:09 vbun04 kernel: RAX: 0000000000000000 RBX: ffff9d4b76995100 RCX: 0000000000001afc
Feb 13 13:28:09 vbun04 kernel: RDX: 0000000000000000 RSI: ffffb464c2c5fc68 RDI: 0000000000000000
Feb 13 13:28:09 vbun04 kernel: RBP: ffffb464c2c5fdd8 R08: 0000000000000400 R09: ffffb464c2c5fbe0
Feb 13 13:28:09 vbun04 kernel: R10: ffff9d4b75d01170 R11: 0000000000000000 R12: ffff9d4b7b3df540
Feb 13 13:28:09 vbun04 kernel: R13: 0000000000000001 R14: 0000000000000000 R15: ffffffffb9f15920
Feb 13 13:28:09 vbun04 kernel: FS: 00007f6c91911540(0000) GS:ffff9d4b7db00000(0000) knlGS:0000000000000000
Feb 13 13:28:09 vbun04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 13 13:28:09 vbun04 kernel: CR2: 0000000000000020 CR3: 00000000723ac003 CR4: 00000000000606e0
Feb 13 13:28:09 vbun04 kernel: Call Trace:
Feb 13 13:28:09 vbun04 kernel: ? wake_up_state+0x10/0x20
Feb 13 13:28:09 vbun04 kernel: ? __send_signal+0x1eb/0x3f0
Feb 13 13:28:09 vbun04 kernel: get_signal+0x159/0x880
Feb 13 13:28:09 vbun04 kernel: do_signal+0x34/0x280
Feb 13 13:28:09 vbun04 kernel: ? bad_area+0x47/0x50
Feb 13 13:28:09 vbun04 kernel: exit_to_usermode_loop+0xbf/0x160
Feb 13 13:28:09 vbun04 kernel: prepare_exit_to_usermode+0x77/0xa0
Feb 13 13:28:09 vbun04 kernel: retint_user+0x8/0x8
Feb 13 13:28:09 vbun04 kernel: RIP: 0033:0x55ec3a6bf11e
Feb 13 13:28:09 vbun04 kernel: Code: 04 24 02 00 17 70 89 c5 48 b8 30 30 30 30 30 30 30 30 89 ef 48 89 44 24 08 e8 8e ff ff ff be 0a 00 00 00 89 ef e8 72 ff ff ff <c7> 04 25 21 00 00 00 21 00 00 00 48 8b 44 24 18
Feb 13 13:28:09 vbun04 kernel: RSP: 002b:00007ffd88899fb0 EFLAGS: 00010217
Feb 13 13:28:09 vbun04 kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f6c918424eb
Feb 13 13:28:09 vbun04 kernel: RDX: 0000000000000010 RSI: 000000000000000a RDI: 0000000000000003
Feb 13 13:28:09 vbun04 kernel: RBP: 0000000000000003 R08: 0000000000000000 R09: 00007f6c919331f0
Feb 13 13:28:09 vbun04 kernel: R10: 0000000000000000 R11: 0000000000000217 R12: 000055ec3a6bf150
Feb 13 13:28:09 vbun04 kernel: R13: 00007ffd8889a0b0 R14: 0000000000000000 R15: 0000000000000000
Feb 13 13:28:09 vbun04 kernel: Modules linked in: vboxsf(OE) nls_utf8 isofs vboxvideo(OE) intel_rapl_msr intel_rapl_common crct10dif_pclmul vmwgfx crc32_pclmul ghash_clmulni_intel aesni_intel ttm drm_kms_helper a
Feb 13 13:28:09 vbun04 kernel: CR2: 0000000000000020
Feb 13 13:28:09 vbun04 kernel: ---[ end trace 278d665c8727286a ]---
Feb 13 13:28:09 vbun04 kernel: RIP: 0010:do_coredump+0x536/0xb30
Feb 13 13:28:09 vbun04 kernel: Code: 00 48 8b bd 18 ff ff ff 48 85 ff 74 05 e8 c2 47 fa ff 65 48 8b 04 25 c0 6b 01 00 48 8b 00 48 8b 7d a0 a8 04 0f 85 65 05 00 00 <48> 8b 57 20 0f b7 02 66 25 00 f0 66 3d 00 80 0f
Feb 13 13:28:09 vbun04 kernel: RSP: 0000:ffffb464c2c5fca8 EFLAGS: 00010246
Feb 13 13:28:09 vbun04 kernel: RAX: 0000000000000000 RBX: ffff9d4b76995100 RCX: 0000000000001afc
Feb 13 13:28:09 vbun04 kernel: RDX: 0000000000000000 RSI: ffffb464c2c5fc68 RDI: 0000000000000000
Feb 13 13:28:09 vbun04 kernel: RBP: ffffb464c2c5fdd8 R08: 0000000000000400 R09: ffffb464c2c5fbe0
Feb 13 13:28:09 vbun04 kernel: R10: ffff9d4b75d01170 R11: 0000000000000000 R12: ffff9d4b7b3df540
Feb 13 13:28:09 vbun04 kernel: R13: 0000000000000001 R14: 0000000000000000 R15: ffffffffb9f15920
Feb 13 13:28:09 vbun04 kernel: FS: 00007f6c91911540(0000) GS:ffff9d4b7db00000(0000) knlGS:0000000000000000
Feb 13 13:28:09 vbun04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 13 13:28:09 vbun04 kernel: CR2: 0000000000000020 CR3: 00000000723ac003 CR4: 00000000000606e0

Steps to Reproduce:

Set kernel.core_pattern sysctl to "|" to disable coredumps.

$ sudo sysctl kernel.core_pattern="|"

Compile and run following code:

#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h>

int main()
{
    int listenfd = 0;
    struct sockaddr_in serv_addr;

    listenfd = socket(AF_INET, SOCK_STREAM, 0);
    memset(&serv_addr, '0', sizeof(serv_addr));

    serv_addr.sin_family = AF_INET;
    serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    serv_addr.sin_port = htons(6000);

    bind(listenfd, (struct sockaddr*)&serv_addr, sizeof(serv_addr));

    listen(listenfd, 10);

    *(int*)33 = 33;

    return 0;
}

Process segfaulted and becomes unkillable, opened socket ports busy.

ProblemType: Bug
DistroRelease: Ubuntu 19.10
Package: linux-image-generic 5.3.0.29.33
ProcVersionSignature: Ubuntu 5.3.0-29.31-generic 5.3.13
Uname: Linux 5.3.0-29-generic x86_64
AlsaDevices:
 total 0
 crw-rw----+ 1 root audio 116, 1 фев 13 13:24 seq
 crw-rw----+ 1 root audio 116, 33 фев 13 13:24 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.11-0ubuntu8.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDesktop: KDE
Date: Thu Feb 13 13:49:19 2020
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb: Error: [Errno 2] No such file or directory: 'lsusb': 'lsusb'
MachineType: innotek GmbH VirtualBox
PciMultimedia:

ProcFB: 0 svgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.3.0-29-generic root=UUID=cd3a004e-4e27-4c93-8ae6-e1942680f565 ro mitigations=off ipv6.disable=1 net.ifnames=0 video=1280x960
RelatedPackageVersions:
 linux-restricted-modules-5.3.0-29-generic N/A
 linux-backports-modules-5.3.0-29-generic N/A
 linux-firmware 1.183.3
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog:

dmi.bios.date: 12/01/2006
dmi.bios.vendor: innotek GmbH
dmi.bios.version: VirtualBox
dmi.board.name: VirtualBox
dmi.board.vendor: Oracle Corporation
dmi.board.version: 1.2
dmi.chassis.type: 1
dmi.chassis.vendor: Oracle Corporation
dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr:
dmi.product.family: Virtual Machine
dmi.product.name: VirtualBox
dmi.product.version: 1.2
dmi.sys.vendor: innotek GmbH

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Matthew Ruffell (mruffell)]

I cannot reproduce this. I compiled the reproducer program and tested on a Eoan VM running in KVM, with the same 5.3.0-29-generic kernel:

ubuntu@ubuntu:~$ ./socktest
Segmentation fault (core dumped)
ubuntu@ubuntu:~$ tail /var/log/kern.log
Feb 14 04:00:41 ubuntu kernel: [ 134.951620] socktest[1598]: segfault at 21 ip 000055892e4cb2a3 sp 00007ffed8905de0 error 6 in socktest[55892e4cb000+1000]
Feb 14 04:00:41 ubuntu kernel: [ 134.951634] Code: 48 8d 4d e0 8b 45 dc ba 10 00 00 00 48 89 ce 89 c7 e8 71 fe ff ff 8b 45 dc be 0a 00 00 00 89 c7 e8 52 fe ff ff b8 21 00 00 00 <c7> 00 21 00 00 00 b8 00 00 00 00 48 8b 4d f8 64 48 33 0c 25 28 00
ubuntu@ubuntu:~$ ps ax | grep socktest
 1602 pts/0 S+ 0:00 grep --color=auto socktest
ubuntu@ubuntu:~$ uname -rv
5.3.0-29-generic #31-Ubuntu SMP Fri Jan 17 17:27:26 UTC 2020

The segfault happens as expected, but there is no null pointer dereference and no stack trace in dmesg. The process is terminated and I can still bind to port 6000 with netcat.

I see that you have Virtual Box kernel modules loaded. Can you try reproduce this on a fresh Eoan VM with no Virtual Box drivers installed?

[Vyacheslav (galdralag)]

After previous comment I checked my system settings and find what affect to this bug.
I usually disable system core dump in all linux distributions using kernel.core_pattern=| in sysctl.conf

When kernel.core_pattern has default all ok.

So kernel.core_pattern=| cause this bug. It occurs since kernel 5.3

[Matthew Ruffell (mruffell)]

Yes, you are absolutely correct. I can reproduce this now, when kernel.core_pattern is set to "|".

I can also confirm that the first kernel that this is broken in is 5.3, as it works fine in 5.2 and below.

I will look into this and hopefully get this fixed for you. Thanks for reporting!

Engineering Notes:

RIP: 0010:do_coredump+0x536/0xb30

$ eu-addr2line do_coredump+0x536 -e vmlinux-5.3.0-29-generic
/build/linux-OZAq_R/linux-5.3.0/include/linux/fs.h:2841:7

This is the function file_start_write():

2839 static inline void file_start_write(struct file *file)
2840 {
2841 if (!S_ISREG(file_inode(file)->i_mode))
2842 return;
2843 __sb_start_write(file_inode(file)->i_sb, SB_FREEZE_WRITE, true);
2844 }

This is called from do_coredump():

565 void do_coredump(const kernel_siginfo_t *siginfo)
566 {
...
788 if (!dump_interrupted()) {
789 file_start_write(cprm.file);
790 core_dumped = binfmt->core_dump(&cprm);
791 file_end_write(cprm.file);
792 }
...
810 }

On kernels 5.2 and before, kernel.core_pattern=| normally outputs:

[ 39.328638] Core dump to | pipe failed

This is output is from the pipe section of do_coredump():

565 void do_coredump(const kernel_siginfo_t *siginfo)
566 {
...
623 ispipe = format_corename(&cn, &cprm, &argv, &argc);
624
625 if (ispipe) {
...
681 sub_info = call_usermodehelper_setup(helper_argv[0],
682 helper_argv, NULL, GFP_KERNEL,
683 umh_pipe_setup, NULL, &cprm);
684 if (sub_info)
685 retval = call_usermodehelper_exec(sub_info,
686 UMH_WAIT_EXEC);
687
688 kfree(helper_argv);
689 if (retval) {
690 printk(KERN_INFO "Core dump to |%s pipe failed\n",
691 cn.corename);
692 goto close_fail;
693 }
...
810 }

With kernel.core_pattern=|, format_corename() still interprets this as the pipe path with:

191 static int format_corename(struct core_name *cn, struct coredump_params *cprm,
192 size_t **argv, int *argc)
193 {
...
196 int ispipe = (*pat_ptr == '|');
...
335 return ispipe;
336 }

Which checks the first byte of the core_pattern string to see if it is a "|".

This means we really want the call to call_usermodehelper_exec() in do_coredump() to fail, so we take the if(retval) step and output "Core dump to |%s pipe failed\n".

Note that cn.corename is NULL when kernel.core_pattern=|.

For some reason with 5.3 and later, this call to call_usermodehelper_exec() is not failing, and execution continues on and breaks at file_start_write(cprm.file); at the end of do_coredump().

call_usermodehelper_exec() calls umh_pipe_setup() and creates the pipe used. Something has likely changed in the pipe subsystem.

I will update when I figure out what.

[Matthew Ruffell (mruffell)]

The regression was introduced by the following commit:

commit 315c69261dd3fa12dbc830d4fa00d1fad98d3b03
Author: Paul Wise <email address hidden>
Date: Fri Aug 2 21:49:05 2019 -0700
Subject: coredump: split pipe command whitespace before expanding template

You can read it here:
https://github.com/torvalds/linux/commit/315c69261dd3fa12dbc830d4fa00d1fad98d3b03

This landed in 5.3-rc3, and is still in latest mainline. Reading through the commit message, the linux-fsdevel message and the debian bug found in the commit message, it just seems that the new changes omit the NULL case, and construct the helper_argv[] array slightly differently than it did before.

I confirmed this commit is the root cause by reverting this commit on a v5.3 build and seeing the old correct behaviour return.

I'll try write a patch to fix this, or I will contact the upstream maintainer for a fix in the coming days.

[Matthew Ruffell (mruffell)]

I managed to figure out what was going on, and made a patch to fix the problem. I sent it upstream for feedback:

Cover Letter:
https://lore<email address hidden>/

Patch:
https://lore<email address hidden>/

I will update when upstream responds.

[Matthew Ruffell (mruffell)]

Sudip Mukherjee made a more elegant fix than I did, and he posted it to the fsdevel mailing list:

https://lore.<email address hidden>/

I got an email saying that this has now been pulled into the -mm tree.

I will update when this patch reaches mainline. After that, I will backport it to the Ubuntu kernels.

[Matthew Ruffell (mruffell)]

Sudip's commit landed in mainline last week, in time for 5.7-rc3:

commit db973a7289dad24e6c017dcedc6aee886579dc3a
Author: Sudip Mukherjee <email address hidden>
Date: Mon Apr 20 18:14:20 2020 -0700
Subject: coredump: fix null pointer dereference on coredump

As of this morning, it has been queued up in Greg-KH's upstream stable tree, for release in 5.4.36 and 5.6.8.

The kernel team will likely pull the patch in the next time they pull in upstream -stable patches for Eoan and Focal.

[Matthew Ruffell (mruffell)]

The commit has landed in 5.3.0-56-generic on eoan and 5.4.0-38-generic on focal, currently in -proposed.

Verification for each kernel:

I set the sysctl kernel.core_pattern to "|" with:

$ sudo sysctl kernel.core_pattern="|"

I then compiled the following reproducer:

int main()
{
    *(int*)33 = 33;

    return 0;
}

When running the program, we get a segfault, and the coredump handler kicks in, and we see the following in dmesg:

[ 34.025572] format_corename failed
[ 34.025572] Aborting core

This is a change in the original behaviour, but technically more correct than it used to be, since it simply bails out earlier in the coredump process than before when we confirm we have an invalid core_pattern name string.

The systems no longer get a null pointer dereference in the kernel, and no longer panic. The system keeps working as usual now.

I have confirmed the above on both 5.3.0-56-generic on eoan and 5.4.0-38-generic on focal, so I am happy to mark this as verified.

[Matthew Ruffell (mruffell)]

The fix has been released in the form of 5.4.0-40-generic on focal, and 5.3.0-62-generic on Eoan.