A deadlock can occur in zap_pid_ns_processes() which can hang the system due to RCU getting stuck.
zap_pid_ns_processes() has a busy loop that calls kernel_wait4() on a child process of the namespace init task, waiting for it to exit. The problem is, it clears TIF_SIGPENDING, but not TIF_NOTIFY_SIGNAL as well, leading us to get stuck in the busy loop forever, due to the child sleeping in synchronize_rcu(), and is never woken up due to the parent being stuck in the busy loop and never calling schedule() or rcu_note_context_switch().
$ sudo docker run telescope.azurecr.io/issue-repro/zombie:v1.1.11
$ ./rcu-npm-repro.sh
go repro:
$ go mod init rcudeadlock.go
$ go mod tidy
$ CGO_ENABLED=0 go build -o ./rcudeadlock ./
$ sudo ./rcudeadlock
Look at dmesg. After some minutes, you should see the hung task timeout from the impact section.
[Where problems can occur]
We are clearing TIF_NOTIFY_SIGNAL in the child, in order for signal_pending() to return false and not lead us to a busy wait loop.
This change should work as intended.
If a regression were to occur, it could potentially affect all processes in namespaces.
[Other Info]
Upstream mailing list discussion: https://lore<email address hidden>/T/
BugLink: https:/ /bugs.launchpad .net/bugs/
[Impact]
A deadlock can occur in zap_pid_ ns_processes( ) which can hang the system due to RCU getting stuck.
zap_pid_ ns_processes( ) has a busy loop that calls kernel_wait4() on a child process of the namespace init task, waiting for it to exit. The problem is, it clears TIF_SIGPENDING, but not TIF_NOTIFY_SIGNAL as well, leading us to get stuck in the busy loop forever, due to the child sleeping in synchronize_rcu(), and is never woken up due to the parent being stuck in the busy loop and never calling schedule() or rcu_note_ context_ switch( ).
A oops is:
Watchdog: BUG: soft lockup - CPU#3 stuck for 276s! [rcudeadlock:1836] read_lock+ 0xe/0x30 log_lvl+ 0x1d6/0x2ea log_lvl+ 0x1d6/0x2ea wait4+0xaf/ 0x150 part.0+ 0x23/0x29 cold+0x8/ 0xd timer_fn+ 0x1be/0x220 detector_ update_ enable+ 0x60/0x60 run_queues+ 0x107/0x230 clock_tsc_ cs+0x9/ 0x30 interrupt+ 0x101/0x220 isr+0x20/ 0x30 hyperv_ stimer0+ 0x32/0x70 hyperv_ stimer0+ 0x7b/0x90 hyperv_ stimer0+ 0x1b/0x20 lock+0xe/ 0x30 wait4+0xaf/ 0x150 group_exited+ 0x50/0x50 ns_processes+ 0x111/0x1a0 original_ parent+ 0x348/0x360 0x4a/0x210 0x24f/0x3c0 exit+0x3b/ 0xb0 0x150/0x900 signal_ or_restart+ 0xde/0x100 futex+0x78/ 0x1e0 user_mode_ loop+0xc4/ 0x160 user_mode_ prepare+ 0xa3/0xb0 exit_to_ user_mode+ 0x27/0x50 call+0x1022/ 0x1fa0 64+0x63/ 0xb0 add_tctx_ node+0x111/ 0x1a0 io_uring_ enter+0x10d/ 0x540 single_ queue+0x59/ 0x90 user_mode_ prepare+ 0x37/0xb0 exit_to_ user_mode+ 0x2c/0x50 call+0x1819/ 0x1fa0 64+0x63/ 0xb0 wake_up+ 0x200/0x5a0 0x159/0x190 0x162/0x1f0 futex+0x78/ 0x1e0 fpu_return+ 0x4e/0xc0 user_mode_ prepare+ 0x37/0xb0 exit_to_ user_mode+ 0x2c/0x50 call+0x1022/ 0x1fa0 64+0x63/ 0xb0 addr_fault+ 0x1e7/0x670 user_mode_ prepare+ 0x37/0xb0 exit_to_ user_mode+ 0xe/0x20 exit+0x1d/ 0x30 fault+0x89/ 0x170 SYSCALL_ 64_after_ hwframe+ 0x6c/0xd6
CPU: 3 PID: 1836 Comm: rcudeadlock Tainted: G L 5.15.0-117-generic #127-Ubuntu
RIP: 0010:_raw_
Code: f0 0f b1 17 74 08 31 c0 5d c3 cc cc cc cc b8 01 00 00 00 5d c3 cc cc cc cc 0f 1f 00 0f 1f 44 00 00 b8 00 02 00 00 f0 0f c1 07 <a9> ff 01 00 00 75 05 c3 cc cc cc cc 55 48 89 e5 e8 4d 79 36 ff 5d
CR2: 000000c0002b0000
Call Trace:
<IRQ>
? show_trace_
? show_trace_
? kernel_
? show_regs.
? show_regs.
? watchdog_
? lockup_
? __hrtimer_
? read_hv_
? hrtimer_
? hv_stimer0_
? __sysvec_
? sysvec_
</IRQ>
<TASK>
? asm_sysvec_
? _raw_read_
? do_wait+0xa0/0x310
kernel_
? thread_
zap_pid_
forget_
exit_notify+
do_exit+
do_group_
get_signal+
arch_do_
? __x64_sys_
exit_to_
exit_to_
syscall_
? x64_sys_
do_syscall_
? __io_uring_
? fput+0x13/0x20
? __do_sys_
? __smp_call_
? exit_to_
? syscall_
? x64_sys_
? do_syscall_
? try_to_
? wake_up_q+0x50/0x90
? futex_wake+
? do_futex+
? __x64_sys_
? switch_
? exit_to_
? syscall_
? x64_sys_
? do_syscall_
? do_user_
? exit_to_
? irqentry_
? irqentry_
? exc_page_
entry_
</TASK>
There is no known workaround.
[Fix]
This was fixed in the below commit in 6.10-rc5:
commit 7fea700e04bd3f4 24c2d836e984257 82f97b494e ns_processes: clear TIF_NOTIFY_SIGNAL along with TIF_SIGPENDING /git.kernel. org/pub/ scm/linux/ kernel/ git/torvalds/ linux.git/ commit/ ?id=7fea700e04b d3f424c2d836e98 425782f97b494e
Author: Oleg Nesterov <email address hidden>
Date: Sat Jun 8 14:06:16 2024 +0200
Subject: zap_pid_
Link: https:/
This patch has made its way to upstream stable, and is already applied to Ubuntu
kernels.
[Testcase]
There are two possible testcases to reproduce this issue.
This reproducer is courtesy of Rachel Menge, using the reproducers in her github repo:
https:/ /github. com/rlmenge/ rcu-soft- lock-issue- repro
Start a Jammy or Noble VM on Azure, D8sV3 will be plenty.
$ git clone https:/ /github. com/rlmenge/ rcu-soft- lock-issue- repro.git
npm repro:
Install Docker.
$ sudo docker run telescope. azurecr. io/issue- repro/zombie: v1.1.11
$ ./rcu-npm-repro.sh
go repro:
$ go mod init rcudeadlock.go
$ go mod tidy
$ CGO_ENABLED=0 go build -o ./rcudeadlock ./
$ sudo ./rcudeadlock
Look at dmesg. After some minutes, you should see the hung task timeout from the impact section.
[Where problems can occur]
We are clearing TIF_NOTIFY_SIGNAL in the child, in order for signal_pending() to return false and not lead us to a busy wait loop.
This change should work as intended.
If a regression were to occur, it could potentially affect all processes in namespaces.
[Other Info]
Upstream mailing list discussion: /lore<email address hidden>/T/
https:/