Ubuntu
linux-hwe-5.15 package

BUG:soft lockup - CPU#0 stuck for 36s! rcu_core_si kernel/rcu/tree.c:2807

Bug #1983436 reported by saltf1sh on 2022-08-03

264

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	linux-hwe-5.13 (Ubuntu)	New	Undecided	Unassigned
	linux-hwe-5.15 (Ubuntu)	New	Undecided	Unassigned

Bug Description

We would like to report the following bug which has been found by our modified version of syzkaller.
rcu_core_si in kernel/rcu/tree.c:2807 in the Linux kernel through 5.13 allows attackers to cause a denial of service (soft lockup) via a large number of different function calls.
description: BUG: soft lockup in rcu_core_si
affected file: kernel/rcu/tree.c
kernel version: 5.13
kernel config, syzkaller reproducer and raw console output are all in the attachments.
======================================================
Crash log:
======================================================
watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [syz-executor.6:14479]
Modules linked in:
CPU: 0 PID: 14479 Comm: syz-executor.6 Not tainted 5.13.19+ #6
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
RIP: 0010:cred_label security/apparmor/include/cred.h:27 [inline]
RIP: 0010:apparmor_cred_free+0x5f/0x1a0 security/apparmor/lsm.c:69
Code: 01 00 00 48 63 1d a1 fd 4d 02 49 03 5c 24 78 48 b8 00 00 00 00 00 fc ff df 48 89 da 48 c1 ea 03 80 3c 02 00 0f 85 08 01 00 00 <4c> 8b 2b 4d 85 ed 74 68 e8 74 53 4b ff be 04 00 00 00 4c 89 ef bb
RSP: 0018:ffff888056609dc8 EFLAGS: 00010246
RAX: dffffc0000000000 RBX: ffff888005c8fc80 RCX: ffffffff967eb4fd
RDX: 1ffff11000b91f90 RSI: 0000000000000100 RDI: ffff888005821000
RBP: ffff888056609de8 R08: 0000000000000001 R09: ffffed1000b04201
R10: ffff888005821003 R11: ffffed1000b04200 R12: ffff888005821000
R13: ffff888005821000 R14: ffff888005821078 R15: ffff888007ba8000
FS: 00007f5f81a40700(0000) GS:ffff888056600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffe8252bb80 CR3: 0000000003cf6006 CR4: 0000000000770ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
<IRQ>
security_cred_free+0x83/0x130 security/security.c:1881
put_cred_rcu+0x71/0x360 kernel/cred.c:115
rcu_do_batch kernel/rcu/tree.c:2559 [inline]
rcu_core+0x536/0x12f0 kernel/rcu/tree.c:2794
rcu_core_si+0xe/0x10 kernel/rcu/tree.c:2807
__do_softirq+0x187/0x576 kernel/softirq.c:559
invoke_softirq kernel/softirq.c:433 [inline]
__irq_exit_rcu kernel/softirq.c:637 [inline]
irq_exit_rcu+0x120/0x150 kernel/softirq.c:649
sysvec_apic_timer_interrupt+0x79/0x90 arch/x86/kernel/apic/apic.c:1100
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:0xffffffffc01e0801
Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 0f 1f 44 00 00 <55> 48 89 e5 53 41 55 31 c0 45 31 ed 48 89 fb b8 ff ff ff 7f 41 5d
RSP: 0018:ffff888006ec7d58 EFLAGS: 00000246
RAX: ffffffffc01e07fc RBX: 000000007fff0000 RCX: ffffffff95ccef6a
RDX: ffff888007ba8000 RSI: ffffc90000763048 RDI: ffff888006ec7e10
RBP: ffff888006ec7eb8 R08: 0000000000000001 R09: ffffed1005374c87
R10: ffff888029ba6437 R11: ffffed1005374c86 R12: ffff888006ec7e10
R13: ffff888029ba6400 R14: ffffc90000763000 R15: dffffc0000000000
</TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 27 Comm: khungtaskd Not tainted 5.13.19+ #6
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
RIP: 0010:check_hung_uninterruptible_tasks kernel/hung_task.c:190 [inline]
RIP: 0010:watchdog+0x1e1/0xa60 kernel/hung_task.c:294
Code: 45 a8 e8 e2 74 fd ff 49 8d 87 40 03 00 00 48 b9 00 00 00 00 00 fc ff df 48 89 45 c0 48 c1 e8 03 80 3c 08 00 0f 85 dd 07 00 00 <49> 8b 9f 40 03 00 00 48 be 00 00 00 00 00 fc ff df 4c 8d 63 10 4c
RSP: 0000:ffff888001d77ea0 EFLAGS: 00010246
RAX: 1ffff11000dea98b RBX: ffff88800669b630 RCX: dffffc0000000000
RDX: ffff888001d6a080 RSI: 0000000000000000 RDI: ffff888005e4c918
RBP: ffff888001d77f00 R08: 0000000000000001 R09: fffffbfff34354d9
R10: ffffffff9a1aa6c7 R11: fffffbfff34354d8 R12: ffff88800669c010
R13: 00000000003fff85 R14: 0000000100021a4b R15: ffff888006f54918
FS: 0000000000000000(0000) GS:ffff888056700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffcc3c83ce8 CR3: 0000000004310004 CR4: 0000000000770ee0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
<TASK>
kthread+0x352/0x430 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
</TASK>
----------------
Code disassembly (best guess):
   0: 01 00 add %eax,(%rax)
   2: 00 48 63 add %cl,0x63(%rax)
   5: 1d a1 fd 4d 02 sbb $0x24dfda1,%eax
   a: 49 03 5c 24 78 add 0x78(%r12),%rbx
   f: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
  16: fc ff df
  19: 48 89 da mov %rbx,%rdx
  1c: 48 c1 ea 03 shr $0x3,%rdx
  20: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
  24: 0f 85 08 01 00 00 jne 0x132
* 2a: 4c 8b 2b mov (%rbx),%r13 <-- trapping instruction
  2d: 4d 85 ed test %r13,%r13
  30: 74 68 je 0x9a
  32: e8 74 53 4b ff callq 0xff4b53ab
  37: be 04 00 00 00 mov $0x4,%esi
  3c: 4c 89 ef mov %r13,%rdi
  3f: bb .byte 0xbb
--

Tags:

Revision history for this message

saltf1sh (saltf1sh) wrote on 2022-08-03:

syzkaller reproducer Edit (518 bytes, text/plain)

Revision history for this message

saltf1sh (saltf1sh) wrote on 2022-08-03:

kernel config Edit (262.0 KiB, text/plain)

Revision history for this message

saltf1sh (saltf1sh) wrote on 2022-08-03:

raw console output Edit (9.0 KiB, text/html)

affects:

linux (Ubuntu) → linux-hwe-5.13 (Ubuntu)

saltf1sh (saltf1sh) on 2022-08-04

information type:

Private Security → Public Security

Revision history for this message

Simon Déziel (sdeziel) wrote on 2022-08-24:

@saltf1sh, thanks for reporting this, however, the 5.13 kernel is no longer supported (since July 2022). Are you able to reproduce the problem on a kernel version that's still supported (like 5.4 or 5.15)?

Revision history for this message

saltf1sh (saltf1sh) wrote on 2022-08-25:

Download full text (9.3 KiB)

@sdeziel,thanks for your prompt. Today I have reproduced the bug in kernel version 5.4 and kernel version 5.15 respectively. The experiment proves that the bug still exists.
I use this command:
"./syz-execprog -executor=./syz-executor -repeat=0 -procs=8 -cover=0 -threaded=0 -collide=0 ./SyzReproRcu".
Crash can still be triggered in the kernel.
You can download CReproducerRcu.c. Locally through "gcc CReproducerRcu.c -o CReproducerRcu" gets the file CReproducerRcu. Run "./CReproducerRcu" locally and wait for about one minute. You can see the message "rcu: INFO: rcu_sched self-detected stall on CPU". I'm not sure if this is a bug.
You can use vm_5dot4.log and vm_5dot15.log to view each step of my operation in QEMU. According to stack trace, the problem may be rcu_sched_clock_irq? I'm not sure which file in the kernel corresponds to the bug.
======================================================
The following are links related to kernel 5.4:
Description: rcu: INFO: rcu_sched self-detected stall on CPU
Kernel version: 5.4
Kernel config: https://github.com/LancyRiver/SoftLockup/blob/master/rcu_core_si/v5.4/.config
C reproducer: https://github.com/LancyRiver/SoftLockup/blob/master/rcu_core_si/v5.4/CReproducerRcu.c
Crash log: https://github.com/LancyRiver/SoftLockup/blob/master/rcu_core_si/v5.4/CrashLog.txt
Syz repro: https://github.com/LancyRiver/SoftLockup/blob/master/rcu_core_si/v5.4/SyzReproRcu
Vm.log: https://github.com/LancyRiver/SoftLockup/blob/master/rcu_core_si/v5.4/vm_5dot4.log

Crash log:
[   65.129421] audit: type=1400 audit(1661428172.369:15): avc:  denied  { execmem } for  pid=360 comm="CReproducerRcu" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=1
[   79.840127] hrtimer: interrupt took 14052 ns
[  104.359596] rcu: INFO: rcu_sched self-detected stall on CPU
[  104.368030] rcu: 	0-....: (20995 ticks this GP) idle=eee/1/0x4000000000000004 softirq=3077/3077 fqs=4694 
[  104.373374] 	(t=21000 jiffies g=4473 q=220481)
[  104.375926] NMI backtrace for cpu 0
[  104.378133] CPU: 0 PID: 12886 Comm: CReproducerRcu Not tainted 5.4.192 #1
[  104.381887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[  104.387042] Call Trace:
[  104.388557]  <IRQ>
[  104.390225]  dump_stack+0x95/0xc8
[  104.392112]  nmi_cpu_backtrace.cold+0x55/0x94
[  104.394758]  ? lapic_can_unplug_cpu+0x70/0x70
[  104.397442]  nmi_trigger_cpumask_backtrace+0x15a/0x1b0
[  104.400354]  rcu_dump_cpu_stacks+0x15d/0x1a7
[  104.402749]  rcu_sched_clock_irq.cold+0x4c8/0x90b
[  104.405510]  ? hrtimer_run_queues+0x1d/0x310
[  104.407876]  update_process_times+0x24/0x60
[  104.410279]  tick_sched_handle+0x10f/0x150
[  104.412622]  tick_sched_timer+0x41/0x120
[  104.414804]  __hrtimer_run_queues+0x308/0x7c0
[  104.417218]  ? tick_sched_do_timer+0x160/0x160
[  104.419750]  ? enqueue_hrtimer+0x230/0x230
[  104.422074]  ? kvm_clock_get_cycles+0xd/0x10
[  104.424437]  ? ktime_get_update_offsets_now+0x17d/0x250
[  104.427363]  hrtimer_interrupt+0x2c9/0x6c0
[  104.429693]  smp_apic_timer_interrupt+0xd4/0x380
[  104.432226]  apic_timer_interrupt+0xf/0x20
[  104.434517] RIP: 0010:clocksource_watchdog+0xa/0x8b0
[  104.437264] Code: 48 c7 c7 48 ed 37 bd e8 34 11 2e 00 e9 74 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 41 54 55 53 <48> 83 ec 50 e8 9d 30 0b 00 48 c7 c7 40 7d ca bd e8 91 e5 22 02 8b
[  104.447427] RSP: 0018:ffff88806d209d88 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[  104.451622] RAX: ffffffffb9e945c0 RBX: ffff88806d209e80 RCX: ffffffffb9e7fe19
[  104.455952] RDX: 0000000000000100 RSI: ffffffffb9e945c0 RDI: ffffffffbdca7d80
[  104.460191] RBP: 0000000000000100 R08: 0000000000000001 R09: ffffed100da45aae
[  104.464058] R10: ffffed100da45aad R11: ffff88806d22d56f R12: ffffffffbdca7d80
[  104.467990] R13: 00000000fffcaa29 R14: 00000000fffcaa29 R15: ffff88806d22d540
[  104.471933]  ? apic_timer_interrupt+0xa/0x20
[  104.474346]  ? __clocksource_unstable+0x170/0x170
[  104.476956]  ? call_timer_fn+0x39/0x2b0
[  104.479082]  ? __clocksource_unstable+0x170/0x170
[  104.481732]  call_timer_fn+0x45/0x2b0
[  104.483781]  ? __clocksource_unstable+0x170/0x170
[  104.486381]  ? __clocksource_unstable+0x170/0x170
[  104.488998]  run_timer_softirq+0x53b/0x1230
[  104.491289]  ? call_timer_fn+0x2b0/0x2b0
[  104.493461]  ? apic_timer_interrupt+0xa/0x20
[  104.495802]  __do_softirq+0x17c/0x627
[  104.497954]  irq_exit+0x114/0x140
[  104.499798]  smp_apic_timer_interrupt+0xde/0x380
[  104.502340]  apic_timer_interrupt+0xf/0x20
[  104.504607]  </IRQ>
======================================================
The following are links related to kernel 5.15:
Description: rcu: INFO: rcu_sched self-detected stall on CPU
Kernel version: hwe-5.15
Kernel config: https://github.com/LancyRiver/SoftLockup/blob/master/rcu_core_si/v5.15/.config
C reproducer: https://github.com/LancyRiver/SoftLockup/blob/master/rcu_core_si/v5.15/CReproducerRcu.c
Crash log: https://github.com/LancyRiver/SoftLockup/blob/master/rcu_core_si/v5.15/CrashLog.txt
Syz repro: https://github.com/LancyRiver/SoftLockup/blob/master/rcu_core_si/v5.15/SyzReproRcu
Vm.log: https://github.com/LancyRiver/SoftLockup/blob/master/rcu_core_si/v5.15/vm_5dot15.log

Crash log:
[  115.490784] audit: type=1400 audit(1661428486.901:15): avc:  denied  { execmem } for  pid=281 comm="CReproducerRcu" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=1
[  132.604057] hrtimer: interrupt took 13982 ns
[  136.826220] rcu: INFO: rcu_sched self-detected stall on CPU
[  136.828843] rcu: 	0-....: (1 GPs behind) idle=ef3/1/0x4000000000000000 softirq=2418/2419 fqs=5209 
[  136.832988] 	(t=21004 jiffies g=3289 q=201640)
[  136.835107] NMI backtrace for cpu 0
[  136.836798] CPU: 0 PID: 291 Comm: CReproducerRcu Not tainted 5.15.39 #1
[  136.839721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[  136.843880] Call Trace:
[  136.845158]  <IRQ>
[  136.846144]  dump_stack_lvl+0x4d/0x66
[  136.848247]  nmi_cpu_backtrace.cold+0xd0/0xd5
[  136.850270]  ? _raw_write_lock_irqsave+0xd0/0xd0
[  136.852437]  ? lapic_can_unplug_cpu+0x70/0x70
[  136.854520]  nmi_trigger_cpumask_backtrace+0x15a/0x1b0
[  136.856961]  rcu_dump_cpu_stacks+0x1b8/0x202
[  136.858971]  rcu_sched_clock_irq.cold+0x1d2/0x417
[  136.860977]  ? hrtimer_run_queues+0x1d/0x310
[  136.863037]  update_process_times+0x11b/0x180
[  136.865024]  ? tick_sched_handle.isra.0+0x150/0x150
[  136.867320]  tick_sched_handle.isra.0+0x105/0x150
[  136.869365]  tick_sched_timer+0xca/0xf0
[  136.871140]  __hrtimer_run_queues+0x2e2/0x6a0
[  136.873114]  ? enqueue_hrtimer+0x1c0/0x1c0
[  136.874967]  ? kvm_clock_get_cycles+0xd/0x10
[  136.876857]  ? ktime_get_update_offsets_now+0x17f/0x240
[  136.879252]  hrtimer_interrupt+0x2c9/0x6c0
[  136.881099]  __sysvec_apic_timer_interrupt+0xc8/0x260
[  136.883367]  sysvec_apic_timer_interrupt+0x65/0x90
[  136.885456]  </IRQ>
[  136.886389]  <TASK>
[  136.887369]  asm_sysvec_apic_timer_interrupt+0x12/0x20
[  136.889669] RIP: 0010:task_function_call+0xf8/0x140
[  136.891991] Code: 44 8b 64 24 38 e8 e8 bd f3 ff 41 83 fc f5 74 b0 e8 dd bd f3 ff 48 b8 00 00 00 00 00 fc ff df 49 01 c5 41 c7 45 00 00 00 00 00 <41> c7 45 08 00 00 00 00 48 8b 44 24 78 65 48 33 04 25 28 00 00 00
[  136.900280] RSP: 0018:ffff888006cd7c30 EFLAGS: 00000282
[  136.902629] RAX: dffffc0000000000 RBX: ffff888005bbba00 RCX: ffffffffac847503
[  136.905834] RDX: ffff888005bbba00 RSI: 0000000000000000 RDI: ffff88800372b008
[  136.908889] RBP: ffffed1000b77749 R08: ffffffffb0c86018 R09: ffffffffb0c8600f
[  136.932729] R10: ffffffffb0c86017 R11: ffffffffb0c8601b R12: 0000000000000000
[  136.936527] R13: ffffed1000d9af86 R14: ffff888005bbba48 R15: dffffc0000000000
[  136.940063]  ? task_function_call+0xe3/0x140
[  136.942475]  ? perf_event_addr_filters_exec+0x300/0x300
[  136.944767]  ? ctx_resched+0x220/0x220
[  136.946522]  ? exclusive_event_installable+0x1a1/0x220
[  136.948839]  perf_install_in_context+0x25f/0x540
[  136.951008]  ? perf_group_attach+0x380/0x380
[  136.952979]  ? mutex_lock+0x89/0xd0
[  136.954676]  ? __mutex_lock_slowpath+0x10/0x10
[  136.956706]  ? exclusive_event_installable+0x1a1/0x220
[  136.959052]  __do_sys_perf_event_open+0x15ba/0x2340
[  136.961345]  ? perf_remove_from_context+0x1a0/0x1a0
[  136.963591]  ? kasan_unpoison+0x23/0x50
[  136.965394]  ? fpregs_assert_state_consistent+0x74/0xb0
[  136.967753]  ? exit_to_user_mode_prepare+0x2f/0x150
[  136.969969]  do_syscall_64+0x3b/0x90
[  136.971778]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  136.974181] RIP: 0033:0x7f2fcb723469
[  136.977080] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ff 49 2b 00 f7 d8 64 89 01 48
[  136.985256] RSP: 002b:00007ffeb8cd4c88 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
[  136.988673] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2fcb723469
[  136.991908] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000080
[  136.995134] RBP: 00007ffeb8cd4ca0 R08: 0000000000000000 R09: 00007ffeb8cd4ca0
[  136.998343] R10: 00000000ffffffff R11: 0000000000000246 R12: 000056542ba00ce0
[  137.001573] R13: 00007ffeb8cd4dc0 R14: 0000000000000000 R15: 0000000000000000
[  137.004773]  </TASK>
======================================================

Report a bug

This report contains Public Security information

Everyone can see this security related information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulinux-hwe-5.15 package

BUG:soft lockup - CPU#0 stuck for 36s! rcu_core_si kernel/rcu/tree.c:2807

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
linux-hwe-5.15 package