[PowerVM] Kernel BUG @ kernel/irq_work.c:157! - 24x7 hw counters
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Low
|
Unassigned | ||
Utopic |
Fix Released
|
Medium
|
Chris J Arges |
Bug Description
[Impact]
Using perf with hv_24x7 events can cause a kernel BUG.
[Fix]
The following upstream commits:
d658972
48bee8a
f34b6c7
ec2aef5
[Test Case]
Steps to recreate the problem:
1. Install Ubuntu 15.04 as a PowerVM guest.
2. Install perf tool
3. Run following scripts to test 24/7 Power8 hardware counter event with perf. tool
=== Script 1
#!/bin/bash
count=0;
offset=0x128
PERF_ARGS="-r 10 -C 0"
while [ $count -lt 100 ]; do
perf stat $PERF_ARGS -x ' ' perf stat $PERF_ARGS -x ' ' -e $EVENT ls
count=)
done
==== Script 2
#!/bin/bash
offset=0;
PERF_ARGS="-r 10 -C 0"
while [ $offset -lt 8192 ]; do
perf stat $PERF_ARGS -x ' ' perf stat $PERF_ARGS -x ' ' -e $EVENT ls
offset=)
done
After few iterations I hit the following BUG.
tt2.sh tt.sh
tt2.sh tt.sh
tt2.sh tt.sh
275679187521558 hv_24x7/
tt2.sh tt.sh
[ 4657.314709] softirq: huh, entered softirq 7 SCHED c00000000010abc0 with preem
pt_count 00000100, exited with bfff0000?
[ 4657.314727] kernel BUG at /build/
[ 4657.314732] Oops: Exception in kernel mode, sig: 5 [#1]
[ 4657.314740] Modules linked in: rtc_generic pseries_rng
[ 4657.314749] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.16.0-25-generic #33-U
[ 4657.314755] task: c000000001375e00 ti: c0000000013d0000 task.ti: c0000000013d0000
[ 4657.314759] NIP: c0000000001e8ffc LR: c00000000001fe70 CTR: c000000000002800ic)
[ 4657.314770] MSR: 8000000000029033 <SF,EE,
[ 4657.314782] CFAR: c00000000001fe6c SOFTE: 0
GPR04: 0000000000000010 00000000009c0000 c000000001424a98 0000000000000002
GPR12: 8000000000009033 c00000000e9a0000 0000000006a3fcd0 0000000000000060
GPR16: 0000000000200000 0000000000000000 c000000000e57c00 0000000000000000
GPR20: c000000001595dca c000000001595478 0000000000000001 000000000000ffff
GPR28: c000000000e40380 c000000000e40300 c0000000013d3590 c000000000e56f08
[ 4657.314832] NIP [c0000000001e8ffc] irq_work_
[ 4657.314841] Call Trace:
4000 (unreliable)
[ 4657.314861] [c0000000013d34f0] [c00000000001ff90] timer_interrupt
[ 4657.314871] [c0000000013d3520] [c000000000002914] decrementer_
[ 4657.314884] --- Exception: 901 at arch_local_
[ 4657.314896] [c0000000013d3810] [c00000000012ed08] vprintk_
[ 4657.314908] [c0000000013d38e0] [c000000000a02650] printk+0x84/0x98
[ 4657.314918] [c0000000013d3910] [c0000000000b51b4] __do_softirq+
[ 4657.314927] [c0000000013d3a00] [c0000000000b57b8] irq_exit+
[ 4657.314948] [c0000000013d3a60] [c000000000002c14] doorbell_
[ 4657.314963] --- Exception: a01 at plpar_hcall_
[ 4657.314963] LR = check_and_
[ 4657.314997] [c0000000013d3df0] [c00000000084077c] cpuidle_
[ 4657.315030] [c0000000013d3f00] [c000000000d63ea8] start_kernel+
[ 4657.315047] Instruction dump:
[ 4657.315052] eba1ffe8 7c0803a6 ebc1fff0 ebe1fff8 4e800020 3c4c011f 3842c110 78290464
[ 4657.315068] 81290014 752a000f 7d380026 55291ffe <0b090000> 4bfffec8 60000000
60000000
[ 4657.315090] ---[ end trace ee202cccd2211e5d ]---
[ 4657.320224]
[ 4657.362675] Unable to handle kernel paging request for data at address 0xc000
000b35515048
[ 4657.362680] Faulting instruction address: 0xc00000000006a37c
[ 4657.362684] Oops: Kernel access of bad area, sig: 11 [#2]
[ 4657.362686] SMP NR_CPUS=2048 NUMA pSeries
[ 4657.362695] CPU: 12 PID: 7 Comm: rcu_sched Tainted: G D 3.16.0-25-
[ 4657.362699] task: c0000000eb581540 ti: c0000000eb604000 task.ti: c0000000eb60
[ 4657.362703] NIP: c00000000006a37c LR: c0000000000865a8 CTR: c00000000006a340
[ 4657.362706] REGS: c0000000eb607800 TRAP: 0300 Tainted: G D (3.16.0-25-generic)
00000000
[ 4657.362718] CFAR: c0000000000865a4 DAR: c000000b35515048 DSISR: 40000000 SOFTE: 0
GPR00: c0000000000865a8 c0000000eb607a80 c0000000013d50f0 00000000013d30d0
GPR08: 0000000000cc0000 c000000b35515000 c00000000e9a0000 0000000000000000
GPR12: c00000000006a340 c00000000e9a6c00 0000000000000000 0000000000000001
GPR20: 0000000000000000 c000000001389700 0000000000000000 0000000000000001
GPR28: c000000001420a68 0000000000000000 00000000013d30d0 0000000000000001
[ 4657.362758] NIP [c00000000006a37c] icp_hv_
[ 4657.362762] LR [c0000000000865a8] pSeries_
[ 4657.362765] Call Trace:
0 (unreliable)
[ 4657.362774] [c0000000eb607af0] [c0000000000865a8] pSeries_
[ 4657.362778] [c0000000eb607b20] [c0000000000426f0] smp_muxed_
0x70/0x90
[ 4657.362783] [c0000000eb607b60] [c0000000000f3a58] resched_
[ 4657.362786] [c0000000eb607b90] [c0000000000f3da0] resched_
[ 4657.362791] [c0000000eb607be0] [c00000000013f170] rcu_implicit_
[ 4657.362795] [c0000000eb607c10] [c00000000013de1c] force_qs_
[ 4657.362799] [c0000000eb607c90] [c0000000001407f0] rcu_gp_
[ 4657.362803] [c0000000eb607d80] [c0000000000e0820] kthread+0x110/0x130
[ 4657.362807] [c0000000eb607e30] [c00000000000a468] ret_from_
[ 4657.362810] Instruction dump:
[ 4657.362812] fbc1fff0 fbe1fff8 f8010010 f821ff91 7c7e1b78 60000000 60000000 3d220008
[ 4657.362818] 39493f00 1d3e0900 e94a0000 7d2a4a14 <abe90048> 7c0004ac 3860006c
7fe4fb78
[ 4657.362825] ---[ end trace ee202cccd2211e5e ]---
[ 4657.365085]
[ 4659.320264] Kernel panic - not syncing: Attempted to kill the idle task!
[ 4659.325500] ---[ end Kernel panic - not syncing: Attempted to kill the idle task!
Backported following 4 commits/patches from upstream[1]:
1. commit d658972
Author: Himangi Saraogi <email address hidden>
Date: Tue Jul 22 23:40:19 2014 +0530
2. commit 48bee8a
Author: Cody P Schafer <email address hidden>
Date: Tue Sep 30 23:03:17 2014 -0700
3. https:/
4. https:/
to the vivid kernel[2]. The problem does not repro.
Will Canonical cherry-pick those commits or should we backport ?
(they apply without conflicts).
[1] The patches 3 and 4 above were posted recently, Powerpc
maintainer plans to merge them.
[2] git://kernel.
===
break-fix: - ec2aef5a8d3c142
break-fix: - f34b6c72c3ebaa2
break-fix: - 48bee8a6c98e343
break-fix: - d6589722846a57a
CVE References
tags: | added: architecture-ppc64le bugnameltc-119744 severity-critical targetmilestone-inin1504 |
affects: | ubuntu → linux (Ubuntu) |
tags: | added: kernel-da-key |
Changed in linux (Ubuntu): | |
assignee: | nobody → Chris J Arges (arges) |
importance: | Undecided → Medium |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Utopic): | |
assignee: | nobody → Chris J Arges (arges) |
Changed in linux (Ubuntu): | |
assignee: | Chris J Arges (arges) → nobody |
Changed in linux (Ubuntu Utopic): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu): | |
importance: | Medium → Undecided |
Changed in linux (Ubuntu Utopic): | |
status: | New → In Progress |
description: | updated |
Changed in linux (Ubuntu): | |
status: | In Progress → Triaged |
description: | updated |
tags: | added: kernel-bug-break-fix |
Changed in linux (Ubuntu): | |
importance: | Undecided → Low |
assignee: | nobody → Andy Whitcroft (apw) |
milestone: | none → ubuntu-15.01 |
Changed in linux (Ubuntu Utopic): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu): | |
status: | Triaged → Confirmed |
Changed in linux (Ubuntu): | |
assignee: | Andy Whitcroft (apw) → nobody |
Changed in linux (Ubuntu): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu): | |
status: | Fix Committed → Confirmed |
tags: | added: verification-done-utopic |
tags: | removed: verification-done-utopic |
tags: | added: verification-done-utopic |
Changed in linux (Ubuntu): | |
status: | Confirmed → Fix Released |
tags: | removed: kernel-bug-break-fix |
Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https:/ /wiki.ubuntu. com/Bugs/ FindRightPackag e. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.
To change the source package that this bug is filed about visit https:/ /bugs.launchpad .net/ubuntu/ +bug/1410519/ +editstatus and add the package name in the text box next to the word Package.
[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]