Hard lockups due to unrestricted lapic timer delay
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Undecided
|
Guilherme G. Piccoli | ||
Xenial |
Fix Released
|
High
|
Guilherme G. Piccoli | ||
Bionic |
Fix Released
|
Low
|
Guilherme G. Piccoli |
Bug Description
[Impact]
* There is a long-time report of an issue with the TSC delay present
in wait_lapic_expire() - basically the guest could have an expiration
timer configured in a way it induces host to wait a long time (with
preemption disabled), so there's a potential scenario for host lockups.
* The stack trace we have access (from an user report of this issue)
is (summarized) below:
NMI watchdog: Watchdog detected hard LOCKUP on cpu 16
[...]
CPU: 16 PID: 3024910 Comm: CPU 0/KVM Not tainted 4.4.0-139-generic #165-Ubuntu
RIP: 0010:[<addr>] [<addr>] delay_tsc+0x20/0x60
[...]
__delay+0x15/0x20
wait_lapic_
vcpu_enter_
kvm_arch_
kvm_vcpu_
do_vfs_
? __do_page_
? fire_user_
SyS_ioctl+0x79/0x90
entry_SYSCALL_
This matches the reported problem in the KVM mailing-list: https:/
* A fix was proposed in the above thread, but discarded in favor of the
following approach: https:/
The patch was merged in Linus tree, hence we hereby request the SRU:
b606f189c7d5 ("KVM: LAPIC: cap __delay at lapic_timer_
There's one additional patch needed, which is just the header adjustment
for exporting a necessary function.
* The patch is missing only in 4.4 kernel series; Bionic (4.15) and the other newer releases have the patch already.
[Test Case]
* Unfortunately this is a hard to reproduce issue; we have reports of
this lockup from an user, hence the SRU request here.
Also, the patch was introduced originally in kernel 4.7, approx. 2.5 years
ago. So, we are confident that community is running this code long enough
without errors reported. Also, checked in the Linus tree and no fixes
for this code were introduced since kernel 4.7.
[Regression Potential]
* The code modification requested here affects the amount of delay in
a specific timer; the patch introduces a maximum time for delay, preventing unbounded delays in host.
The regression potential is considered low, and given the nature of the
modification, latency issues in guests are likely to be the most problematic regression potential we have.
Changed in linux (Ubuntu Bionic): | |
status: | New → Fix Released |
importance: | Undecided → Low |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
Changed in linux (Ubuntu Xenial): | |
status: | New → Confirmed |
importance: | Undecided → High |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
description: | updated |
Changed in linux (Ubuntu Xenial): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu): | |
status: | Confirmed → Fix Released |
SRU request sent to the kernel team mailing list: https:/ /lists. ubuntu. com/archives/ kernel- team/2019- February/ 098872. html