Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow.... the SMP kvm guest becomes slow.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Expired
|
Undecided
|
Unassigned | ||
qemu (Ubuntu) |
Expired
|
Medium
|
Unassigned | ||
Bug Description
KVM Host is running a clean 9.10 server install with just qemu-kvm and virt-manager kernel=
uname -a: Linux VMMASTER 2.6.31-17-generic #54-Ubuntu SMP Thu Dec 10 16:20:31 UTC 2009 i686 GNU/Linux
version signature: Ubuntu 2.6.31-
KVM Guest is running a clean 9.10 server install with some userland services (apache/
uname -a: Linux VM1 2.6.31-
version signature: Ubuntu 2.6.31-
KVM guest startup command (as invoked by virt-manager):
/usr/bin/kvm -S -M pc-0.11 -m 1024 -smp 2 -name lexx -uuid [UUID] -monitor unix:/var/
Problem description:
After a while (and high network IO) I see this pop up in my guest dmesg:
hrtimer: interrupt too slow, forcing clock min delta to 215997540 ns
after that the guest becomes very slow to respond, sometimes taking seconds to echo my ssh input back, on a local lan. Only a reboot of the kvm guest fixes this, until the dreaded hrtimer message pops up
After a lot of googling and trying a lot of things I found this discussion on the patchwork kernel mailinglist, which contains a possible solution:
http://
Please look into it, perhaps this solves a lot of kvm-users' problems
I'd like to patch my kvm guests' kernel myself to test this hrtimer patch, do you have the correct procedure for me so i can create a custom kernel?
tags: | added: kernel-series-unknown |
Changed in linux (Ubuntu): | |
importance: | Undecided → Medium |
status: | New → Triaged |
tags: | added: patch |
Changed in kvm (Ubuntu): | |
status: | Triaged → New |
Changed in kvm (Ubuntu): | |
status: | New → Confirmed |
tags: | added: bjf-tracking |
I've applied the attached patch to linux-image- 2.6.31- 16-generic and am running the patched kernel since late last night. I do see the hrtimer messages but with much lower values and even going down a few times (3375->1500 and 5062->3375):
[ 3304.175297] hrtimer: interrupt too slow, forcing clock min delta to 1500 ns
[ 3304.319296] hrtimer: interrupt too slow, forcing clock min delta to 2250 ns
[ 3662.181424] hrtimer: interrupt too slow, forcing clock min delta to 3375 ns
[ 3674.120250] hrtimer: interrupt too slow, forcing clock min delta to 1500 ns
[ 4967.256258] hrtimer: interrupt too slow, forcing clock min delta to 2250 ns
[ 5667.966180] hrtimer: interrupt too slow, forcing clock min delta to 5062 ns
[ 7238.413947] hrtimer: interrupt too slow, forcing clock min delta to 3375 ns
[ 9633.922211] hrtimer: interrupt too slow, forcing clock min delta to 5062 ns
[11509.242590] hrtimer: interrupt too slow, forcing clock min delta to 7593 ns
[17762.000426] hrtimer: interrupt too slow, forcing clock min delta to 7593 ns
[18016.880026] hrtimer: interrupt too slow, forcing clock min delta to 11389 ns
My system is now still very responsive, which never happened before after a night of rsync backup runs.
I'll keep it running for a couple of days and report back