KVM guests freeze on an AMD-based host since it was upgraded to kernel 2.6.32-32

[Seb James]

Binary package hint: qemu-kvm

Since the 31st May I've been seeing random crashes in the virtual machines that we run on an AMD processor based system in our office. We run Ubuntu 10.04 on this server, which is a Dell T105 and I keep it patched with all the updates. The virtual machines we run are all very similar - they're our own build of Linux based around a 2.6.31 kernel.

On 31st of May, from looking at dpkg files, I see that I upgraded the system's kernel to linux-image-2.6.32-32-server.

The crashes occur most commonly when these machines are doing lots of processing, with libpoppler, ghostscript and our own software using most of the cpu time.

I found that if I downgrade the kernel from linux-image-2.6.32-32-server to linux-image-2.6.32-31-server on the Dell T105, then the system is stable.

I tried upgrading the kernel to a prerelease linux-image-2.6.32-33-server but still saw the freezes.

I tried the same virtual machines on two Dell T105 servers (1 dual-core AMD Opteron 1214 processor) and on a Dell R210 server, running an Intel Xeon processor. The crashes can be reliably caused on both of the AMD based machines, but it seems that on the Intel based machine, linux-image-2.6.32-32-server will happily run KVM virtual machines without crashes.

Looks like one of changes in the kernel between 2.6.32-31 and 2.6.32-32 is the cause of this problem.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: qemu-kvm 0.12.3+noroms-0ubuntu9.9
ProcVersionSignature: Ubuntu 2.6.32-32.62-generic 2.6.32.38+drm33.16
Uname: Linux 2.6.32-32-generic x86_64
Architecture: amd64
Date: Sat Jun 18 15:52:52 2011
InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release amd64 (20100429)
KvmCmdLine:
 UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
 root 1934 1 5 178312 201544 0 15:49 ? 00:00:12 /usr/bin/kvm -S -M pc-0.11 -enable-kvm -m 498 -smp 1 -name vpa5900 -uuid 770320e2-2d12-92cf-e5a9-97da88c3fd27 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/vpa5900.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/var/vm/vpa5900/sda.qcow2,if=scsi,index=0,boot=on,format=qcow2,cache=none -net nic,macaddr=52:54:00:61:36:00,vlan=0,model=virtio,name=virtio.0 -net tap,fd=35,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 192.168.0.18:0 -vga std
 root 1986 1 5 179013 204496 0 15:49 ? 00:00:13 /usr/bin/kvm -S -M pc-0.11 -enable-kvm -m 498 -smp 1 -name vpa5901 -uuid 57cff0db-ab19-01fe-a2ea-3e127473c0e2 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/vpa5901.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/var/vm/vpa5901/sda.qcow2,if=scsi,index=0,boot=on,format=qcow2,cache=none -net nic,macaddr=52:54:00:4e:3e:33,vlan=0,model=virtio,name=virtio.0 -net tap,fd=36,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 192.168.0.18:1 -vga std
 root 2117 1 82 182383 337276 0 15:49 ? 00:02:48 /usr/bin/kvm -S -M pc-0.11 -enable-kvm -m 512 -smp 1 -name vpa11000 -uuid 7fd6b400-6ef4-20f4-b1e3-1ecc413ecdf7 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/vpa11000.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/var/vm/vpa11000/sda.qcow2,if=scsi,index=0,boot=on,format=qcow2,cache=none -net nic,macaddr=52:54:00:03:72:ac,vlan=0,model=virtio,name=virtio.0 -net tap,fd=39,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 192.168.0.18:5100 -vga std
Lsusb:
 Bus 002 Device 003: ID 413c:2003 Dell Computer Corp. Keyboard
 Bus 002 Device 002: ID 093a:2510 Pixart Imaging, Inc. Hama Optical Mouse
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Dell Inc. PowerEdge T105
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.32-32-generic root=UUID=a5f141c2-ffe1-4084-aea8-0a071c334561 ro quiet splash
ProcEnviron:
 LANG=en_GB.utf8
 SHELL=/bin/bash
SourcePackage: qemu-kvm
dmi.bios.date: 02/01/2008
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.1.0
dmi.board.name: 0RR825
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 7
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.1.0:bd02/01/2008:svnDellInc.:pnPowerEdgeT105:pvr(none):rvnDellInc.:rn0RR825:rvrA00:cvnDellInc.:ct7:cvr:
dmi.product.name: PowerEdge T105
dmi.product.version: (none)
dmi.sys.vendor: Dell Inc.

[Seb James (sebjames)]

I've just noticed that although I refer to 2.6.32-32-server in my bug report, the Dell T105 for which I submitted the bug was running 2.6.32-32-generic. I have observed the same behaviour for 2.6.32-32-server on our other Dell T105 and confirmed that in each case, the crashes occur for 2.6.32-32 and not for 2.6.32-31.

[Seb James (sebjames)]

A better title for this would be "KVM guests freeze on AMD-based host since upgrade to kernel 2.6.32-32"

[Serge Hallyn (serge-hallyn)]

@Seb,

could you check whether this is a dup of bug 790789 by adding "clock=acpi_pm" to the guest boot command?

---
Ubuntu Bug Squad volunteer triager
http://wiki.ubuntu.com/BugSquad

[Seb James (sebjames)]

Ok, running the guest with clock=acpi_pm added to the Linux command line, whilst the host is running the 2.6.32-32 kernel has given me stable behaviour for 20 minutes with constant CPU usage, so it very much looks like this is a duplicate of bug 790789.