qemu windows guest hangs on 100% cpu usage

[yebiaoxiang]

hi:
I have two VM , one is winXP Prefessional SP3 32bit, another one is WindowsServer2008 Enterprise SP2 64bit.
When I hot reboot winXP in guest OS, it'll hangs on progress bar, and all the vcpu thread in qemu is 100% usage.
There are no message in kernel log when it happened, I try to rebuild kvm and add some debug info , I found the cpu exit reason is EXIT_REASON_PAUSE_INSTRUCTION.
It seems like all the vcpu always in spinlock waiting. I'm not sure whether it's qemu's bug or kvm's.
Any help would be appreciated.

How reproducible:
WinXP: seems always.
WinServer2008: rare.

Steps to Reproduce:
winXP: hot reboot the xp guest os, hot reboot is necessary.
WinServer2008: not sure, I didn't do anything, it just happened.

The different between WinXP and WInServer2008:
1. When WinXP hangs, the boot progress bar is rolling, I think that vnc is work fine.
2. When WinServer2008 hangs, the vnc show the last screen and the screen won't change anything include system time.
3. When the VM hangs , if I execute "virsh suspend vm-name" and "virsh resume vm-name", the WinServer2008 will change to normal , and work fine not hangs anymore. But WinXP not change anything, still hangs.

qemu version:
QEMU emulator version 1.5.0, Copyright (c) 2003-2008 Fabrice Bellard

host info:
Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
Ubuntu 12.04 LTS \n \l
Linux cvknode2026 3.13.6 #1 SMP Fri Dec 12 09:17:35 CST 2014 x86_64 x86_64 x86_64 GNU/Linux

 qemu command line (guest OS XP):
root 7124 1178 7.6 7750360 3761644 ? Sl 14:02 435:23 /usr/bin/kvm -name x -S -machine pc-i440fx-1.5,accel=kvm,usb=off,system=windows -cpu qemu64,hv_relaxed,hv_spinlocks=0x2000 -m 6144 -smp 12,maxcpus=72,sockets=12,cores=6,threads=1 -uuid d3832129-f77d-4b21-bbf7-fd337f53e572 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/x.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,clock=vm,driftfix=slew -no-hpet -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device usb-ehci,id=ehci,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/vms/images/sn1-of-ff.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=directsync -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,id=drive-ide0-1-1,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1,bootindex=2 -netdev tap,fd=24,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=0c:da:41:1d:f8:40,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/x.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0 -vnc 0.0.0.0:0 -device VGA,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

 all qemu thread (guest OS XP):
root@cvknode2026:/proc/7124/task# top -d 1 -H -p 7124
top - 14:37:05 up 7 days, 4:07, 1 user, load average: 10.71, 10.90, 10.19
Tasks: 14 total, 12 running, 2 sleeping, 0 stopped, 0 zombie
Cpu(s): 38.8%us, 11.2%sy, 0.0%ni, 50.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49159888k total, 35665128k used, 13494760k free, 436312k buffers
Swap: 8803324k total, 0k used, 8803324k free, 28595100k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P SWAP WCHAN COMMAND
 7130 root 20 0 7568m 3.6g 6628 R 101 7.7 33:43.48 3 3.8g - kvm
 7132 root 20 0 7568m 3.6g 6628 R 101 7.7 33:43.13 1 3.8g - kvm
 7133 root 20 0 7568m 3.6g 6628 R 101 7.7 33:42.70 6 3.8g - kvm
 7135 root 20 0 7568m 3.6g 6628 R 101 7.7 33:42.33 11 3.8g - kvm
 7137 root 20 0 7568m 3.6g 6628 R 101 7.7 33:42.59 17 3.8g - kvm
 7126 root 20 0 7568m 3.6g 6628 R 100 7.7 34:06.76 4 3.8g - kvm
 7127 root 20 0 7568m 3.6g 6628 R 100 7.7 33:44.14 8 3.8g - kvm
 7128 root 20 0 7568m 3.6g 6628 R 100 7.7 33:43.64 13 3.8g - kvm
 7129 root 20 0 7568m 3.6g 6628 R 100 7.7 33:43.64 7 3.8g - kvm
 7131 root 20 0 7568m 3.6g 6628 R 100 7.7 33:44.24 10 3.8g - kvm
 7134 root 20 0 7568m 3.6g 6628 R 100 7.7 33:42.47 12 3.8g - kvm
 7136 root 20 0 7568m 3.6g 6628 R 100 7.7 33:42.16 2 3.8g - kvm
 7124 root 20 0 7568m 3.6g 6628 S 1 7.7 0:30.65 14 3.8g poll_sche kvm
 7139 root 20 0 7568m 3.6g 6628 S 0 7.7 0:01.71 14 3.8g futex_wai kvm

all thread's kernel stack (guest OS XP):
root@cvknode2026:/proc/7124/task# cat 7130/stack
[<ffffffffa02b1fa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7132/stack
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7133/stack
[<ffffffffa02b1fa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7135/stack
[<ffffffffa02b1fa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffa02b6788>] vmx_vcpu_run+0x88/0x760 [kvm_intel]
[<ffffffffa0413aec>] __vcpu_run+0x63c/0xc30 [kvm]
[<ffffffffa0414188>] kvm_arch_vcpu_ioctl_run+0xa8/0x270 [kvm]
[<ffffffffa03fc042>] kvm_vcpu_ioctl+0x512/0x6d0 [kvm]
[<ffffffff811d4326>] do_vfs_ioctl+0x86/0x4f0
[<ffffffff811d4821>] SyS_ioctl+0x91/0xb0
[<ffffffff817610ad>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7137/stack
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7126/stack
[<ffffffffa02b1fa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7127/stack
[<ffffffffa02b74f6>] handle_pause+0x16/0x30 [kvm_intel]
[<ffffffffa02ba0d4>] vmx_handle_exit+0x94/0x8b0 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7128/stack
[<ffffffffa02b1fa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7129/stack
[<ffffffffa02b1fa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7131/stack
[<ffffffffa02b1fa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7134/stack
[<ffffffffa02b74fe>] handle_pause+0x1e/0x30 [kvm_intel]
[<ffffffffa02ba0d4>] vmx_handle_exit+0x94/0x8b0 [kvm_intel]
[<ffffffffa0413aec>] __vcpu_run+0x63c/0xc30 [kvm]
[<ffffffffa0414188>] kvm_arch_vcpu_ioctl_run+0xa8/0x270 [kvm]
[<ffffffffa03fc042>] kvm_vcpu_ioctl+0x512/0x6d0 [kvm]
[<ffffffff811d4326>] do_vfs_ioctl+0x86/0x4f0
[<ffffffff811d4821>] SyS_ioctl+0x91/0xb0
[<ffffffff817610ad>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7136/stack
[<ffffffffa02b1fa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7124/stack
[<ffffffff811d50c9>] poll_schedule_timeout+0x49/0x70
[<ffffffff811d678a>] do_sys_poll+0x50a/0x590
[<ffffffff811d68eb>] SyS_poll+0x6b/0x100
[<ffffffff817610ad>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvknode2026:/proc/7124/task# cat 7139/stack
[<ffffffff810daf77>] futex_wait_queue_me+0xd7/0x150
[<ffffffff810dc087>] futex_wait+0x1a7/0x2c0
[<ffffffff810ddc14>] do_futex+0x334/0xb70
[<ffffffff810de592>] SyS_futex+0x142/0x1a0
[<ffffffff817610ad>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff

 qemu command line (guest OS WinServer2008):
root 25258 996 21.5 21174412 14181580 ? Sl Aug27 73740:11 /usr/bin/kvm -name zjx_1-clone -S -machine pc-i440fx-1.5,accel=kvm,usb=off,system=windows -cpu qemu64,hv_relaxed,hv_spinlocks=0x2000 -m 16384 -smp 12,maxcpus=72,sockets=12,cores=6,threads=1 -uuid 8c8b9abf-e9a6-4c3e-93cd-137a9550e593 -no-user-config -nodefaults -chardev so
cket,id=charmonitor,path=/var/lib/libvirt/qemu/zjx_1-clone.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,clock=vm,driftfix=slew -no-hpet -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device usb-ehci,id=ehci,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus
=pci.0,addr=0x5 -drive file=/vms/aaa/zjx_1-clone.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=directsync -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/vms/isos/virtio-win2008R2.vfd,if=none,id=drive-fdc0-0-0,readonly=on,format=raw,cache=directsync -global isa-fdc.driveA=drive-fdc0-0-0 -drive if=none,id=drive-ide0-1-1,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1,bootindex=2 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=0c:da:41:1d:b6:47,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-ser
ial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/zjx_1-clone.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0 -vnc 0.0.0.0:3 -device VGA,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7

 all qemu thread (guest OS WinServer2008):
 top -d 1 -H -p 25258
top - 14:53:37 up 24 days, 21:27, 2 users, load average: 19.12, 20.56, 20.20
Tasks: 14 total, 13 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 48.1%us, 18.2%sy, 0.0%ni, 33.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65674944k total, 64651012k used, 1023932k free, 194608k buffers
Swap: 8803324k total, 4140324k used, 4663000k free, 363712k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P WCHAN COMMAND
25281 root 20 0 20.2g 13g 4020 R 157 21.6 5864:12 14 - kvm
25284 root 20 0 20.2g 13g 4020 R 155 21.6 5863:02 4 - kvm
25294 root 20 0 20.2g 13g 4020 R 153 21.6 5851:59 3 - kvm
25287 root 20 0 20.2g 13g 4020 R 152 21.6 5861:20 15 - kvm
25299 root 20 0 20.2g 13g 4020 R 152 21.6 5847:14 1 - kvm
25258 root 20 0 20.2g 13g 4020 R 122 21.6 3372:41 13 - kvm
25269 root 20 0 20.2g 13g 4020 R 101 21.6 5929:42 5 - kvm
25301 root 20 0 20.2g 13g 4020 R 101 21.6 5847:26 10 - kvm
25292 root 20 0 20.2g 13g 4020 R 100 21.6 5853:18 7 - kvm
25297 root 20 0 20.2g 13g 4020 R 100 21.6 5843:37 16 - kvm
25272 root 20 0 20.2g 13g 4020 R 98 21.6 5872:52 2 - kvm
25277 root 20 0 20.2g 13g 4020 R 93 21.6 5878:21 0 - kvm
25290 root 20 0 20.2g 13g 4020 R 51 21.6 5863:15 8 - kvm
25314 root 20 0 20.2g 13g 4020 S 0 21.6 0:41.42 1 futex_wai kvm

all thread's kernel stack (guest OS WinServer2008):
root@cvk11:/proc/25258/task# cat 25281/stack
[<ffffffffa03cdfa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffa03d60d4>] vmx_handle_exit+0x94/0x8b0 [kvm_intel]
[<ffffffffa062cbb4>] __vcpu_run+0x704/0xc30 [kvm]
[<ffffffffa062d188>] kvm_arch_vcpu_ioctl_run+0xa8/0x270 [kvm]
[<ffffffffa0615042>] kvm_vcpu_ioctl+0x512/0x6d0 [kvm]
[<ffffffff811d4326>] do_vfs_ioctl+0x86/0x4f0
[<ffffffff811d4821>] SyS_ioctl+0x91/0xb0
[<ffffffff817610ad>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25284/stack
[<ffffffffa0613537>] kvm_vcpu_yield_to+0x47/0xa0 [kvm]
[<ffffffffa06136ab>] kvm_vcpu_on_spin+0x11b/0x150 [kvm]
[<ffffffffa03cdfa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25294/stack
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25287/stack
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25299/stack
[<ffffffffa03d34f6>] handle_pause+0x16/0x30 [kvm_intel]
[<ffffffffa03d60d4>] vmx_handle_exit+0x94/0x8b0 [kvm_intel]
[<ffffffffa062caec>] __vcpu_run+0x63c/0xc30 [kvm]
[<ffffffffa062d188>] kvm_arch_vcpu_ioctl_run+0xa8/0x270 [kvm]
[<ffffffffa0615042>] kvm_vcpu_ioctl+0x512/0x6d0 [kvm]
[<ffffffff811d4326>] do_vfs_ioctl+0x86/0x4f0
[<ffffffff811d4821>] SyS_ioctl+0x91/0xb0
[<ffffffff817610ad>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25258/stack
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25269/stack
[<ffffffffa03d34fe>] handle_pause+0x1e/0x30 [kvm_intel]
[<ffffffffa03d60d4>] vmx_handle_exit+0x94/0x8b0 [kvm_intel]
[<ffffffffa062caec>] __vcpu_run+0x63c/0xc30 [kvm]
[<ffffffffa062d188>] kvm_arch_vcpu_ioctl_run+0xa8/0x270 [kvm]
[<ffffffffa0615042>] kvm_vcpu_ioctl+0x512/0x6d0 [kvm]
[<ffffffff811d4326>] do_vfs_ioctl+0x86/0x4f0
[<ffffffff811d4821>] SyS_ioctl+0x91/0xb0
[<ffffffff817610ad>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25301/stack
[<ffffffffa03d34fe>] handle_pause+0x1e/0x30 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25292/stack
[<ffffffffa03cdfa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25297/stack
[<ffffffffa03cdfa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25272/stack
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25277/stack
[<ffffffffa03cdfa3>] clear_atomic_switch_msr+0x133/0x170 [kvm_intel]
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25290/stack
[<ffffffffffffffff>] 0xffffffffffffffff
root@cvk11:/proc/25258/task# cat 25314/stack
[<ffffffff810daf77>] futex_wait_queue_me+0xd7/0x150
[<ffffffff810dc087>] futex_wait+0x1a7/0x2c0
[<ffffffff810ddc14>] do_futex+0x334/0xb70
[<ffffffff810de592>] SyS_futex+0x142/0x1a0
[<ffffffff817610ad>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff

[Thomas Huth (th-huth)]

Triaging old bug tickets... can you still reproduce this issue with the latest version of QEMU? Or could we close this ticket nowadays?

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]