Certain VMs do not run under KVM using karmic's kernel

[Bobbi Manners]

Binary package hint: kvm

I have a number of virtual machines set up which run correctly under KVM (using Virtual Machine Manager / libvirt) with Jaunty kernel 2.6.28-11.

When I boot kernel 2.6.30-rc5 ("vmlinuz-2.6.30-020630rc5-generic"), the current Karmic alpha kernel, KVM is unable to run some of my virtual machines, but others work OK. The ones that do not work appear to freeze very early in the boot process, spinning the CPU.

Working OK under 2.6.30-rc5:
- FreeBSD 7.1
- Kubuntutu 09.04
- Minix 3

Fail to boot under 2.6.30-rc5, but boot OK with 2.6.28-10:
- Windows XP
- Windows 7 Beta
- OpenSolaris 09.11

Please let me know if you would like me to collect any diagnostic data to assist in tracking down this regression.

[Bobbi Manners (bobbi.manners)]

Sorry, I meant OpenSolaris 08.11 (not 09.11) above.

(Actually, on further testing it appears that my OpenSolaris VM is not booting all the way under 2.6.28-10 either, but it certainly gets much further than it does using the newer kernel.)

[Dave Lane (lightweight)]

I'm also experiencing a similar (possible the same) KVM problem on Jaunty with various Win XP virtual machines, with kernel 2.6.30-020630-generic #020630 SMP. Running VMs via virt-manager, it appears that the Windows VMs actually boot, but the VNC screen remains blank/black. The screen goes blank right after the libvirt bios screen. In some cases, I get the message that the Windows instance isn't responding to VNC (presumably meaning that it's crashed).

When running kvm on the command line, e.g.
usr/bin/kvm -S -M pc -m 512 -smp 1 -name xpsp3_ie7 -uuid 5d26354f-748a-b22f-edd8-2b56a6362a7a -monitor pty -vga std -pidfile /var/run/libvirt/qemu/xpsp3_ie7.pid -no-acpi -boot c -drive file=/home/kvm/images/xpsp3_ie7_qcow2.img,if=ide,index=0,boot=on -net nic,macaddr=00:16:3e:7a:8b:5d,vlan=0 -net tap,fd=19,script=,vlan=0,ifname=vnet1 -serial none -parallel none -usb -usbdevice tablet

I get the warning or error:
TUNGETIFF ioctl() failed: Bad file descriptor
char device redirected to /dev/pts/7

I have a Jeos (Ubuntu Hardy) VM which I run headless, and this still seems to function properly (i.e. I can SSH into it), although also gives a black screen when trying to view the console via VNC.

This is all running on an ASUS laptop with Intel Core 2 Duo CPU T7100 @1.80GHz and 32bit Jaunty.

For the record, I installed the 2.6.30 kernel to allow testing of the new UXA support for Intel graphics chipsets.

Dave

[Dave Lane (lightweight)]

I should point out that all the Win XP VMs functioned without issues on 2.6.28 (the default Jaunty 32bit kernel).

[Bobbi Manners (bobbi.manners)]

I can confirm that my Windows XP, Windows 7 RC and OpenSolaris 08.11 virtual machines all fail to boot under KVM with kernel 2.6.30.

I first encountered this problem when I switched from the stock Jaunty 2.6.28 kernel to 2.6.30-rc5. This comment is just to confirm that the problem persists with the released 2.6.30 kernel.

Other virtual machines, including Ubuntu Jaunty itself, continue to boot and run fine under KVM under kernel 2.6.30.

[Dustin Kirkland  (kirkland)]

I'm marking this confirmed, as several users have confirmed this issue (though I haven't myself).

I also added a linux kernel task for this, as the issue looks to be there.

To the reporters: can you please paste your kvm command line as reported by "pgrep kvm" ?

:-Dustin

[Bobbi Manners (bobbi.manners)]

I am using 'Virtual Machine Manager' / libvirt.

My kvm command line is as follows:

bob@gecko2:~$ ps -wwfp $(pgrep -d, -x kvm)
UID PID PPID C STIME TTY TIME CMD
bob 26562 1 91 19:38 ? 00:01:30 /usr/bin/kvm -S -M pc -m 384 -smp 1 -name WindowsXP -uuid c4de242a-13c9-2586-bd73-38771db848c3 -monitor pty -pidfile /home/bob/.libvirt/qemu/run/WindowsXP.pid -boot c -drive file=/home/bob/KVM Disk Images/WindowsXP.img,if=ide,index=0,boot=on -drive file=,if=ide,media=cdrom,index=2 -net nic,macaddr=00:16:36:6d:79:55,vlan=0 -net user,vlan=0 -serial pty -parallel none -usb -vnc 127.0.0.1:0

[Volker (w-launchpad-lendecke-de)]

Just to add another data point: My Windows XP kvm guest stopped working after a kernel upgrade this morning. With 2.6.28-13-generic Windows does not work, booting into 2.6.28-11-generic repairs that issue.

Volker

[pixeltecs (pcross)]

I have also had a similar problem. Booting Windows XP worked fine this morning until I received an auto-update that included a new kernel. When I boot 2.6.28-13-generic, it fails. When I boot 2.6.28-11-generic, it works fine.

When it fails: The bios screen shows. I can get into the windows boot menu fine but once I start the actual OS boot process, it seems to just sit and spin on the processor.

When it works: It seems to spin the processor around 50% for a few seconds but the screen flickers and it eventually breaks out of it. When it fails, it seems like it just never gets passed whatever it was doing there.

BTW, I have another XP machine that seems to work fine. Difference is that I don't run it as root and it probably doesn't have all of its updates.

My info (using same command as Bob Manners):

pcross@pcross-laptop:~$ ps -wwfp $(pgrep -d, -x kvm)
UID PID PPID C STIME TTY TIME CMD
root 4135 1 42 15:29 ? 00:05:49 /usr/bin/kvm -S -M pc -m 512 -smp 1 -name WindowsXP_root -uuid deaeb78d-5ded-d124-8870-d3748ce4f5ab -monitor pty -pidfile /var/run/libvirt/qemu//WindowsXP_root.pid -localtime -no-acpi -boot c -drive file=/var/lib/libvirt/images/WindowsXP_root.img,if=ide,index=0,boot=on -net nic,macaddr=54:52:00:3e:98:a5,vlan=0 -net tap,fd=16,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:0 -k en-us -soundhw es1370

[Tom Goh (tomgohj)]

I have the same problem but booting all my Ubuntu VMs which are running Hardy. I was using Jaunty with the server kernel (to get access to full 4GB memory) originally and upgraded to 2.6.30 generic kernel to fix my Intel Graphics issues. After this all my KVMs will not boot fully. It does the POST then gets stuck at "Starting up ..."

Here is my output

tomg@rad4-x300:~$ ps -wwfp $(pgrep -d, -x kvm)
UID PID PPID C STIME TTY TIME CMD
root 4051 1 82 11:12 ? 00:00:23 /usr/bin/kvm -S -M pc -m 512 -smp 1 -name x300-alfresco -uuid a43c32e7-f0d5-bbb4-3784-d55fe13027da -monitor pty -pidfile /var/run/libvirt/qemu//x300-alfresco.pid -boot c -drive file=/home/tomg/VMs/x300-alfresco,if=ide,index=0,boot=on -drive file=,if=ide,media=cdrom,index=2 -net nic,macaddr=54:52:00:7c:3b:6b,vlan=0,model=virtio -net tap,fd=16,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-us -soundhw es1370

[Jarda Benkovsky (edheldil001)]

I have the same problem: after today's upgrade to kernel 2.6.28-13, my virtual machines with default Jaunty installation stopped working, they hang after displaying grub devices and 'Starting up', spinning the CPU.

This is my KVM start command:

gksudo -- kvm -hda mname/kvm.img -m 512 -net nic -net tap,script=/etc/kvm/kvm-ifup -vnc 127.0.0.1:0 -sdl -daemonize

If I add -no-kvm option, the VM boots, but ept-get update segfaults.

Another VM using CentOS 5.3 boots without (apparent) problems. If I reboot the host to 2.6.28-11, the Jaunty VMs work again.

[Gustavo Niemeyer (niemeyer)]

Same problem here. Fresh Jaunty images built with ubuntu-vm-builder moments ago are hanging over after "Starting up ...". Hardy images work fine.

[dawntiger (dawntiger-alan)]

I have a same problem. Windows XP guest os can not be accessed with vnc.

Host: Ubuntu Jaunty with kernel 2.6.30-020630-generic
KVM: 84+dfsg-0ubuntu12

/usr/bin/kvm -S -M pc -m 384 -smp 1 -name xp -uuid 24101b4d-dd28-45c5-3934-b790fc0eef4a -monitor pty -pidfile /var/run/libvirt/qemu//xp.pid -localtime -no-acpi -boot c -drive file=/home/alan/VMs/libvirt/images/xp.qcow2,if=ide,index=0,boot=on -net nic,macaddr=54:52:00:7d:39:ae,vlan=0,model=ne2k_pci -net tap,fd=17,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:0 -k en-us -soundhw es1370

[Iggy (iggy-theiggy)]

For people with this issue, does using "-cpu qemu64,-nx" on your kvm command line help?

[Tom Goh (tomgohj)]

I ran

tomg@rad4-x300:~$ sudo kvm -S -M pc -m 512 -smp 1 -name x300-alfresco -uuid a43c32e7-f0d5-bbb4-3784-d55fe13027da -monitor pty -pidfile /var/run/libvirt/qemu//x300-alfresco.pid -boot c -drive file=/home/tomg/VMs/x300-alfresco,if=ide,index=0,boot=on -drive file=,if=ide,media=cdrom,index=2 -net nic,macaddr=54:52:00:7c:3b:6b,vlan=0,model=virtio -net tap,fd=16,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-us -soundhw es1370 -cpu qemu64,-nx

This is the output
TUNGETIFF ioctl() failed: Bad file descriptor
char device redirected to /dev/pts/1
char device redirected to /dev/pts/2

I usually run kvm with virt-manager so do not know if that output is normal. I still cannot ping the VM

[Iggy (iggy-theiggy)]

libvirt's suggestion to get custom command line parameters i to use a wrapper script that calls kvm + your options + all options passed to it and set that as your executable. Why they don't allow adding options, I don't know. That's your best bet. Running it like you did will never work because libvirt normally sets up the network connections and passes them as open files to kvm.

[Dustin Kirkland  (kirkland)]

Ubuntu Karmic's kernel is now on 2.6.31-2.15.

Can anyone reproduce this problem on a 2.6.31 kernel?

:-Dustin

[dawntiger (dawntiger-alan)]

I've tried to upgrade kvm from 84 to 87, but it did not work.

Host: Ubuntu Jaunty with kernel 2.6.30-020630-generic
KVM: 87

/usr/local/kvm/bin/qemu-system-x86_64 -S -M pc -m 384 -smp 1 -name xp -uuid 24101b4d-dd28-45c5-3934-b790fc0eef4a -monitor pty -pidfile /var/run/libvirt/qemu//xp.pid -localtime -no-acpi -boot c -drive file=/home/alan/VMs/libvirt/images/xp.qcow2,if=ide,index=0,boot=on -net nic,macaddr=54:52:00:7d:39:ae,vlan=0,model=ne2k_pci -net tap,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:0 -k en-us -soundhw es1370

And the command 'qemu-system-x86_64' built from KVM-87 code can not allow the parameter 'fd' in network setting, so I had to remove 'fd=17,' from the upper command sample.

Other thing I have to mention here, I opened visual effect at normal level.

[langeman (langeman)]

Between the lines; I can conform as as well:

2.6.28-11-generic is OK
2.6.28-13-generic VM (Windows XP with SP3) is not working.

[munzli (munzli)]

@Iggy:
the options "-cpu qemu64,-nx" got my windows bootet again, but it restarts all of a sudden.

[Andreas Hasenack (ahasenack)]

Adding -cpu core2duo,-nx made my karmic image boot again. Same for -cpu qemu64,-nx. But without the -nx, it doesn't boot.

I'm on jaunty with 2.6.28-13-generic.

Bug #396219 is related, if not duplicated.

[dawntiger (dawntiger-alan)]

Host: Ubuntu Jaunty with kernel 2.6.30-020630-generic
KVM: 84+dfsg-0ubuntu12

Adding -cpu qemu32 made kvm work well.

[Morten Lund (m-lund)]

I can confirm the problem exists with the latest Karmic kernel:

mlund@mlund-kubuntu:~$ uname -a
Linux mlund-kubuntu 2.6.31-3-generic #19-Ubuntu SMP Tue Jul 14 16:04:41 UTC 2009 i686 GNU/Linux

Running the VM with Qemu and kqemu enabled works fine, but it is a lot slower (of course)

[Pat Osterday (pat-osterday)]

Just confirming that I have the problem on 9.04, 2.6.28-13-generic - 2.6.28-11-generic works fine. I was hoping the KVM update that just got pushed fixed the problem, but no go.

There is also a bug posted on Red Hat's Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=510695

[gerryk (gerryk)]

Confirming this on Kernel 2.6.28-15-generic #49-Ubuntu SMP/KVM 0.9.1

VMs running OpenSolaris 9, FreeBSD 7.2 and Windows XP SP2 all hand with the following from dmesg...

[135369.886301] kvm: 26463: cpu0 unhandled wrmsr: 0xc0010117 data 0
[135369.896799] kvm: 26463: cpu0 unhandled rdmsr: 0xc0010117
[135369.897059] kvm: 26463: cpu0 unhandled rdmsr: 0xc0010117

[Bryan McLellan (btm)]

Confirmed on linux-image-2.6.31-9-generic with kvm=1:84+dfsg-0ubuntu16 + libvirt-bin=0.7.0-1ubuntu3 on karmic.

WinXP would run through the install but not boot after the reboot.

Same with Win7

Then running Win7 from the command line with '-cpu qemu32' allowed it to boot.

[Dustin Kirkland  (kirkland)]

Can anyone reproduce this on an up-to-date karmic system, running qemu-kvm-0.11?

:-Dustin

[Bryan McLellan (btm)]

I can reproduce this. My windows 7 VM will only boot to where it says "Starting Windows". The animation that usually appears above it does not and one of the two VCPUs pegs out.

If I set my emulator in the XML configuration to a shell script of:

exec /usr/bin/kvm -cpu qemu32 "$@"

Then the VM starts up normally. After the most recent upgrade this morning, I also had to add the path to my shell script to /etc/apparmor.d/abstractions/libvirt-qemu in the same fashion as the other binaries or I would get permission denied errors when trying to start the VM.

[Dustin Kirkland  (kirkland)]

If you're experiencing this problem, could you please attach /proc/cpuinfo ?

Thanks,
:-Dustin

[Bobbi Manners (bobbi.manners)]

bob@gecko2:~/Array/uaas$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 14
model name : Genuine Intel(R) CPU T2400 @ 1.83GHz
stepping : 8
cpu MHz : 1000.000
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc arch_perfmon bts pni monitor vmx est tm2 xtpr pdcm
bogomips : 3659.07
clflush size : 64
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 14
model name : Genuine Intel(R) CPU T2400 @ 1.83GHz
stepping : 8
cpu MHz : 1000.000
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc arch_perfmon bts pni monitor vmx est tm2 xtpr pdcm
bogomips : 3658.86
clflush size : 64
power management:

[Dustin Kirkland  (kirkland)]

Thanks guys.

Can anyone reproduce this on AMD hardware? All three I see here are
Intel. This might well be an Intel-specific issue.

:-DUstin

[pml (phi)]

Hi,

just for making the list more complete: another Intel CPU...

CU,
-- pml

[Esa Häkkinen (esa+hakkinen)]

T2500 is also older Core Duo (coreduo) processor.

Reproduced from command line: succesfully installed & running winxp sp3 with "kvm -cpu coreduo,-nx".

How do I define "no NX" this in /etc/libvirt/qemu/winxp.xml configuration file?

<domain>
  <cpu name='coreduo'>
    <feature name='nx' option='disable'/>
  </cpu>

-> won't work, nx feature is not disabled.

neither success with less documented xml file options/features:

  <os>
    <type arch='i686' machine='pc'>hvm</type>
    <feature name='nx' policy='disable'/>
    <feature name='nx' option='disable'/>
    <feature name='nx' mode='set' value='off'/>
    <boot dev='hd'/>
  </os>
  <cpu model='coreduo,-nx'>
    <feature name='nx' policy='disable'/>
    <feature name='nx' option='disable'/>
    <feature name='nx' mode='set' value='off'/>
  </cpu>

[jrigling (jrigling)]

Testing Results:

Running Ubuntu Jaunty
  kvm: 1:84+dfsg-0ubuntu12.3
  on Linux 2.6.28-11, Windows XP VM worked properly
  on Linux 2.6.28-15, Windows XP VM hanged with black screen

Upgraded to Ubuntu Karmic
  kvm: 1:84+dfsg-0ubuntu12.3
  on Linux 2.6.31-11, Windows XP VM hanged with black screen

Installed qemu-kvm instead of the old kvm package
  qemu-kvm: 0.11.0-0ubuntu1
  on Linux 2.6.31-11, Windows XP VM worked properly

It seems that upgrading to Karmic failed to switch over to qemu-kvm package. When qemu-kvm is installed manually, everything seems to work fine.

[Bernhard Bock (bernhard-bock)]

Let me re-emphasize the last comment from jrigling:

"apt-get install qemu-kvm" solves the problem in karmic for me. It uninstalls "kvm" and "qemu" packages from jaunty and installs the new "qemu-kvm" package from karmic. Problem gone.

This is more likely an installer / upgrade bug than a kernel bug in karmic.

[Urs (urs-ak)]

Fresh 9.10 beta + latest updates
Linux 2.6.31-11

Windows 7 VM hanged with garbage on screen with first boot after successfully installation. Next - boot hanged with black screen.

[Ryan Davies (iownsu)]

Im also getting the same bug as this.

Please refer to my post on Ubuntuforums:
http://ubuntuforums.org/showthread.php?t=1375229

[Aaron C. de Bruyn (darkpixel2k)]

Same issue for me on a fully-updated Karmic (as of 15 minutes ago).
I have an AMD CPU.

[Aaron C. de Bruyn (darkpixel2k)]

As a work-around, I changed:
<type arch='x86_64' machine='pc'>hvm</type>

to:
<type arch='i686' machine='pc-0.11'>hvm</type>

XP booted right up.