64-bit linux guests fail to start on oneiric running 3.0 kernel

[Klaus Rennecke]

Host: Ubuntu 11.10 kernel vmlinuz-3.0.0-12-generic or vmlinuz-3.0.0-12-server on AMD Athlon(tm) II P360 Dual-Core
Guests: SLES 10 or 11, all 64 bit

32 bit windows guest starts fine. All 64 bit linux guests loop during boot, when GRUB is starting. VMs are managed using libvirt 0.9.2-4ubuntu15 and virt-manager 0.9.0.

Log file shows:
 KVM internal error. Suberror: 1
 emulation failure

repeated for each GRUB attempt.

Starting the same host with vmlinuz-2.6.38-11-generic makes all VMs run OK.
---
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
DistroRelease: Ubuntu 11.10
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427.1)
KvmCmdLine:
 UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
 113 3739 1 15 573004 24800 1 16:51 ? 00:00:06 /usr/bin/kvm -S -M pc-0.14 -cpu qemu64,+wdt,+skinit,+osvw,+3dnowprefetch,+sse4a,+abm,+cr8legacy,+extapic,+cmp_legacy,+lahf_lm,+rdtscp,+pdpe1gb,+popcnt,+cx16,+ht,+vme -enable-nesting -enable-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name gasse -uuid 7cb0c88c-db14-b3f0-6696-4bd717165b9a -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/gasse.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -no-acpi -boot order=c,menu=on -drive file=/var/lib/libvirt/images/gasse-0.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=19,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:c6:f5:b4,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
MachineType: Hewlett-Packard HP 625
NonfreeKernelModules: wl fglrx
Package: qemu-kvm 0.14.1+noroms-0ubuntu6
PackageArchitecture: amd64
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.0.0-13-server root=UUID=338f05b6-6307-4f1a-a8b6-9b12a0edc5ee ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 3.0.0-13.22-server 3.0.6
Tags: oneiric running-unity
Uname: Linux 3.0.0-13-server x86_64
UpgradeStatus: Upgraded to oneiric on 2011-10-18 (35 days ago)
UserGroups: adm admin cdrom dialout kvm libvirtd lpadmin plugdev sambashare
dmi.bios.date: 12/07/2010
dmi.bios.vendor: Hewlett-Packard
dmi.bios.version: 68DVA Ver. F.06
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: 1475
dmi.board.vendor: Hewlett-Packard
dmi.board.version: KBC Version 72.0E
dmi.chassis.asset.tag: CNU1060YXZ
dmi.chassis.type: 10
dmi.chassis.vendor: Hewlett-Packard
dmi.modalias: dmi:bvnHewlett-Packard:bvr68DVAVer.F.06:bd12/07/2010:svnHewlett-Packard:pnHP625:pvr:rvnHewlett-Packard:rn1475:rvrKBCVersion72.0E:cvnHewlett-Packard:ct10:cvr:
dmi.product.name: HP 625
dmi.sys.vendor: Hewlett-Packard
---
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
DistroRelease: Ubuntu 11.10
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427.1)
KvmCmdLine:
 UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
 113 3976 1 15 573004 25340 0 15:04 ? 00:00:04 /usr/bin/kvm -S -M pc-0.14 -cpu qemu64,+wdt,+skinit,+osvw,+3dnowprefetch,+sse4a,+abm,+cr8legacy,+extapic,+cmp_legacy,+lahf_lm,+rdtscp,+pdpe1gb,+popcnt,+cx16,+ht,+vme -enable-nesting -enable-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name gasse -uuid 7cb0c88c-db14-b3f0-6696-4bd717165b9a -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/gasse.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -no-acpi -boot order=c,menu=on -drive file=/var/lib/libvirt/images/gasse-0.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=19,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:c6:f5:b4,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
MachineType: Hewlett-Packard HP 625
NonfreeKernelModules: wl fglrx
Package: qemu-kvm 0.14.1+noroms-0ubuntu6
PackageArchitecture: amd64
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.0.0-14-server root=UUID=338f05b6-6307-4f1a-a8b6-9b12a0edc5ee ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 3.0.0-14.23-server 3.0.9
Tags: oneiric running-unity
Uname: Linux 3.0.0-14-server x86_64
UpgradeStatus: Upgraded to oneiric on 2011-10-18 (56 days ago)
UserGroups: adm admin cdrom dialout kvm libvirtd lpadmin plugdev sambashare
dmi.bios.date: 12/07/2010
dmi.bios.vendor: Hewlett-Packard
dmi.bios.version: 68DVA Ver. F.06
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: 1475
dmi.board.vendor: Hewlett-Packard
dmi.board.version: KBC Version 72.0E
dmi.chassis.asset.tag: CNU1060YXZ
dmi.chassis.type: 10
dmi.chassis.vendor: Hewlett-Packard
dmi.modalias: dmi:bvnHewlett-Packard:bvr68DVAVer.F.06:bd12/07/2010:svnHewlett-Packard:pnHP625:pvr:rvnHewlett-Packard:rn1475:rvrKBCVersion72.0E:cvnHewlett-Packard:ct10:cvr:
dmi.product.name: HP 625
dmi.sys.vendor: Hewlett-Packard

[Klaus Rennecke (kre)]

Tested linux-image-3.0.0-13-server 3.0.0-13.22 from oneiric-proposed -- same failure.

Tested linux-image-2.6.38-12-server 2.6.38-12.51 from natty-updates -- works fine.

[Klaus Rennecke (kre)]

Tested linux-image-3.0.0-13-server 3.0.0-13.22 from oneiric-updates -- same failure.

[Klaus Rennecke (kre)]

correct package

[Serge Hallyn (serge-hallyn)]

Thanks for submitting this bug.

In order to have apport provide some more information which could help in debugging this, could you please run:

   apport-collect 882997

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Serge Hallyn (serge-hallyn)]

Thanks for the information. I don't see any kernel dump info in the currentdmesg. Could you first try starting the virtual machine that fails, and then re-run the apport-collect command?

[Klaus Rennecke (kre)]

Apport was done with the failing VM running, sort of. You can see it in the KvmCmdLine, added to the bug description by apport. The VM oscillates though, it starts through BIOS, and fails when GRUB starts. So it does not really "run". No dump to be seen in dmesg, but the VM log file (from /var/log/libvirt/qemu/) shows:

KVM internal error. Suberror: 1
emulation failure

for each iteration through GRUB in the guest. See attached segment of the log file generated while apport was used.

Not that the same VM runs fine without change when the host was booted into the old kernel 2.6.38-12-server from natty.

None of my 64 bit VMs run with the stock oneiric kernels. Basic guest set-up is SuSE Linux on most of these. I am not at liberty to provide the current images (they are huge anyway) but if really needed I can start with a scratch guest installation (checked already that they fail too).

[Klaus Rennecke (kre)]

Just to be sure it's clear: Kernel version of the *host* matters, the guest VMs are not changed.

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

apport information

[Klaus Rennecke (kre)]

After updating apport information with the new kernel from the security updates today, I did a bit more testing on the guest VM side. Aside from the host kernel version, it does seem to depend on the guest OS or boot loader(?):

* Oneiric 64 bit: works
* SLES-11-SP1: works (!)
* SLES-10-SP4: broken

Since the guest already re-starts when initializing GRUB, I would guess that the boot loader is to blame. Althogh I would expect the VM host to be able to run these beasts, after all it is supposed to be a hardware virtualization.

Do you have any suggestions to further narrow down the cause of this issue?

[Serge Hallyn (serge-hallyn)]

Would it be possible for you to try with the latest upstream qemu?

sudo apt-get build-dep qemu-kvm
sudo apt-get install git
git clone git://git.qemu.org/qemu.git
cd qemu
./configure --target-list=x86_64-softmmu
make

then run with

./x86_64-softmmu/qemu-system-x86_64 (qemu-arguments)

[Klaus Rennecke (kre)]

1. Trying to break it:
 Installed the SLES-11-SP1 with the parameters for SLES-10 causing it to use IDE disk instead of virtio.
 Runs fine though, IDE seems not to be the problem.

2. Trying to fix it:
 Compiled from the sources: commit da5361cc685c004d8bb4e7c5e7b3a52c7aca2c56 Mon Dec 12 17:21:34 2011 +0100
 Had to copy the executable to /usr/bin/qemu-system-x86_64 moving aside the original first. Libvirt would not pick it up from a soft link, apparently failing in trying to auto-sense the version. Strange but inconsequential for the issue at hand.
 Crashes nicely now:

2011-12-14 06:34:29.783: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name chamae -uuid bda73a8f-37fc-e671-235e-67afa23de4fd -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/chamae.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -drive file=/var/lib/libvirt/images/chamae.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=19,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:81:b8:be,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
char device redirected to /dev/pts/2
KVM internal error. Suberror: 1
emulation failure
EAX=00000010 EBX=0004e090 ECX=00063f00 EDX=00002cb6
ESI=00000052 EDI=0002ce88 EBP=00001ff0 ESP=00001fd8
EIP=00001879 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0008 00000000 ffffffff 00809300 DPL=0 DS16 [-WA]
CS =3c16 0003c160 0000ffff 00009b00 DPL=0 CS16 [-RA]
SS =0000 00000000 0000ffff 00009300 DPL=0 DS16 [-WA]
DS =3c16 0003c160 0000ffff 00009300 DPL=0 DS16 [-WA]
FS =0000 00000000 0000ffff 00009300 DPL=0 DS16 [-WA]
GS =0000 00000000 0000ffff 00009300 DPL=0 DS16 [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT= 0003c740 0000000f
IDT= 00000000 000003ff
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=06 6a 08 0f a1 eb 0b f6 06 f2 05 08 74 04 6a 08 0f a9 24 fe <0f> 22 c0 a0 f2 05 08 06 f1 05 66 58 c3 c6 06 f1 05 00 fb c3 66 50 e8 af 03 5e 0f a1 b0 00

[Serge Hallyn (serge-hallyn)]

Seems like this may be the same bug as 997311?

I want to try installing grub1 into a lucid VM and see if I can get that to crash.

[Serge Hallyn (serge-hallyn)]

Can you confirm whether this is still happening in 12.04 and/or 12.10?

[Serge Hallyn (serge-hallyn)]

Marking incomplete pending feedback. Assuming this is not fixed in12.04 I'll try again to reproduce.