kvm hangs for freebsd guests since raring upgrade

[Vincent Ladeuil]

Until I upgraded to raring, freebsd guest has been one of my most stable guest under kvm (and previously under vbox for that matter).

Since the upgrade to raring, my existing guest hangs very early in the boot process ending with the following in /var/log/libvirt/qemu/freebsd8.log :

2013-03-20 07:47:53.020+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -name freebsd8 -S -M pc-1.0 -m 4096 -smp 2,sockets=2,cores=1,threads=1 -uuid 1babac4e-de14-5f5d-73e9-dfe524966f9a -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/freebsd8.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/caviar3/libvirt/images/freebsd-8.0-64bits.qcow2,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=30,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=08:00:27:5f:9f:06,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
W: kvm binary is deprecated, please use qemu-system-x86_64 instead
char device redirected to /dev/pts/6 (label charserial0)
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80855a40 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=0000000000000000 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A]
SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff
CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 40 5a 85 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00
qemu: terminating on signal 15 from pid 2474
2013-03-20 07:49:36.834+0000: shutting down

To isolate the issue (and hoping for some change in more recent freebsd versions) I tried creating a new vm but hit the same issue.

Using ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RELEASE-amd64-bootonly.iso , I end up with the following /var/log/libvirt/qemu/freebsd9.log which includes the kvm command :

2013-03-20 07:52:03.780+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name freebsd9 -S -M pc-i440fx-1.4 -m 4096 -smp 2,sockets=2,cores=1,threads=1 -uuid 1ae60371-46e3-641b-4d85-9b3e5ac3cb3c -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/freebsd9.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/caviar3/libvirt/images/freebsd9.img,if=none,id=drive-ide0-0-0,format=raw -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/caviar3/libvirt/installers/freebsd/FreeBSD-9.1-RELEASE-amd64-bootonly.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=30,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:3e:70:67,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
W: kvm binary is deprecated, please use qemu-system-x86_64 instead
char device redirected to /dev/pts/6 (label charserial0)
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80bcccd0 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=0000000000000000 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A]
SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff
CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 d0 cc bc 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00

From that,

  KVM internal error. Suberror: 1
  emulation failure

doesn't tell me much about where to look at next :-/
---
ApportVersion: 2.9.2-0ubuntu1
Architecture: amd64
DistroRelease: Ubuntu 13.04
InstallationDate: Installed on 2011-06-23 (636 days ago)
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64+mac (20110427.1)
MarkForUpload: True
Package: linux
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 3.8.0-13.23-generic 3.8.3
Tags: third-party-packages raring
Uname: Linux 3.8.0-13-generic x86_64
UnreportableReason: This is not an official Ubuntu package. Please remove any third party package and try again.
UpgradeStatus: Upgraded to raring on 2013-03-17 (4 days ago)
UserGroups: adm admin cdrom dialout libvirtd lpadmin plugdev sambashare vboxusers wireshark

[Serge Hallyn (serge-hallyn)]

Hm, when I just run 'kvm -hda freebsd.img -cdrom Free*.iso -m 512", it does boot for me.

[Serge Hallyn (serge-hallyn)]

that's the livecd though, let me finish an install and see if the installed disk boots.

[Serge Hallyn (serge-hallyn)]

I can't reproduce this, though I was using kvm by hand, not
through libvirt.

[Serge Hallyn (serge-hallyn)]

I can't reproduce this in libvirt either.

Could you please attach the .xml for your VM, and the result of 'virsh capabilities'

[Vincent Ladeuil (vila)]

As discussed on IRC, doing:
  sudo kvm -hda freebsd-8.0-64bits.qcow2 -serial stdio

doesn't trigger the bug, I'll investigate on which part of the xml is causing the issue and report here.

[Serge Hallyn (serge-hallyn)]

Inside the freebsd guest, you need to have 'console="comconsole"' in /boot/loader.conf in the freebsd guest. Then you can do

qemu-system-x86_64 -hda freebsd.img -m 512 -vnc :1 -serial stdio -enable-kvm

to see the hang. Do it without -enable-kvm, and it doesn't hang.

I set up a precise container on a raring host, with precise's qemu and seabios. The results there were the same! Finally, both bug reporters say that in quantal there was no such hang. This points to the kernel as the culprit.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1157589

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Vincent Ladeuil (vila)]

apport information

[Vincent Ladeuil (vila)]

apport information

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.9 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.9-rc4-raring/

[Vincent Ladeuil (vila)]

Fixed upstream, great. Rough estimate about when this will land for raring ?

[Joseph Salisbury (jsalisbury)]

Can you also test the latest v3.8.4 stable kernel?
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8.4-raring/

If the bug still exits in 3.8.4, we will need to perform a reverse bisect to identify the commit that fixes in this bug in v3.9-rc4.

[Vincent Ladeuil (vila)]

Testing done, the bug is present in 3.8.4

[John Klug (ski-brimson)]

I have what I believe is this same problem while booting FreeBSD from GRUB2 using UEFI (grub2 and FreeBSD set to serial). I was using 12.04's kernel. Should have written down which one, but I had all the updates as of 2013 May 9. I updated the kernel to V3.9RC4 as suggested, and the problem remains.

Linux peru 3.9.0-030900rc4-generic #201303232035 SMP Sun Mar 24 00:36:21 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

qemu-system-x86_64 -nographic -L /usr/share/qemu-ovmf/bios -hda Fed17_FBSD.disk.cow2 -m 4096

KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000624
ESI=00000000 EDI=0000ffff EBP=00000000 ESP=0000ff6b
EIP=00007000 EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =a4b4 000a4b40 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 00000000 0000ffff
IDT= 00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

[Serge Hallyn (serge-hallyn)]

Is someone working on, or planning to work on, a bisect to find out which commit fixed it in v3.9-rc4?