qemu 2.7 / iPXE crash

[Greg]

I am running Arch linux

vanilla 4.7.2 kernel
qemu 2.7
libvirt 2.2.0
virt-manager 1.4.0

Since the upgrade from qemu 2.6.1 to 2.7 a few days ago. I'm no longer
able to PXE boot at all. Everything else appears to function normally.
Non PXE booting and everything else is perfect. Obviously have
restarted everying etc. Have tried the various network drivers also.

This occurs on domains created with 2.6.1 or with 2.7

When I choose PXE boot, the machine moves to a paused state (crashed)
immediately after the 'starting PXE rom execution...' message appears.

Reverting to qemu 2.6.1 package corrects the issue.

The qemu.log snippet follows below.

I'm not sure how to troubleshoot this problem to determine if it's a
packaging error by the distribution or a problem with qemu/kvm/kernel?

Any help would be much appreciated - Thanks,
Greg

--- qemu.log:

2016-09-12 16:36:33.867+0000: starting up libvirt version: 2.2.0, qemu
version: 2.7.0, hostname: seneca
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=spice /usr/sbin/qemu-system-x86_64 -name guest=c,debug-
threads=on -S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-
c/master-key.aes -machine pc-i440fx-2.7,accel=kvm,usb=off,vmport=off
-cpu Nehalem -m 2048 -realtime mlock=off -smp
1,sockets=1,cores=1,threads=1 -uuid 348009be-26d5-4dc7-b515-
e8b45f5117ac -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-
c/monitor.sock,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global
PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot
menu=on,strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7
-device ich9-usb-
uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-
uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-
usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/var/lib/libvirt/images/c.qcow2,format=qcow2,if=none,id=drive-
virtio-disk0 -device virtio-blk-
pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-
disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:a0:95:7c,bus=pci.0,addr=0x
3 -chardev pty,id=charserial0 -device isa-
serial,chardev=charserial0,id=serial0 -chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain
-6-c/org.qemu.guest_agent.0,server,nowait -device
virtserialport,bus=virtio-
serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_age
nt.0 -chardev spicevmc,id=charchannel1,name=vdagent -device
virtserialport,bus=virtio-
serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0
-device usb-tablet,id=input0,bus=usb.0,port=1 -spice
port=5901,addr=127.0.0.1,disable-ticketing,image-
compression=off,seamless-migration=on -device qxl-
vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vga
mem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-
hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-
codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2
-chardev spicevmc,id=charredir1,name=usbredir -device usb-
redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-
balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
main_channel_link: add main channel client
red_dispatcher_set_cursor_peer:
inputs_connect: inputs channel client create
KVM internal error. Suberror: 1
emulation failure
EAX=801a8d00 EBX=000000a0 ECX=00002e20 EDX=0009d5e8
ESI=7ffa3c00 EDI=7fef4000 EBP=ffffffff ESP=00007b92
EIP=000006ab EFL=00000087 [--S--PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 ffffffff 00c09300
CS =9c4c 0009c4c0 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =9cd0 0009cd00 ffffffff 00c09300
FS =0000 00000000 ffffffff 00c09300
GS =0000 00000000 ffffffff 00c09300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 00000000 00000000
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=00 16 66 9c 66 60 0f a8 0f a0 06 1e 16 0e fa 2e 8e 1e 90 06 <0f>
ae 06 d0 1c 0f 01 0e c6 1c 0f 01 06 c0 1c fc 66 b9 38 00 00 00 66 ba 10
02 00 00 66 68

--- /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
stepping : 5
microcode : 0x11
cpu MHz : 3066.648
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx
est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm tpr_shadow
vnmi flexpriority ept vpid dtherm
bugs :
bogomips : 6135.85
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

[Greg (rollenwiese)]

sudo qemu-system-x86_64 -boot n -net nic,model=virtio,vlan=0 -net bridge,vlan=0,br=br1 -drive file=/tmp/qc2.img,format=qcow2,index=0,media=disk -m 1024

Without -enable-kvm, the above command work perfectly. I can PXE boot from the tftp server on my LAN just fine.

When KVM is enabled, qemu crashes immediately displaying only this:

Booting from ROM...
iPXE (PCI 00:03.0) starting execution

[Michael Prokop (mikagrml)]

I can confirm the issue, I stumbled upon it on a Proxmox system using the pve-qemu-kvm package versions 2.7.0-3 + 2.7.0-4 and have reported the bug in Proxmox bug tracker as https://bugzilla.proxmox.com/show_bug.cgi?id=1182 with further details.

I was able to reproduce the problem also with latest git of qemu:

  % ./qemu-system-x86_64 -version
  QEMU emulator version 2.7.50 (v2.7.0-1343-g4429532-dirty)

When disabling the KVM feature QEMU loads fine with iPXE/PXE boot.
I'd be happy to provide further information if needed.

[Laszlo Ersek (Red Hat) (lersek)]

Can you post the host dmesg that is written at the time of the guest crash?

[Laszlo Ersek (Red Hat) (lersek)]

Please add the output of the following command too:

tail /sys/module/kvm/holders/kvm_intel/parameters/*

Thanks.

[Laszlo Ersek (Red Hat) (lersek)]

(I should have given the pattern /sys/module/kvm_intel/parameters/*, but the result is the same.)

[Greg (rollenwiese)]

Laszlo, I'll grab that info for you soon. In the meantime here's the bug tracker for Arch. Someone has completed a git bisect which may be helpful:

https://bugs.archlinux.org/task/50778

[Laszlo Ersek (lacos-caesar)]

The ipxe bisection is extremely helpful; can you please thank Peter Pickford in the arch tracker on our behalf?

So, the culprit iPXE commit is

commit 71560d185475117b10994d839afe059577e7768c
Author: Michael Brown <email address hidden>
Date: Wed Apr 27 11:03:18 2016 +0100

    [librm] Preserve FPU, MMX and SSE state across calls to virt_call()

We have actually seen this, in https://bugzilla.redhat.com/show_bug.cgi?id=1356762

This is a feature gap in KVM's instruction *emulation*.

In one of the previous comments, I asked for the KVM module parameters / settings -- I'm pretty sure that once you upload them, they will match Paolo's RHBZ comment in <https://bugzilla.redhat.com/show_bug.cgi?id=1356762#c12>.

Namely, I expect that the affected host does not support "unrestricted_guest"; i.e., it cannot natively virtualize the FXSAVE instruction (in big real mode that iPXE runs in). Given that "emulate_invalid_guest_state" is set to "yes" on your host (well, I expect that at least; I think it's the default if unrestricted_guest is missing), KVM "manually" emulates 16-bit big real mode for iPXE. However, FXSAVE emulation is missing from KVM.

RHBZ#1356762 is the bug that tracks the Request for Enhancement.

[Laszlo Ersek (lacos-caesar)]

(In retrospect, the QEMU code dump "<0f> ae 06 d0 1c" is also a match.)

[Laszlo Ersek (Red Hat) (lersek)]

Gerd, do you think we should rebuild the iPXE binaries bundled with QEMU with the offending iPXE commit (71560d185475) reverted, at least until KVM gets FXSAVE emulation in big real mode? I think this would be reasonable, as that iPXE commit works around a bug in the IBM Tivoli Provisioning Manager VMM.

(In other words, the iPXE commit that breaks QEMU's bundled binaries, for a number of KVM users, targets a hypervisor that's different from QEMU/KVM/Xen -- thus normally we wouldn't care about that commit at all.)

Thanks.

[Laszlo Ersek (Red Hat) (lersek)]

(--> the rebuilt binaries should go into v2.7.1, if we agree)

[Laszlo Ersek (Red Hat) (lersek)]

We could also try changing upstream iPXE so that the FXSAVE trick is not active for CONFIG=qemu.

[Laszlo Ersek (Red Hat) (lersek)]

BTW, this bug can be easily reproduced on hosts that do feature unrestricted_guest, just reload the kvm_intel module with unrestricted_guest=N.

[Laszlo Ersek (Red Hat) (lersek)]

(In other news, Launchpad continues to suck incredibly. Did you see how it broke up "unrestricted_guest" in my previous comment?)

[Laszlo Ersek (Red Hat) (lersek)]

Some more reports on ipxe-devel:

http://lists.ipxe.org/pipermail/ipxe-devel/2016-October/005203.html
http://lists.ipxe.org/pipermail/ipxe-devel/2016-October/005210.html

Radim just posted the KVM feature patches:

[PATCH 0/2] KVM: x86: emulate fxsave and fxrstor
https://www.spinics.net/lists/kernel/msg2370327.html

I thought suppressing the regression within iPXE proper could be helpful in the interim:

[ipxe-devel] [PATCH 0/2] mask lack of KVM's FXSAVE/FXRSTOR emulation in the QEMU build
http://lists.ipxe.org/pipermail/ipxe-devel/2016-October/005221.html

[Greg (rollenwiese)]

Laszlo, as requested:

[gregory@seneca ~]$ tail /sys/module/kvm/holders/kvm_intel/parameters/*
==> /sys/module/kvm/holders/kvm_intel/parameters/emulate_invalid_guest_state <==
Y

==> /sys/module/kvm/holders/kvm_intel/parameters/enable_apicv <==
N

==> /sys/module/kvm/holders/kvm_intel/parameters/enable_shadow_vmcs <==
N

==> /sys/module/kvm/holders/kvm_intel/parameters/ept <==
Y

==> /sys/module/kvm/holders/kvm_intel/parameters/eptad <==
N

==> /sys/module/kvm/holders/kvm_intel/parameters/fasteoi <==
Y

==> /sys/module/kvm/holders/kvm_intel/parameters/flexpriority <==
Y

==> /sys/module/kvm/holders/kvm_intel/parameters/nested <==
N

==> /sys/module/kvm/holders/kvm_intel/parameters/ple_gap <==
0

==> /sys/module/kvm/holders/kvm_intel/parameters/ple_window <==
4096

==> /sys/module/kvm/holders/kvm_intel/parameters/ple_window_grow <==
2

==> /sys/module/kvm/holders/kvm_intel/parameters/ple_window_max <==
1073741823

==> /sys/module/kvm/holders/kvm_intel/parameters/ple_window_shrink <==
0

==> /sys/module/kvm/holders/kvm_intel/parameters/pml <==
N

==> /sys/module/kvm/holders/kvm_intel/parameters/unrestricted_guest <==
N

==> /sys/module/kvm/holders/kvm_intel/parameters/vmm_exclusive <==
Y

==> /sys/module/kvm/holders/kvm_intel/parameters/vpid <==
Y

[Laszlo Ersek (Red Hat) (lersek)]

Thanks. It's indeed the same issue, you have unrestricted_guest=N and emulate_invalid_guest_state=Y.

[Laszlo Ersek (Red Hat) (lersek)]

The iPXE patches are now upstream (a big "thank you" to the iPXE maintainer!); QEMU 2.8 -- with Gerd willing -- should bundle iPXE binaries containing that fix.

http://lists.ipxe.org/pipermail/ipxe-devel/2016-November/005244.html

[Laszlo Ersek (Red Hat) (lersek)]

Fixed in:

commit 423f7cf233fe262c777db7f87db3e9fac29e02d1
Author: Gerd Hoffmann <email address hidden>
Date: Wed Nov 9 09:48:44 2016 +0100

    ipxe: update to 20161108 snapshot

[Thomas Huth (th-huth)]

Commit 423f7cf233fe262 has been released with QEMU v2.8