Since this bug has been around for almost *a year*, and it's extremely
annoying, I'm trying to work out if this is a bug in the guest kernel,
qemu, or libvirt.  I'm not any closer to working that out.

libvirt passes the following CPU/machine-related flags:

  -machine pc-i440fx-1.6,accel=kvm,usb=off
  -cpu Westmere,+rdtscp,+avx,+osxsave,+xsave,+tsc-deadline,+pcid,+pdcm,+xtpr,+tm2,+est,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
  -m 500
  -realtime mlock=off
  -smp 1,sockets=1,cores=1,threads=1

Host CPU flags are reported to be:

flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
cpuid level	: 13

Guest CPU flags are reported to be:

flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc rep_good nopl pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes avx hypervisor lahf_lm
cpuid level	: 13

(In reply to Eduardo Habkost from comment #12)
> "-cpu Westmere,+avx" actually should enable the bit on CPUID if and only if
> KVM is able to handle the feature. When KVM can't handle the feature, it
> should be filtered out before the guest CPUID table is built. I still don't
> understand why exactly the guest got an invalid operation exception, as the
> instruction was supposed to be working. Maybe it's related to the "level"
> field and the xsave feature (that is required for AVX, as far as I recall),
> that needs level >= 0xD.

I see the xsave flag in the host CPU flags, and in the libvirt-generated qemu
command line.  I do NOT see the xsave flag in the guest flags.  Not sure what
that means.

Assuming "level" means "cpuid level", then both report 13 == 0xD.

> I don't know if the guest is really allowed to use the feature when the AVX
> bit is set but the necessary xsave bits are not present (in it is not, then
> this is a guest bug).

As far as I can tell from the kernel code, cpu_has_avx just checks the
avx feature flag.  It doesn't check for xsave.  The xor code which is
throwing the invalid opcode is only checking cpu_has_avx, ie. only checking
for the avx flag.

According to the Intel PRM it does appear that you shouldn't use avx unless
xsave is supported, although it doesn't appear to be an absolute requirement.
I'm assuming it's something to do with those extra registers not being
saved over a context switch, which doesn't sound like an invalid opcode
situation to me (corrupt data OTOH).

Why would xsave bit not be present in the guest?

> If the guest was simply misled by the CPUID
> information, and correct in trying to use the instructions, it is a QEMU bug
> (QEMU should have disabled the feature, and abort in case the "enforce" flag
> is set). On either case, it is not a libvirt bug to ask for "-cpu
> Westmere,+avx".
> 
> But it would be interesting if libvirt could treat some CPU features as "can
> be safely disabled". It would be much better if libvirt used "-cpu
> SandyBridge,-x2apic" on that host, instead of "-cpu Westmere,+<lots of
> flags>".