Illegal instruction in vp9_quantize_b_ssse3() in libvpx on 14.04.2

[David Lindsay]

While building Xpra - a VNC-like application that uses video codecs for image transport - from source, I ran into a documented SIGILL.

When Xpra starts up, I see:

  Program received signal SIGILL, Illegal instruction.
  0x00007fffe9150c27 in vp9_quantize_b_ssse3 () from /usr/lib/x86_64-linux-gnu/libvpx.so.1
  (gdb) bt
  #0 0x00007fffe9150c27 in vp9_quantize_b_ssse3 () from /usr/lib/x86_64-linux-gnu/libvpx.so.1
  ...

Then I `(gdb) disass $pc-32,$pc+32', and...

  Dump of assembler code from 0x7fffe9150c07 to 0x7fffe9150c47:
   0x00007fffe9150c07 <vp9_quantize_b_ssse3+551>: rex.B pshufw $0xe,%mm0,%mm7
   0x00007fffe9150c0c <vp9_quantize_b_ssse3+556>: pmaxsw %xmm7,%xmm8
   0x00007fffe9150c11 <vp9_quantize_b_ssse3+561>: pshuflw $0xe,%xmm8,%xmm7
   0x00007fffe9150c17 <vp9_quantize_b_ssse3+567>: pmaxsw %xmm7,%xmm8
   0x00007fffe9150c1c <vp9_quantize_b_ssse3+572>: pshuflw $0x1,%xmm8,%xmm7
   0x00007fffe9150c22 <vp9_quantize_b_ssse3+578>: pmaxsw %xmm7,%xmm8
=> 0x00007fffe9150c27 <vp9_quantize_b_ssse3+583>: pextrw $0x0,%xmm8,(%rdx)
   0x00007fffe9150c2e <vp9_quantize_b_ssse3+590>: retq
   0x00007fffe9150c2f <vp9_quantize_b_ssse3+591>: mov %r8,%rdi
   0x00007fffe9150c32 <vp9_quantize_b_ssse3+594>: mov %rcx,%rdx
   0x00007fffe9150c35 <vp9_quantize_b_ssse3+597>: mov 0x30(%rsp),%rcx
   0x00007fffe9150c3a <vp9_quantize_b_ssse3+602>: lea (%rdi,%rsi,2),%rdi
   0x00007fffe9150c3e <vp9_quantize_b_ssse3+606>: lea (%rdx,%rsi,2),%rdx
   0x00007fffe9150c42 <vp9_quantize_b_ssse3+610>: neg %rsi
   0x00007fffe9150c45 <vp9_quantize_b_ssse3+613>: pxor %xmm7,%xmm7

I Googled "libvpx illegal instruction" and discovered this bug report - https://trac.ffmpeg.org/ticket/3370 - detailing **exactly the same crash**. (It's where I got the gdb command from, actually.)

It wasn't an ffmpeg bug though, it's a WebM bug, and the problem was dutifully pushed upstream to https://bugs.chromium.org/p/webm/issues/detail?id=702 ...where it was apparently fixed.

It would seem Ubuntu's version of this package does not include this fix.

In my own case, Xpra will compile fine without WebM support and use H264 encoding instead, which actually seems to use less CPU, and is Xpra's default anyway. I'm reporting this issue primarily for the benefit of others.

For reference, some of my /proc/cpuinfo:

vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Pentium(R) Dual-Core CPU E5200 @ 2.50GHz
stepping : 10
microcode : 0xa07
...
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm xsave lahf_lm dtherm
...

The full crashdump/basic gdb analysis is attached. It also includes full /proc/cpuinfo at the end.