OpenStack Compute (nova)

Ubuntu Xenial guests segfault on hosts with CPUs supporting AVX2

Bug #1578079 reported by Benoît Knecht on 2016-05-04

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Invalid	Undecided	Unassigned

Bug Description

Description
===========
When creating an Ubuntu Xenial instance from the official cloud image (https://cloud-images.ubuntu.com/xenial/current/), I noticed that sometimes it would boot just fine, sometimes it would crash almost immediately with a segmentation fault. After some investigation, I noticed that Xenial guest were crashing when running on my more recent hypervisors (Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz), while they would run just fine on my older hypervisors (Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz).

The difference between those two CPUs is that the newer one supports AVX2, while the older one supports AVX only.

I realize that this bug is actually a manifestation of #1524069, and has more to do with Libvirt/QEMU than Nova, but it also looks like Nova could be made to configure Libvirt in a way that would work around this bug, which would be great.

Steps to reproduce
==================
* openstack server create --image ubuntu-1604-xenial --flavor m1.small --nic net-id=c418b807-55e0-4bcf-b31d-6206594a5311 foo

Expected result
===============
Have a functioning Xenial instance.

Actual result
=============
The instance crashes shortly after loading raid6:

Begin: Loading essential drivers ... [ 3.549857] md: linear personality registered for level -1
[ 3.554621] md: multipath personality registered for level -4
[ 3.559487] md: raid0 personality registered for level 0
[ 3.564545] md: raid1 personality registered for level 1
[ 3.639509] raid6: sse2x1 gen() 6920 MB/s
[ 3.707518] raid6: sse2x1 xor() 5781 MB/s
[ 3.775508] raid6: sse2x2 gen() 9630 MB/s
[ 3.843506] raid6: sse2x2 xor() 5438 MB/s
[ 3.911506] raid6: sse2x4 gen() 11722 MB/s
[ 3.979503] raid6: sse2x4 xor() 7960 MB/s
[ 3.983515] invalid opcode: 0000 [#1] SMP
[ 3.984205] Modules linked in: raid6_pq(+) libcrc32c raid1 raid0 multipath linear psmouse floppy
[ 3.985859] CPU: 0 PID: 230 Comm: modprobe Not tainted 4.4.0-21-generic #37-Ubuntu
[ 3.986881] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Ubuntu-1.8.2-1ubuntu1~cloud0 04/01/2014
[ 3.988118] task: ffff8800bb1d7080 ti: ffff8800b9a48000 task.ti: ffff8800b9a48000
[ 3.989124] RIP: 0010:[<ffffffffc0061a8d>] [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq]
[ 3.992121] RSP: 0018:ffff8800b9a4bb78 EFLAGS: 00010246
[ 3.992808] RAX: 0000000000000000 RBX: ffff8800b9a4bbc8 RCX: 00000000fffedeae
[ 3.993669] RDX: 0000000000000080 RSI: 0000000000001000 RDI: 0000000000000012
[ 3.994541] RBP: ffff8800b9a4bba8 R08: 0000000000000000 R09: 00000000000001bd
[ 3.995446] R10: 00000000fffede9d R11: 00000000000001bd R12: 0000000000001000
[ 3.996307] R13: ffff8800bb022000 R14: ffff8800bb023000 R15: 0000000000000012
[ 3.997167] FS: 00007fa2699f6700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[ 3.998245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3.998974] CR2: 000056292fa0b008 CR3: 00000000b9a19000 CR4: 00000000001006f0
[ 3.999840] Stack:
[ 4.000204] 0000000000000080 ffffffffc00747a0 0000000000000001 0000000000003fb0
[ 4.001439] ffffffffc0062238 ffff8800bb022000 ffff8800b9a4bc88 ffffffffc0079116
[ 4.002684] 00000000fffedeae 0000000000002ee4 ffffffffc0064600 ffffffffc0065600
[ 4.004020] Call Trace:
[ 4.004429] [<ffffffffc0079116>] init_module+0x116/0x1000 [raid6_pq]
[ 4.005243] [<ffffffffc0079000>] ? 0xffffffffc0079000
[ 4.005917] [<ffffffff81002123>] do_one_initcall+0xb3/0x200
[ 4.006651] [<ffffffff818205f8>] ? preempt_schedule_common+0x18/0x30
[ 4.007451] [<ffffffff8182062c>] ? _cond_resched+0x1c/0x30
[ 4.008214] [<ffffffff811eaeb3>] ? kmem_cache_alloc_trace+0x183/0x1f0
[ 4.009021] [<ffffffff8118c163>] do_init_module+0x5f/0x1cf
[ 4.009734] [<ffffffff81109df7>] load_module+0x1667/0x1c00
[ 4.010458] [<ffffffff811063a0>] ? __symbol_put+0x60/0x60
[ 4.011163] [<ffffffff812126b0>] ? kernel_read+0x50/0x80
[ 4.011861] [<ffffffff8110a5d4>] SYSC_finit_module+0xb4/0xe0
[ 4.012590] [<ffffffff8110a61e>] SyS_finit_module+0xe/0x10
[ 4.013302] [<ffffffff818244f2>] entry_SYSCALL_64_fastpath+0x16/0x71
[ 4.014104] Code: 55 41 54 53 48 89 d3 48 8d 14 c5 00 00 00 00 41 89 ff 49 89 f4 48 83 ec 08 4c 8b 2c c3 4c 8b 74 13 08 48 89 55 d0 e8 53 83 fd c0 <c5> fd 6f 05 8b 2e 01 00 c5 e5 ef db 4d 85 e4 48 8b 55 d0 0f 84
[ 4.020068] RIP [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq]
[ 4.023317] RSP <ffff8800b9a4bb78>
[ 4.023847] ---[ end trace b3853dc6e5fc1f8f ]---
Segmentation fault

Environment
===========
1. I'm running OpenStack Liberty:

ii nova-common 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:2.30.1-1~cloud0 all client library for OpenStack Compute API

2. I'm using Libvirt (1.2.16-2ubuntu11.15.10.3~cloud0) + KVM (1:2.3+dfsg-5ubuntu9.2~cloud0)

3. I'm using Ceph 9.2.1-1trusty as a storage backend

4. I'm using Neutron with Openvswitch for networking

Logs & Configs
==============
Here's the full console log for an instance that crashes:

[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.4.0-21-generic (buildd@lgw01-21) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 (Ubuntu 4.4.0-21.37-generic 4.4.6)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] x86/fpu: Legacy x87 FPU detected.
[ 0.000000] x86/fpu: Using 'lazy' FPU context switches.
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdffff] usable
[ 0.000000] BIOS-e820: [mem 0x00000000bffe0000-0x00000000bfffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] Hypervisor detected: KVM
[ 0.000000] e820: last_pfn = 0x140000 max_arch_pfn = 0x400000000
[ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT
[ 0.000000] e820: last_pfn = 0xbffe0 max_arch_pfn = 0x400000000
[ 0.000000] found SMP MP-table at [mem 0x000f6470-0x000f647f] mapped at [ffff8800000f6470]
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] Using GB pages for direct mapping
[ 0.000000] RAMDISK: [mem 0x371c6000-0x378dafff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000F6240 000014 (v00 BOCHS )
[ 0.000000] ACPI: RSDT 0x00000000BFFE1672 000030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACP 0x00000000BFFE0BDA 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
[ 0.000000] ACPI: DSDT 0x00000000BFFE0040 000B9A (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040
[ 0.000000] ACPI: SSDT 0x00000000BFFE0C4E 0009AC (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: APIC 0x00000000BFFE15FA 000078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013fffffff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x13fff9000-0x13fffdfff]
[ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[ 0.000000] kvm-clock: cpu 0, msr 1:3fff5001, primary cpu clock
[ 0.000000] kvm-clock: using sched offset of 1523727129 cycles
[ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x000000013fffffff]
[ 0.000000] Device empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff]
[ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffdffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x000000013fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000013fffffff]
[ 0.000000] ACPI: PM-Timer IO Port: 0x608
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
[ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbffe0000-0xbfffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
[ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices
[ 0.000000] Booting paravirtualized kernel on KVM
[ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[ 0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:1 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88013fc00000 s98008 r8192 d28968 u2097152
[ 0.000000] KVM setup async PF for cpu 0
[ 0.000000] kvm-stealtime: cpu 0, msr 13fc0d940
[ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1032041
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Memory: 4035988K/4193784K available (8356K kernel code, 1278K rwdata, 3920K rodata, 1476K init, 1292K bss, 157796K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 64.
[ 0.000000] RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=1.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1
[ 0.000000] NR_IRQS:16640 nr_irqs:256 16
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [tty1] enabled
[ 0.000000] console [ttyS0] enabled
[ 0.000000] tsc: Detected 2599.996 MHz processor
[ 0.203234] Calibrating delay loop (skipped) preset value.. 5199.99 BogoMIPS (lpj=10399984)
[ 0.205106] pid_max: default: 32768 minimum: 301
[ 0.206149] ACPI: Core revision 20150930
[ 0.208546] ACPI: 2 ACPI AML tables successfully acquired and loaded
[ 0.210119] Security Framework initialized
[ 0.211083] Yama: becoming mindful.
[ 0.211892] AppArmor: AppArmor initialized
[ 0.213706] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[ 0.218689] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[ 0.220490] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.221766] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.223302] Initializing cgroup subsys io
[ 0.224173] Initializing cgroup subsys memory
[ 0.225039] Initializing cgroup subsys devices
[ 0.225972] Initializing cgroup subsys freezer
[ 0.226861] Initializing cgroup subsys net_cls
[ 0.227736] Initializing cgroup subsys perf_event
[ 0.228640] Initializing cgroup subsys net_prio
[ 0.229564] Initializing cgroup subsys hugetlb
[ 0.230503] Initializing cgroup subsys pids
[ 0.231432] CPU: CPU feature xsave disabled, no CPUID level 0xd
[ 0.233625] mce: CPU supports 10 MCE banks
[ 0.234535] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[ 0.235488] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
[ 0.250916] Freeing SMP alternatives memory: 28K (ffffffff820b2000 - ffffffff820b9000)
[ 0.258620] ftrace: allocating 31878 entries in 125 pages
[ 0.292291] smpboot: Max logical packages: 1
[ 0.292979] smpboot: APIC(0) Converting physical 0 to logical package 0
[ 0.294062] x2apic enabled
[ 0.294775] Switched APIC routing to physical x2apic.
[ 0.296242] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.297081] smpboot: CPU0: Intel Core i7 9xx (Nehalem Class Core i7) (family: 0x6, model: 0x1a, stepping: 0x3)
[ 0.298727] Performance Events: unsupported p6 CPU model 26 no PMU driver, software events only.
[ 0.300782] x86: Booted up 1 node, 1 CPUs
[ 0.301405] smpboot: Total of 1 processors activated (5199.99 BogoMIPS)
[ 0.303089] devtmpfs: initialized
[ 0.306053] evm: security.selinux
[ 0.306590] evm: security.SMACK64
[ 0.308354] evm: security.SMACK64EXEC
[ 0.308915] evm: security.SMACK64TRANSMUTE
[ 0.309542] evm: security.SMACK64MMAP
[ 0.310113] evm: security.ima
[ 0.310586] evm: security.capability
[ 0.311236] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.312590] pinctrl core: initialized pinctrl subsystem
[ 0.313427] RTC time: 5:35:03, date: 05/04/16
[ 0.315133] NET: Registered protocol family 16
[ 0.315908] cpuidle: using governor ladder
[ 0.316534] cpuidle: using governor menu
[ 0.317108] PCCT header not found.
[ 0.317686] ACPI: bus type PCI registered
[ 0.318298] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 0.319192] PCI: Using configuration type 1 for base access
[ 0.320935] ACPI: Added _OSI(Module Device)
[ 0.321524] ACPI: Added _OSI(Processor Device)
[ 0.322129] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 0.322761] ACPI: Added _OSI(Processor Aggregator Device)
[ 0.324221] ACPI: Interpreter enabled
[ 0.324764] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20150930/hwxface-580)
[ 0.326118] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20150930/hwxface-580)
[ 0.327447] ACPI: (supports S0 S3 S4 S5)
[ 0.328000] ACPI: Using IOAPIC for interrupt routing
[ 0.328670] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.331711] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 0.332501] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
[ 0.333346] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[ 0.334219] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[ 0.335856] acpiphp: Slot [3] registered
[ 0.336431] acpiphp: Slot [4] registered
[ 0.336997] acpiphp: Slot [5] registered
[ 0.337563] acpiphp: Slot [6] registered
[ 0.339449] acpiphp: Slot [7] registered
[ 0.348585] acpiphp: Slot [8] registered
[ 0.349160] acpiphp: Slot [9] registered
[ 0.349731] acpiphp: Slot [10] registered
[ 0.350315] acpiphp: Slot [11] registered
[ 0.350891] acpiphp: Slot [12] registered
[ 0.351465] acpiphp: Slot [13] registered
[ 0.352053] acpiphp: Slot [14] registered
[ 0.352637] acpiphp: Slot [15] registered
[ 0.353224] acpiphp: Slot [16] registered
[ 0.353799] acpiphp: Slot [17] registered
[ 0.354382] acpiphp: Slot [18] registered
[ 0.354956] acpiphp: Slot [19] registered
[ 0.355533] acpiphp: Slot [20] registered
[ 0.356105] acpiphp: Slot [21] registered
[ 0.356686] acpiphp: Slot [22] registered
[ 0.357261] acpiphp: Slot [23] registered
[ 0.357834] acpiphp: Slot [24] registered
[ 0.358500] acpiphp: Slot [25] registered
[ 0.359114] acpiphp: Slot [26] registered
[ 0.359721] acpiphp: Slot [27] registered
[ 0.360294] acpiphp: Slot [28] registered
[ 0.360884] acpiphp: Slot [29] registered
[ 0.361463] acpiphp: Slot [30] registered
[ 0.362061] acpiphp: Slot [31] registered
[ 0.362640] PCI host bridge to bus 0000:00
[ 0.363218] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
[ 0.364054] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]
[ 0.364895] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[ 0.365913] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
[ 0.366933] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.377218] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7]
[ 0.378115] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6]
[ 0.379563] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177]
[ 0.380477] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376]
[ 0.387873] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI
[ 0.388875] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB
[ 0.465906] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[ 0.467014] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[ 0.468064] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[ 0.469125] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[ 0.471532] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
[ 0.472585] ACPI: Enabled 16 GPEs in block 00 to 0F
[ 0.473485] vgaarb: setting as boot device: PCI:0000:00:02.0
[ 0.474215] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.475270] vgaarb: loaded
[ 0.475696] vgaarb: bridge control possible 0000:00:02.0
[ 0.476672] SCSI subsystem initialized
[ 0.477271] ACPI: bus type USB registered
[ 0.477860] usbcore: registered new interface driver usbfs
[ 0.478614] usbcore: registered new interface driver hub
[ 0.479309] usbcore: registered new device driver usb
[ 0.480158] PCI: Using ACPI for IRQ routing
[ 0.480957] NetLabel: Initializing
[ 0.481458] NetLabel: domain hash size = 128
[ 0.482057] NetLabel: protocols = UNLABELED CIPSOv4
[ 0.482716] NetLabel: unlabeled traffic allowed by default
[ 0.483498] clocksource: Switched to clocksource kvm-clock
[ 0.490934] AppArmor: AppArmor Filesystem Enabled
[ 0.491635] pnp: PnP ACPI init
[ 0.492436] pnp: PnP ACPI: found 6 devices
[ 0.500252] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[ 0.501486] NET: Registered protocol family 2
[ 0.502247] TCP established hash table entries: 32768 (order: 6, 262144 bytes)
[ 0.503283] TCP bind hash table entries: 32768 (order: 7, 524288 bytes)
[ 0.504146] TCP: Hash tables configured (established 32768 bind 32768)
[ 0.504963] UDP hash table entries: 2048 (order: 4, 65536 bytes)
[ 0.505719] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
[ 0.506557] NET: Registered protocol family 1
[ 0.507157] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 0.507914] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[ 0.508646] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[ 0.510002] Trying to unpack rootfs image as initramfs...
[ 1.717098] Freeing initrd memory: 7252K (ffff8800371c6000 - ffff8800378db000)
[ 1.718412] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 1.719192] software IO TLB [mem 0xbbfe0000-0xbffe0000] (64MB) mapped at [ffff8800bbfe0000-ffff8800bffdffff]
[ 1.725831] Scanning for low memory corruption every 60 seconds
[ 1.731157] futex hash table entries: 256 (order: 2, 16384 bytes)
[ 1.731891] audit: initializing netlink subsys (disabled)
[ 1.732583] audit: type=2000 audit(1462340105.446:1): initialized
[ 1.733541] Initialise system trusted keyring
[ 1.734188] HugeTLB registered 1 GB page size, pre-allocated 0 pages
[ 1.734913] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 1.736700] zbud: loaded
[ 1.737254] VFS: Disk quotas dquot_6.6.0
[ 1.737797] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 1.738984] fuse init (API version 7.23)
[ 1.739650] Key type big_key registered
[ 1.740289] Key type asymmetric registered
[ 1.740823] Asymmetric key parser 'x509' registered
[ 1.741422] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[ 1.742418] io scheduler noop registered
[ 1.742951] io scheduler deadline registered (default)
[ 1.743580] io scheduler cfq registered
[ 1.744138] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 1.744813] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[ 1.745653] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[ 1.746619] ACPI: Power Button [PWRF]
[ 1.747193] GHES: HEST is not enabled!
[ 1.763884] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[ 1.764640] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
[ 1.782055] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
[ 1.782747] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
[ 1.799998] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
[ 1.800665] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
[ 1.817293] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
[ 1.817943] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver
[ 1.819651] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[ 1.842639] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 1.866310] 00:05: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
[ 1.887604] Linux agpgart interface v0.103
[ 1.889620] brd: module loaded
[ 1.890749] loop: module loaded
[ 1.908069] vda: vda1
[ 1.909586] scsi host0: ata_piix
[ 1.910066] scsi host1: ata_piix
[ 1.910562] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0e0 irq 14
[ 1.911256] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0e8 irq 15
[ 1.912214] libphy: Fixed MDIO Bus: probed
[ 1.912753] tun: Universal TUN/TAP device driver, 1.6
[ 1.913312] tun: (C) 1999-2004 Max Krasnyansky <email address hidden>
[ 1.930649] PPP generic driver version 2.4.2
[ 1.931288] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 1.932010] ehci-pci: EHCI PCI platform driver
[ 1.932597] ehci-platform: EHCI generic platform driver
[ 1.933234] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 1.933950] ohci-pci: OHCI PCI platform driver
[ 1.934539] ohci-platform: OHCI generic platform driver
[ 1.935191] uhci_hcd: USB Universal Host Controller Interface driver
[ 1.952374] uhci_hcd 0000:00:01.2: UHCI Host Controller
[ 1.953034] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
[ 1.953952] uhci_hcd 0000:00:01.2: detected 2 ports
[ 1.954651] uhci_hcd 0000:00:01.2: irq 10, io base 0x0000c040
[ 1.955379] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
[ 1.956168] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.957139] usb usb1: Product: UHCI Host Controller
[ 1.957768] usb usb1: Manufacturer: Linux 4.4.0-21-generic uhci_hcd
[ 1.958543] usb usb1: SerialNumber: 0000:00:01.2
[ 1.959264] hub 1-0:1.0: USB hub found
[ 1.959936] hub 1-0:1.0: 2 ports detected
[ 1.960616] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[ 1.962228] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 1.962900] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 1.963656] mousedev: PS/2 mouse device common for all mice
[ 1.964524] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
[ 1.965802] rtc_cmos 00:00: RTC can wake from S4
[ 1.966583] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
[ 1.967397] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram
[ 1.968119] i2c /dev entries driver
[ 1.968702] device-mapper: uevent: version 1.0.3
[ 1.969336] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: <email address hidden>
[ 1.970428] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.971346] NET: Registered protocol family 10
[ 1.972135] NET: Registered protocol family 17
[ 1.972732] Key type dns_resolver registered
[ 1.973409] microcode: CPU0 sig=0x106a3, pf=0x1, revision=0x1
[ 1.974182] microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba
[ 1.975389] registered taskstats version 1
[ 1.975957] Loading compiled-in X.509 certificates
[ 1.977206] Loaded X.509 cert 'Build time autogenerated kernel key: fc7c0e9f152f32eca50ea2d9722926e5127af244'
[ 1.978456] zswap: loaded using pool lzo/zbud
[ 1.980197] Key type trusted registered
[ 1.992720] Key type encrypted registered
[ 1.993250] AppArmor: AppArmor sha1 policy hashing enabled
[ 1.993898] ima: No TPM chip found, activating TPM-bypass!
[ 1.994625] evm: HMAC attrs: 0x1
[ 1.995271] Magic number: 0:849:566
[ 1.995909] rtc_cmos 00:00: setting system clock to 2016-05-04 05:35:05 UTC (1462340105)
[ 1.997035] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[ 1.997702] EDD information not available.
[ 2.073644] Freeing unused kernel memory: 1476K (ffffffff81f41000 - ffffffff820b2000)
[ 2.075469] Write protecting the kernel read-only data: 14336k
[ 2.077486] Freeing unused kernel memory: 1872K (ffff88000182c000 - ffff880001a00000)
[ 2.079999] Freeing unused kernel memory: 176K (ffff880001dd4000 - ffff880001e00000)
Loading, please wait...
starting version 229
[ 2.095265] random: udevadm urandom read with 2 bits of entropy available
[ 2.140733] virtio_net virtio0 ens3: renamed from eth0
[ 2.148962] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4
[ 2.150419] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3
[ 2.179873] FDC 0 is a S82078B
[ 2.723675] tsc: Refined TSC clocksource calibration: 2600.056 MHz
[ 2.725415] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x257a71592ee, max_idle_ns: 440795334892 ns
Begin: Loading essential drivers ... [ 3.549857] md: linear personality registered for level -1
[ 3.554621] md: multipath personality registered for level -4
[ 3.559487] md: raid0 personality registered for level 0
[ 3.564545] md: raid1 personality registered for level 1
[ 3.639509] raid6: sse2x1 gen() 6920 MB/s
[ 3.707518] raid6: sse2x1 xor() 5781 MB/s
[ 3.775508] raid6: sse2x2 gen() 9630 MB/s
[ 3.843506] raid6: sse2x2 xor() 5438 MB/s
[ 3.911506] raid6: sse2x4 gen() 11722 MB/s
[ 3.979503] raid6: sse2x4 xor() 7960 MB/s
[ 3.983515] invalid opcode: 0000 [#1] SMP
[ 3.984205] Modules linked in: raid6_pq(+) libcrc32c raid1 raid0 multipath linear psmouse floppy
[ 3.985859] CPU: 0 PID: 230 Comm: modprobe Not tainted 4.4.0-21-generic #37-Ubuntu
[ 3.986881] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Ubuntu-1.8.2-1ubuntu1~cloud0 04/01/2014
[ 3.988118] task: ffff8800bb1d7080 ti: ffff8800b9a48000 task.ti: ffff8800b9a48000
[ 3.989124] RIP: 0010:[<ffffffffc0061a8d>] [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq]
[ 3.992121] RSP: 0018:ffff8800b9a4bb78 EFLAGS: 00010246
[ 3.992808] RAX: 0000000000000000 RBX: ffff8800b9a4bbc8 RCX: 00000000fffedeae
[ 3.993669] RDX: 0000000000000080 RSI: 0000000000001000 RDI: 0000000000000012
[ 3.994541] RBP: ffff8800b9a4bba8 R08: 0000000000000000 R09: 00000000000001bd
[ 3.995446] R10: 00000000fffede9d R11: 00000000000001bd R12: 0000000000001000
[ 3.996307] R13: ffff8800bb022000 R14: ffff8800bb023000 R15: 0000000000000012
[ 3.997167] FS: 00007fa2699f6700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[ 3.998245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3.998974] CR2: 000056292fa0b008 CR3: 00000000b9a19000 CR4: 00000000001006f0
[ 3.999840] Stack:
[ 4.000204] 0000000000000080 ffffffffc00747a0 0000000000000001 0000000000003fb0
[ 4.001439] ffffffffc0062238 ffff8800bb022000 ffff8800b9a4bc88 ffffffffc0079116
[ 4.002684] 00000000fffedeae 0000000000002ee4 ffffffffc0064600 ffffffffc0065600
[ 4.004020] Call Trace:
[ 4.004429] [<ffffffffc0079116>] init_module+0x116/0x1000 [raid6_pq]
[ 4.005243] [<ffffffffc0079000>] ? 0xffffffffc0079000
[ 4.005917] [<ffffffff81002123>] do_one_initcall+0xb3/0x200
[ 4.006651] [<ffffffff818205f8>] ? preempt_schedule_common+0x18/0x30
[ 4.007451] [<ffffffff8182062c>] ? _cond_resched+0x1c/0x30
[ 4.008214] [<ffffffff811eaeb3>] ? kmem_cache_alloc_trace+0x183/0x1f0
[ 4.009021] [<ffffffff8118c163>] do_init_module+0x5f/0x1cf
[ 4.009734] [<ffffffff81109df7>] load_module+0x1667/0x1c00
[ 4.010458] [<ffffffff811063a0>] ? __symbol_put+0x60/0x60
[ 4.011163] [<ffffffff812126b0>] ? kernel_read+0x50/0x80
[ 4.011861] [<ffffffff8110a5d4>] SYSC_finit_module+0xb4/0xe0
[ 4.012590] [<ffffffff8110a61e>] SyS_finit_module+0xe/0x10
[ 4.013302] [<ffffffff818244f2>] entry_SYSCALL_64_fastpath+0x16/0x71
[ 4.014104] Code: 55 41 54 53 48 89 d3 48 8d 14 c5 00 00 00 00 41 89 ff 49 89 f4 48 83 ec 08 4c 8b 2c c3 4c 8b 74 13 08 48 89 55 d0 e8 53 83 fd c0 <c5> fd 6f 05 8b 2e 01 00 c5 e5 ef db 4d 85 e4 48 8b 55 d0 0f 84
[ 4.020068] RIP [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq]
[ 4.023317] RSP <ffff8800b9a4bb78>
[ 4.023847] ---[ end trace b3853dc6e5fc1f8f ]---
Segmentation fault

And here's the corresponding Libvirt XML:

<domain type='kvm' id='411'>
  <name>instance-000030cd</name>
  <uuid>587a3497-3198-4c19-87c4-ddf449385f71</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="12.0.1"/>
      <nova:name>test-xenial-segfault</nova:name>
      <nova:creationTime>2016-05-04 05:39:55</nova:creationTime>
      <nova:flavor name="m1.small">
        <nova:memory>4096</nova:memory>
        <nova:disk>16</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="07d961b93aed45c0b7989f91c40341ae">username</nova:user>
        <nova:project uuid="dd4f48116bd942f9b145d20e9f1a11be">project</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="4b5b61a5-008c-4424-a899-f6896384a55c"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>12.0.1</entry>
      <entry name='serial'>00000000-0000-0000-0000-0cc47a349290</entry>
      <entry name='uuid'>587a3497-3198-4c19-87c4-ddf449385f71</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-vivid'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='23c33478-af4c-49d0-bb46-9890d6b89084'/>
      </auth>
      <source protocol='rbd' name='libvirt/587a3497-3198-4c19-87c4-ddf449385f71_disk'>
        <host name='172.24.3.17' port='6789'/>
        <host name='172.24.3.18' port='6789'/>
        <host name='172.24.3.19' port='6789'/>
        <host name='172.24.3.25' port='6789'/>
        <host name='172.24.3.26' port='6789'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:3c:5e:88'/>
      <source bridge='qbrba81b423-ab'/>
      <target dev='tapba81b423-ab'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/587a3497-3198-4c19-87c4-ddf449385f71/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/34'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/587a3497-3198-4c19-87c4-ddf449385f71/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='pty'>
      <source path='/dev/pts/35'/>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' port='5915' autoport='yes' listen='172.24.1.35' keymap='fr-ch'>
      <listen type='address' address='172.24.1.35'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-587a3497-3198-4c19-87c4-ddf449385f71</label>
    <imagelabel>libvirt-587a3497-3198-4c19-87c4-ddf449385f71</imagelabel>
  </seclabel>
</domain>

Tags: