| 2014-10-09 13:31:07 |
Blair Bethwaite |
bug |
|
|
added bug |
| 2014-10-09 13:32:36 |
Blair Bethwaite |
bug |
|
|
added subscriber Swe Win Aung |
| 2014-10-09 23:50:01 |
Launchpad Janitor |
qemu-kvm (Ubuntu): status |
New |
Confirmed |
|
| 2014-10-16 09:24:26 |
Serge Hallyn |
bug task added |
|
qemu (Ubuntu) |
|
| 2014-10-16 09:24:26 |
Serge Hallyn |
bug task added |
|
qemu |
|
| 2014-10-16 09:29:07 |
Serge Hallyn |
bug task deleted |
qemu-kvm (Ubuntu) |
|
|
| 2014-10-29 17:18:06 |
Serge Hallyn |
summary |
qemu-kvm guest panic for smp trusty guests |
qemu-kvm guest panic for AMD smp trusty guests |
|
| 2014-11-16 08:08:03 |
Paolo Bonzini |
bug task deleted |
qemu |
|
|
| 2014-11-18 17:45:03 |
Serge Hallyn |
bug task added |
|
linux (Ubuntu) |
|
| 2014-11-18 18:00:08 |
Brad Figg |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2014-11-18 18:00:10 |
Brad Figg |
tags |
|
trusty |
|
| 2014-11-18 22:37:23 |
Chris J Arges |
qemu (Ubuntu): assignee |
|
Chris J Arges (arges) |
|
| 2014-11-18 22:37:26 |
Chris J Arges |
qemu (Ubuntu): status |
New |
In Progress |
|
| 2014-11-18 22:37:33 |
Chris J Arges |
nominated for series |
|
Ubuntu Trusty |
|
| 2014-11-18 22:37:33 |
Chris J Arges |
bug task added |
|
qemu (Ubuntu Trusty) |
|
| 2014-11-18 22:37:33 |
Chris J Arges |
bug task added |
|
linux (Ubuntu Trusty) |
|
| 2014-11-18 22:39:17 |
Chris J Arges |
tags |
trusty |
fixed-upstream trusty |
|
| 2014-11-21 04:49:27 |
Serge Hallyn |
tags |
fixed-upstream trusty |
amd fixed-upstream trusty |
|
| 2014-11-21 16:13:56 |
Chris J Arges |
linux (Ubuntu): assignee |
|
Chris J Arges (arges) |
|
| 2014-11-21 16:13:59 |
Chris J Arges |
linux (Ubuntu): status |
Incomplete |
In Progress |
|
| 2014-11-21 16:14:05 |
Chris J Arges |
bug task deleted |
qemu (Ubuntu) |
|
|
| 2014-11-21 16:14:13 |
Chris J Arges |
bug task deleted |
qemu (Ubuntu Trusty) |
|
|
| 2014-11-21 16:14:20 |
Chris J Arges |
nominated for series |
|
Ubuntu Utopic |
|
| 2014-11-21 16:14:20 |
Chris J Arges |
bug task added |
|
linux (Ubuntu Utopic) |
|
| 2014-11-21 16:14:28 |
Chris J Arges |
linux (Ubuntu Trusty): assignee |
|
Chris J Arges (arges) |
|
| 2014-11-21 16:14:31 |
Chris J Arges |
linux (Ubuntu Utopic): assignee |
|
Chris J Arges (arges) |
|
| 2014-11-24 22:11:52 |
Chris J Arges |
linux (Ubuntu): status |
In Progress |
Fix Released |
|
| 2014-11-26 18:06:53 |
Chris J Arges |
description |
Just upgraded OpenStack compute hosts in our public cloud (using qemu-kvm via libvirt) from Precise to Trusty (14.04.1), now on kernel 3.13.0-36-generic with qemu-kvm 2.0.0+dfsg-2ubuntu1.5.
Following the upgrade, whenever we try to start an smp/multicore Trusty guest (existing or new), we run into this panic [1] inside the guest just towards the end of boot. This happens consistently for smp guests using the Trusty kernel (i.e., it also affects earlier Ubuntus using the HWE kernel from Trusty but not their native versions). I didn't have any other distro images to hand with 3.13.x kernels, but none of the others I tested were affected (in the 3.2 - 3.16 kernel range).
There are scarce similar reports out there, but the one we did find pointed to a CPU feature as the trigger. We were running these hosts with libvirt cpu mode set to "host-passthrough" (so qemu starts with "-cpu host"), on AMD 6200 & 6300 Opteron hardware. Switching the guest domains to use cpu mode "host-model" instead works around the issue and is perfectly acceptable for most of our users.
We have various other Intel compute hosts and they don't seem to be affected.
(1)
[ 11.256924] divide error: 0000 [#1] SMP
[ 11.258133] Modules linked in: kvm_amd kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw lp parport psmouse floppy
[ 11.260228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-36-generic #63-Ubuntu
[ 11.260228] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Bochs 01/01/2011
[ 11.260228] task: ffffffff81c15480 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[ 11.260228] RIP: 0010:[<ffffffff8104ed58>] [<ffffffff8104ed58>] kvm_unlock_kick+0xa8/0x100
[ 11.260228] RSP: 0018:ffff88023fc03c98 EFLAGS: 00010046
[ 11.260228] RAX: 0000000000000005 RBX: 0000000000000000 RCX: 0000000000000001
[ 11.260228] RDX: ffffffff81eaf408 RSI: 0000000000000000 RDI: 0000000000000000
[ 11.260228] RBP: ffff88023fc03cb8 R08: ffffffff81eaf400 R09: 00000000ffffffff
[ 11.260228] R10: ffff880037612cc0 R11: ffffea0002eb0a00 R12: ffff8800374a33c0
[ 11.260228] R13: 0000000000000020 R14: 0000000000000001 R15: 0000000000000286
[ 11.260228] FS: 00007f1e8b538740(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
[ 11.260228] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 11.260228] CR2: 00007f1e8ae09d50 CR3: 0000000001c0e000 CR4: 00000000000406f0
[ 11.260228] Stack:
[ 11.260228] 0000000000000286 0000000000000001 0000000000000001 00000000000000c3
[ 11.260228] ffff88023fc03cc8 ffffffff81717ed6 ffff88023fc03ce0 ffffffff8172641a
[ 11.260228] ffff8800374a33c0 ffff88023fc03d18 ffffffff810aaeb0 ffff88023295e000
[ 11.260228] Call Trace:
[ 11.260228] <IRQ>
[ 11.260228] [<ffffffff81717ed6>] __ticket_unlock_slowpath+0x24/0x34
[ 11.260228] [<ffffffff8172641a>] _raw_spin_unlock_irqrestore+0x3a/0x40
[ 11.260228] [<ffffffff810aaeb0>] __wake_up_sync_key+0x50/0x60
[ 11.260228] [<ffffffff8160ca5a>] sock_def_readable+0x3a/0x70
[ 11.260228] [<ffffffff816fda0a>] packet_rcv+0x2fa/0x430
[ 11.260228] [<ffffffff816228b0>] __netif_receive_skb_core+0x360/0x840
[ 11.260228] [<ffffffff81622da8>] __netif_receive_skb+0x18/0x60
[ 11.260228] [<ffffffff81622e13>] netif_receive_skb+0x23/0x90
[ 11.260228] [<ffffffff815288d4>] virtnet_poll+0x4d4/0x850
[ 11.260228] [<ffffffff81623192>] net_rx_action+0x152/0x250
[ 11.260228] [<ffffffff8106cbac>] __do_softirq+0xec/0x2c0
[ 11.260228] [<ffffffff8106d0f5>] irq_exit+0x105/0x110
[ 11.260228] [<ffffffff817312d6>] do_IRQ+0x56/0xc0
[ 11.260228] [<ffffffff81726a6d>] common_interrupt+0x6d/0x6d
[ 11.260228] <EOI>
[ 11.260228] [<ffffffff8104f596>] ? native_safe_halt+0x6/0x10
[ 11.260228] [<ffffffff8101c62f>] default_idle+0x1f/0xc0
[ 11.260228] [<ffffffff8101cef6>] arch_cpu_idle+0x26/0x30
[ 11.260228] [<ffffffff810bed95>] cpu_startup_entry+0xc5/0x290
[ 11.260228] [<ffffffff8170ca77>] rest_init+0x77/0x80
[ 11.260228] [<ffffffff81d35f6b>] start_kernel+0x433/0x43e
[ 11.260228] [<ffffffff81d35941>] ? repair_env_string+0x5c/0x5c
[ 11.260228] [<ffffffff81d35120>] ? early_idt_handlers+0x120/0x120
[ 11.260228] [<ffffffff81d355ee>] x86_64_start_reservations+0x2a/0x2c
[ 11.260228] [<ffffffff81d35733>] x86_64_start_kernel+0x143/0x152
[ 11.260228] Code: 66 44 39 e8 75 bd 0f b6 35 f6 06 e6 00 40 84 f6 75 2a 83 05 06 07 e6 00 01 48 c7 c0 6a b0 00 00 31 db 0f b7 0c 01 b8 05 00 00 00 <0f> 01 c1 0f 1f 44 00 00 5b 41 5c 41 5d 41 5e 5d c3 89 f0 31 c9
[ 11.260228] RIP [<ffffffff8104ed58>] kvm_unlock_kick+0xa8/0x100
[ 11.260228] RSP <ffff88023fc03c98>
[ 11.260228] ---[ end trace f1c26ff24745b331 ]---
[ 11.260228] Kernel panic - not syncing: Fatal exception in interrupt
[ 11.260228] Shutting down cpus with NMI |
[Impact]
When using KVM on an AMD host with a kernel that has CONFIG_DEBUG_RODATA enabled, a guest with: multiple vCPUs, and exposing features to the guest such as tsc_adjust can cause a divide error on kvm_unlock_kick when booting the VM.
[Test Case]
1) Create a VM on an AMD host with appropriate features (Opteron 6xxx for example)
2) Edit virsh xml to have <cpu mode='host-passthrough'></cpu> and multiple vCPUs.
3) Boot VM with VGA console using virt-manager (I couldn't reproduce strictly monitoring via virsh console).
[Fix]
commit c1118b3602c2329671ad5ec8bdf8e374323d6343 upstream
--
Just upgraded OpenStack compute hosts in our public cloud (using qemu-kvm via libvirt) from Precise to Trusty (14.04.1), now on kernel 3.13.0-36-generic with qemu-kvm 2.0.0+dfsg-2ubuntu1.5.
Following the upgrade, whenever we try to start an smp/multicore Trusty guest (existing or new), we run into this panic [1] inside the guest just towards the end of boot. This happens consistently for smp guests using the Trusty kernel (i.e., it also affects earlier Ubuntus using the HWE kernel from Trusty but not their native versions). I didn't have any other distro images to hand with 3.13.x kernels, but none of the others I tested were affected (in the 3.2 - 3.16 kernel range).
There are scarce similar reports out there, but the one we did find pointed to a CPU feature as the trigger. We were running these hosts with libvirt cpu mode set to "host-passthrough" (so qemu starts with "-cpu host"), on AMD 6200 & 6300 Opteron hardware. Switching the guest domains to use cpu mode "host-model" instead works around the issue and is perfectly acceptable for most of our users.
We have various other Intel compute hosts and they don't seem to be affected.
(1)
[ 11.256924] divide error: 0000 [#1] SMP
[ 11.258133] Modules linked in: kvm_amd kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw lp parport psmouse floppy
[ 11.260228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-36-generic #63-Ubuntu
[ 11.260228] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Bochs 01/01/2011
[ 11.260228] task: ffffffff81c15480 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[ 11.260228] RIP: 0010:[<ffffffff8104ed58>] [<ffffffff8104ed58>] kvm_unlock_kick+0xa8/0x100
[ 11.260228] RSP: 0018:ffff88023fc03c98 EFLAGS: 00010046
[ 11.260228] RAX: 0000000000000005 RBX: 0000000000000000 RCX: 0000000000000001
[ 11.260228] RDX: ffffffff81eaf408 RSI: 0000000000000000 RDI: 0000000000000000
[ 11.260228] RBP: ffff88023fc03cb8 R08: ffffffff81eaf400 R09: 00000000ffffffff
[ 11.260228] R10: ffff880037612cc0 R11: ffffea0002eb0a00 R12: ffff8800374a33c0
[ 11.260228] R13: 0000000000000020 R14: 0000000000000001 R15: 0000000000000286
[ 11.260228] FS: 00007f1e8b538740(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
[ 11.260228] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 11.260228] CR2: 00007f1e8ae09d50 CR3: 0000000001c0e000 CR4: 00000000000406f0
[ 11.260228] Stack:
[ 11.260228] 0000000000000286 0000000000000001 0000000000000001 00000000000000c3
[ 11.260228] ffff88023fc03cc8 ffffffff81717ed6 ffff88023fc03ce0 ffffffff8172641a
[ 11.260228] ffff8800374a33c0 ffff88023fc03d18 ffffffff810aaeb0 ffff88023295e000
[ 11.260228] Call Trace:
[ 11.260228] <IRQ>
[ 11.260228] [<ffffffff81717ed6>] __ticket_unlock_slowpath+0x24/0x34
[ 11.260228] [<ffffffff8172641a>] _raw_spin_unlock_irqrestore+0x3a/0x40
[ 11.260228] [<ffffffff810aaeb0>] __wake_up_sync_key+0x50/0x60
[ 11.260228] [<ffffffff8160ca5a>] sock_def_readable+0x3a/0x70
[ 11.260228] [<ffffffff816fda0a>] packet_rcv+0x2fa/0x430
[ 11.260228] [<ffffffff816228b0>] __netif_receive_skb_core+0x360/0x840
[ 11.260228] [<ffffffff81622da8>] __netif_receive_skb+0x18/0x60
[ 11.260228] [<ffffffff81622e13>] netif_receive_skb+0x23/0x90
[ 11.260228] [<ffffffff815288d4>] virtnet_poll+0x4d4/0x850
[ 11.260228] [<ffffffff81623192>] net_rx_action+0x152/0x250
[ 11.260228] [<ffffffff8106cbac>] __do_softirq+0xec/0x2c0
[ 11.260228] [<ffffffff8106d0f5>] irq_exit+0x105/0x110
[ 11.260228] [<ffffffff817312d6>] do_IRQ+0x56/0xc0
[ 11.260228] [<ffffffff81726a6d>] common_interrupt+0x6d/0x6d
[ 11.260228] <EOI>
[ 11.260228] [<ffffffff8104f596>] ? native_safe_halt+0x6/0x10
[ 11.260228] [<ffffffff8101c62f>] default_idle+0x1f/0xc0
[ 11.260228] [<ffffffff8101cef6>] arch_cpu_idle+0x26/0x30
[ 11.260228] [<ffffffff810bed95>] cpu_startup_entry+0xc5/0x290
[ 11.260228] [<ffffffff8170ca77>] rest_init+0x77/0x80
[ 11.260228] [<ffffffff81d35f6b>] start_kernel+0x433/0x43e
[ 11.260228] [<ffffffff81d35941>] ? repair_env_string+0x5c/0x5c
[ 11.260228] [<ffffffff81d35120>] ? early_idt_handlers+0x120/0x120
[ 11.260228] [<ffffffff81d355ee>] x86_64_start_reservations+0x2a/0x2c
[ 11.260228] [<ffffffff81d35733>] x86_64_start_kernel+0x143/0x152
[ 11.260228] Code: 66 44 39 e8 75 bd 0f b6 35 f6 06 e6 00 40 84 f6 75 2a 83 05 06 07 e6 00 01 48 c7 c0 6a b0 00 00 31 db 0f b7 0c 01 b8 05 00 00 00 <0f> 01 c1 0f 1f 44 00 00 5b 41 5c 41 5d 41 5e 5d c3 89 f0 31 c9
[ 11.260228] RIP [<ffffffff8104ed58>] kvm_unlock_kick+0xa8/0x100
[ 11.260228] RSP <ffff88023fc03c98>
[ 11.260228] ---[ end trace f1c26ff24745b331 ]---
[ 11.260228] Kernel panic - not syncing: Fatal exception in interrupt
[ 11.260228] Shutting down cpus with NMI |
|
| 2014-11-26 18:20:25 |
Chris J Arges |
linux (Ubuntu): assignee |
Chris J Arges (arges) |
|
|
| 2014-11-26 18:20:42 |
Chris J Arges |
description |
[Impact]
When using KVM on an AMD host with a kernel that has CONFIG_DEBUG_RODATA enabled, a guest with: multiple vCPUs, and exposing features to the guest such as tsc_adjust can cause a divide error on kvm_unlock_kick when booting the VM.
[Test Case]
1) Create a VM on an AMD host with appropriate features (Opteron 6xxx for example)
2) Edit virsh xml to have <cpu mode='host-passthrough'></cpu> and multiple vCPUs.
3) Boot VM with VGA console using virt-manager (I couldn't reproduce strictly monitoring via virsh console).
[Fix]
commit c1118b3602c2329671ad5ec8bdf8e374323d6343 upstream
--
Just upgraded OpenStack compute hosts in our public cloud (using qemu-kvm via libvirt) from Precise to Trusty (14.04.1), now on kernel 3.13.0-36-generic with qemu-kvm 2.0.0+dfsg-2ubuntu1.5.
Following the upgrade, whenever we try to start an smp/multicore Trusty guest (existing or new), we run into this panic [1] inside the guest just towards the end of boot. This happens consistently for smp guests using the Trusty kernel (i.e., it also affects earlier Ubuntus using the HWE kernel from Trusty but not their native versions). I didn't have any other distro images to hand with 3.13.x kernels, but none of the others I tested were affected (in the 3.2 - 3.16 kernel range).
There are scarce similar reports out there, but the one we did find pointed to a CPU feature as the trigger. We were running these hosts with libvirt cpu mode set to "host-passthrough" (so qemu starts with "-cpu host"), on AMD 6200 & 6300 Opteron hardware. Switching the guest domains to use cpu mode "host-model" instead works around the issue and is perfectly acceptable for most of our users.
We have various other Intel compute hosts and they don't seem to be affected.
(1)
[ 11.256924] divide error: 0000 [#1] SMP
[ 11.258133] Modules linked in: kvm_amd kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw lp parport psmouse floppy
[ 11.260228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-36-generic #63-Ubuntu
[ 11.260228] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Bochs 01/01/2011
[ 11.260228] task: ffffffff81c15480 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[ 11.260228] RIP: 0010:[<ffffffff8104ed58>] [<ffffffff8104ed58>] kvm_unlock_kick+0xa8/0x100
[ 11.260228] RSP: 0018:ffff88023fc03c98 EFLAGS: 00010046
[ 11.260228] RAX: 0000000000000005 RBX: 0000000000000000 RCX: 0000000000000001
[ 11.260228] RDX: ffffffff81eaf408 RSI: 0000000000000000 RDI: 0000000000000000
[ 11.260228] RBP: ffff88023fc03cb8 R08: ffffffff81eaf400 R09: 00000000ffffffff
[ 11.260228] R10: ffff880037612cc0 R11: ffffea0002eb0a00 R12: ffff8800374a33c0
[ 11.260228] R13: 0000000000000020 R14: 0000000000000001 R15: 0000000000000286
[ 11.260228] FS: 00007f1e8b538740(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
[ 11.260228] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 11.260228] CR2: 00007f1e8ae09d50 CR3: 0000000001c0e000 CR4: 00000000000406f0
[ 11.260228] Stack:
[ 11.260228] 0000000000000286 0000000000000001 0000000000000001 00000000000000c3
[ 11.260228] ffff88023fc03cc8 ffffffff81717ed6 ffff88023fc03ce0 ffffffff8172641a
[ 11.260228] ffff8800374a33c0 ffff88023fc03d18 ffffffff810aaeb0 ffff88023295e000
[ 11.260228] Call Trace:
[ 11.260228] <IRQ>
[ 11.260228] [<ffffffff81717ed6>] __ticket_unlock_slowpath+0x24/0x34
[ 11.260228] [<ffffffff8172641a>] _raw_spin_unlock_irqrestore+0x3a/0x40
[ 11.260228] [<ffffffff810aaeb0>] __wake_up_sync_key+0x50/0x60
[ 11.260228] [<ffffffff8160ca5a>] sock_def_readable+0x3a/0x70
[ 11.260228] [<ffffffff816fda0a>] packet_rcv+0x2fa/0x430
[ 11.260228] [<ffffffff816228b0>] __netif_receive_skb_core+0x360/0x840
[ 11.260228] [<ffffffff81622da8>] __netif_receive_skb+0x18/0x60
[ 11.260228] [<ffffffff81622e13>] netif_receive_skb+0x23/0x90
[ 11.260228] [<ffffffff815288d4>] virtnet_poll+0x4d4/0x850
[ 11.260228] [<ffffffff81623192>] net_rx_action+0x152/0x250
[ 11.260228] [<ffffffff8106cbac>] __do_softirq+0xec/0x2c0
[ 11.260228] [<ffffffff8106d0f5>] irq_exit+0x105/0x110
[ 11.260228] [<ffffffff817312d6>] do_IRQ+0x56/0xc0
[ 11.260228] [<ffffffff81726a6d>] common_interrupt+0x6d/0x6d
[ 11.260228] <EOI>
[ 11.260228] [<ffffffff8104f596>] ? native_safe_halt+0x6/0x10
[ 11.260228] [<ffffffff8101c62f>] default_idle+0x1f/0xc0
[ 11.260228] [<ffffffff8101cef6>] arch_cpu_idle+0x26/0x30
[ 11.260228] [<ffffffff810bed95>] cpu_startup_entry+0xc5/0x290
[ 11.260228] [<ffffffff8170ca77>] rest_init+0x77/0x80
[ 11.260228] [<ffffffff81d35f6b>] start_kernel+0x433/0x43e
[ 11.260228] [<ffffffff81d35941>] ? repair_env_string+0x5c/0x5c
[ 11.260228] [<ffffffff81d35120>] ? early_idt_handlers+0x120/0x120
[ 11.260228] [<ffffffff81d355ee>] x86_64_start_reservations+0x2a/0x2c
[ 11.260228] [<ffffffff81d35733>] x86_64_start_kernel+0x143/0x152
[ 11.260228] Code: 66 44 39 e8 75 bd 0f b6 35 f6 06 e6 00 40 84 f6 75 2a 83 05 06 07 e6 00 01 48 c7 c0 6a b0 00 00 31 db 0f b7 0c 01 b8 05 00 00 00 <0f> 01 c1 0f 1f 44 00 00 5b 41 5c 41 5d 41 5e 5d c3 89 f0 31 c9
[ 11.260228] RIP [<ffffffff8104ed58>] kvm_unlock_kick+0xa8/0x100
[ 11.260228] RSP <ffff88023fc03c98>
[ 11.260228] ---[ end trace f1c26ff24745b331 ]---
[ 11.260228] Kernel panic - not syncing: Fatal exception in interrupt
[ 11.260228] Shutting down cpus with NMI |
[Impact]
When using KVM on an AMD host with a kernel that has CONFIG_DEBUG_RODATA enabled, a guest with: multiple vCPUs, and exposing features to the guest such as tsc_adjust can cause a divide error on kvm_unlock_kick when booting the VM.
This impacts kernels 3.12+.
[Test Case]
1) Create a VM on an AMD host with appropriate features (Opteron 6xxx for example)
2) Edit virsh xml to have <cpu mode='host-passthrough'></cpu> and multiple vCPUs.
3) Boot VM with VGA console using virt-manager (I couldn't reproduce strictly monitoring via virsh console).
[Fix]
commit c1118b3602c2329671ad5ec8bdf8e374323d6343 upstream
--
Just upgraded OpenStack compute hosts in our public cloud (using qemu-kvm via libvirt) from Precise to Trusty (14.04.1), now on kernel 3.13.0-36-generic with qemu-kvm 2.0.0+dfsg-2ubuntu1.5.
Following the upgrade, whenever we try to start an smp/multicore Trusty guest (existing or new), we run into this panic [1] inside the guest just towards the end of boot. This happens consistently for smp guests using the Trusty kernel (i.e., it also affects earlier Ubuntus using the HWE kernel from Trusty but not their native versions). I didn't have any other distro images to hand with 3.13.x kernels, but none of the others I tested were affected (in the 3.2 - 3.16 kernel range).
There are scarce similar reports out there, but the one we did find pointed to a CPU feature as the trigger. We were running these hosts with libvirt cpu mode set to "host-passthrough" (so qemu starts with "-cpu host"), on AMD 6200 & 6300 Opteron hardware. Switching the guest domains to use cpu mode "host-model" instead works around the issue and is perfectly acceptable for most of our users.
We have various other Intel compute hosts and they don't seem to be affected.
(1)
[ 11.256924] divide error: 0000 [#1] SMP
[ 11.258133] Modules linked in: kvm_amd kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw lp parport psmouse floppy
[ 11.260228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-36-generic #63-Ubuntu
[ 11.260228] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Bochs 01/01/2011
[ 11.260228] task: ffffffff81c15480 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[ 11.260228] RIP: 0010:[<ffffffff8104ed58>] [<ffffffff8104ed58>] kvm_unlock_kick+0xa8/0x100
[ 11.260228] RSP: 0018:ffff88023fc03c98 EFLAGS: 00010046
[ 11.260228] RAX: 0000000000000005 RBX: 0000000000000000 RCX: 0000000000000001
[ 11.260228] RDX: ffffffff81eaf408 RSI: 0000000000000000 RDI: 0000000000000000
[ 11.260228] RBP: ffff88023fc03cb8 R08: ffffffff81eaf400 R09: 00000000ffffffff
[ 11.260228] R10: ffff880037612cc0 R11: ffffea0002eb0a00 R12: ffff8800374a33c0
[ 11.260228] R13: 0000000000000020 R14: 0000000000000001 R15: 0000000000000286
[ 11.260228] FS: 00007f1e8b538740(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
[ 11.260228] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 11.260228] CR2: 00007f1e8ae09d50 CR3: 0000000001c0e000 CR4: 00000000000406f0
[ 11.260228] Stack:
[ 11.260228] 0000000000000286 0000000000000001 0000000000000001 00000000000000c3
[ 11.260228] ffff88023fc03cc8 ffffffff81717ed6 ffff88023fc03ce0 ffffffff8172641a
[ 11.260228] ffff8800374a33c0 ffff88023fc03d18 ffffffff810aaeb0 ffff88023295e000
[ 11.260228] Call Trace:
[ 11.260228] <IRQ>
[ 11.260228] [<ffffffff81717ed6>] __ticket_unlock_slowpath+0x24/0x34
[ 11.260228] [<ffffffff8172641a>] _raw_spin_unlock_irqrestore+0x3a/0x40
[ 11.260228] [<ffffffff810aaeb0>] __wake_up_sync_key+0x50/0x60
[ 11.260228] [<ffffffff8160ca5a>] sock_def_readable+0x3a/0x70
[ 11.260228] [<ffffffff816fda0a>] packet_rcv+0x2fa/0x430
[ 11.260228] [<ffffffff816228b0>] __netif_receive_skb_core+0x360/0x840
[ 11.260228] [<ffffffff81622da8>] __netif_receive_skb+0x18/0x60
[ 11.260228] [<ffffffff81622e13>] netif_receive_skb+0x23/0x90
[ 11.260228] [<ffffffff815288d4>] virtnet_poll+0x4d4/0x850
[ 11.260228] [<ffffffff81623192>] net_rx_action+0x152/0x250
[ 11.260228] [<ffffffff8106cbac>] __do_softirq+0xec/0x2c0
[ 11.260228] [<ffffffff8106d0f5>] irq_exit+0x105/0x110
[ 11.260228] [<ffffffff817312d6>] do_IRQ+0x56/0xc0
[ 11.260228] [<ffffffff81726a6d>] common_interrupt+0x6d/0x6d
[ 11.260228] <EOI>
[ 11.260228] [<ffffffff8104f596>] ? native_safe_halt+0x6/0x10
[ 11.260228] [<ffffffff8101c62f>] default_idle+0x1f/0xc0
[ 11.260228] [<ffffffff8101cef6>] arch_cpu_idle+0x26/0x30
[ 11.260228] [<ffffffff810bed95>] cpu_startup_entry+0xc5/0x290
[ 11.260228] [<ffffffff8170ca77>] rest_init+0x77/0x80
[ 11.260228] [<ffffffff81d35f6b>] start_kernel+0x433/0x43e
[ 11.260228] [<ffffffff81d35941>] ? repair_env_string+0x5c/0x5c
[ 11.260228] [<ffffffff81d35120>] ? early_idt_handlers+0x120/0x120
[ 11.260228] [<ffffffff81d355ee>] x86_64_start_reservations+0x2a/0x2c
[ 11.260228] [<ffffffff81d35733>] x86_64_start_kernel+0x143/0x152
[ 11.260228] Code: 66 44 39 e8 75 bd 0f b6 35 f6 06 e6 00 40 84 f6 75 2a 83 05 06 07 e6 00 01 48 c7 c0 6a b0 00 00 31 db 0f b7 0c 01 b8 05 00 00 00 <0f> 01 c1 0f 1f 44 00 00 5b 41 5c 41 5d 41 5e 5d c3 89 f0 31 c9
[ 11.260228] RIP [<ffffffff8104ed58>] kvm_unlock_kick+0xa8/0x100
[ 11.260228] RSP <ffff88023fc03c98>
[ 11.260228] ---[ end trace f1c26ff24745b331 ]---
[ 11.260228] Kernel panic - not syncing: Fatal exception in interrupt
[ 11.260228] Shutting down cpus with NMI |
|
| 2014-11-26 18:21:46 |
Chris J Arges |
linux (Ubuntu Trusty): status |
New |
In Progress |
|
| 2014-11-26 18:21:48 |
Chris J Arges |
linux (Ubuntu Utopic): status |
New |
In Progress |
|
| 2014-11-26 18:21:52 |
Chris J Arges |
linux (Ubuntu Trusty): importance |
Undecided |
Medium |
|
| 2014-11-26 18:21:54 |
Chris J Arges |
linux (Ubuntu Utopic): importance |
Undecided |
Medium |
|
| 2014-11-28 08:05:29 |
Andy Whitcroft |
linux (Ubuntu Utopic): status |
In Progress |
Fix Committed |
|
| 2014-11-28 08:05:35 |
Andy Whitcroft |
linux (Ubuntu Trusty): status |
In Progress |
Fix Committed |
|
| 2014-12-11 11:07:44 |
Daniele Viganò |
attachment added |
|
Full error backtrace https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1379340/+attachment/4278654/+files/NMI_backtrace.txt |
|
| 2014-12-12 16:18:52 |
Ahmon Dancy |
bug |
|
|
added subscriber Ahmon Dancy |
| 2014-12-16 21:29:16 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/trusty-proposed/linux-keystone |
|
| 2014-12-17 19:10:59 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/precise-proposed/linux-lts-trusty |
|
| 2014-12-17 20:21:59 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/trusty-proposed/linux-lts-utopic |
|
| 2014-12-19 17:39:07 |
Brad Figg |
tags |
amd fixed-upstream trusty |
amd fixed-upstream trusty verification-needed-trusty |
|
| 2014-12-19 17:40:03 |
Brad Figg |
tags |
amd fixed-upstream trusty verification-needed-trusty |
amd fixed-upstream trusty verification-needed-trusty verification-needed-utopic |
|
| 2014-12-19 18:13:40 |
Daniele Viganò |
tags |
amd fixed-upstream trusty verification-needed-trusty verification-needed-utopic |
amd fixed-upstream trusty verification-done-precise verification-needed-trusty verification-needed-utopic |
|
| 2014-12-22 08:21:20 |
Daniele Viganò |
tags |
amd fixed-upstream trusty verification-done-precise verification-needed-trusty verification-needed-utopic |
amd fixed-upstream trusty verification-done-precise verification-done-trusty verification-needed-utopic |
|
| 2015-01-08 17:11:48 |
Chris J Arges |
tags |
amd fixed-upstream trusty verification-done-precise verification-done-trusty verification-needed-utopic |
amd fixed-upstream trusty verification-done-precise verification-done-trusty verification-done-utopic |
|
| 2015-01-12 15:41:50 |
Launchpad Janitor |
linux (Ubuntu Utopic): status |
Fix Committed |
Fix Released |
|
| 2015-01-12 15:44:28 |
Launchpad Janitor |
linux (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2015-12-27 17:04:33 |
Dmitry G. |
bug |
|
|
added subscriber Dmitry G. |
