Starting a KVM guest in a guest (nested VM) crash the kernel

[György Szombathelyi]

linux-image-4.15.0.22 (20, and probably earlier, too) panics when starting a guest VM in a nested manner.

compute1 login: [ 4847.733004] general protection fault: 0000 [#1] SMP PTI
[ 4847.736595] Modules linked in: vxlan ip6_udp_tunnel udp_tunnel ebt_arp ebt_among ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6 xt_CT xt_mac xt_comment xt_physdev xt_set ip_set_hash_net ip_set nfnetlink vhost_net vhost tap xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp kvm_intel ip6table_filter ip6_tables iptable_filter ebtable_filter ebtable_nat ebtables iptable_raw br_netfilter bridge stp llc nls_iso8859_1 kvm irqbypass joydev input_leds serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
[ 4847.756237] async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd psmouse virtio_blk virtio_net floppy [last unloaded: kvm_intel]
[ 4847.762893] CPU: 1 PID: 13789 Comm: CPU 0/KVM Not tainted 4.15.0-22-generic #24-Ubuntu
[ 4847.765154] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.2-1ubuntu1~cloud0 04/01/2014
[ 4847.767764] RIP: 0010:native_write_cr4+0x4/0x10
[ 4847.769085] RSP: 0018:ffff9daf4119bb70 EFLAGS: 00010006
[ 4847.770629] RAX: 00000000000626e0 RBX: 0000000000000046 RCX: ffff8edcbfc80000
[ 4847.772771] RDX: ffff8edcbfc94020 RSI: ffff8edcbfca5040 RDI: 00000000000606e0
[ 4847.774965] RBP: ffff9daf4119bb70 R08: fffffffffffffff8 R09: 000000000000000d
[ 4847.777175] R10: ffffc078c76e2000 R11: 0000000000000cd0 R12: 0000000000025040
[ 4847.779213] R13: 0000000000000000 R14: ffff8edc438409a0 R15: ffff8edc43855278
[ 4847.781230] FS: 00007f9a09a02700(0000) GS:ffff8edcbfc80000(0000) knlGS:0000000000000000
[ 4847.783548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4847.785233] CR2: 00007f99fb80cd50 CR3: 0000000152e0a002 CR4: 00000000000626e0
[ 4847.787225] Call Trace:
[ 4847.788062] hardware_disable+0xaa/0xc0 [kvm_intel]
[ 4847.789460] ? vcpu_stat_get_per_vm_open+0x20/0x20 [kvm]
[ 4847.791025] kvm_arch_hardware_disable+0x19/0x40 [kvm]
[ 4847.792664] hardware_disable_nolock+0x2b/0x30 [kvm]
[ 4847.794260] on_each_cpu+0x46/0x60
[ 4847.795302] hardware_disable_all_nolock+0x35/0x40 [kvm]
[ 4847.796896] hardware_disable_all+0x1a/0x30 [kvm]
[ 4847.798267] kvm_put_kvm+0x1f8/0x260 [kvm]
[ 4847.799470] kvm_vm_release+0x21/0x30 [kvm]
[ 4847.800758] __fput+0xea/0x220
[ 4847.801681] ____fput+0xe/0x10
[ 4847.802602] task_work_run+0x9d/0xc0
[ 4847.803692] do_exit+0x2ec/0xb40
[ 4847.804676] do_group_exit+0x43/0xb0
[ 4847.805719] get_signal+0x27b/0x590
[ 4847.806742] do_signal+0x37/0x730
[ 4847.807774] ? do_futex+0x325/0x500
[ 4847.808803] ? SyS_futex+0x13b/0x180
[ 4847.809852] ? restore_altstack+0x51/0x70
[ 4847.811017] exit_to_usermode_loop+0x73/0xd0
[ 4847.812299] do_syscall_64+0x115/0x130
[ 4847.813401] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ 4847.814808] RIP: 0033:0x7f9a142bc10d
[ 4847.815914] RSP: 002b:00007f9a09a01798 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[ 4847.818100] RAX: fffffffffffffe00 RBX: 00005583a8961060 RCX: 00007f9a142bc10d
[ 4847.820065] RDX: 0000000000000002 RSI: 0000000000000080 RDI: 00005583a6925e00
[ 4847.822029] RBP: 00005583a8961084 R08: 00005583a6925e00 R09: 0000000000000000
[ 4847.824002] R10: 0000000000000000 R11: 0000000000000246 R12: 00005583a896108c
[ 4847.825949] R13: 0000000000000000 R14: 00005583a6925e00 R15: 000000000000000f
[ 4847.827922] Code: 0f 1f 80 00 00 00 00 55 48 89 e5 0f 20 d8 5d c3 0f 1f 80 00 00 00 00 55 48 89 e5 0f 22 df 5d c3 0f 1f 80 00 00 00 00 55 48 89 e5 <0f> 22 e7 5d c3 0f 1f 80 00 00 00 00 55 48 89 e5 44 0f 20 c0 5d
[ 4847.832993] RIP: native_write_cr4+0x4/0x10 RSP: ffff9daf4119bb70
[ 4847.834658] ---[ end trace d54d1c7bca906f72 ]---
[ 4847.838001] Fixing recursive fault but reboot is needed!
[ 4850.411516] general protection fault: 0000 [#2] SMP PTI
[ 4850.414947] Modules linked in: vxlan ip6_udp_tunnel udp_tunnel ebt_arp ebt_among ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6 xt_CT xt_mac xt_comment xt_physdev xt_set ip_set_hash_net ip_set nfnetlink vhost_net vhost tap xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp kvm_intel ip6table_filter ip6_tables iptable_filter ebtable_filter ebtable_nat ebtables iptable_raw br_netfilter bridge stp llc nls_iso8859_1 kvm irqbypass joydev input_leds serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
[ 4850.434880] async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd psmouse virtio_blk virtio_net floppy [last unloaded: kvm_intel]
[ 4850.441772] CPU: 1 PID: 14003 Comm: sudo Tainted: G D 4.15.0-22-generic #24-Ubuntu
[ 4850.444372] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.2-1ubuntu1~cloud0 04/01/2014
[ 4850.446997] RIP: 0010:native_flush_tlb_global+0x88/0xa0
[ 4850.448603] RSP: 0000:ffff8edcbfc83f50 EFLAGS: 00010006
[ 4850.450192] RAX: 00000000000606e0 RBX: ffff8edcbfca8880 RCX: ffff8edcbfca8880
[ 4850.452274] RDX: 0000000000060660 RSI: 0000000000000001 RDI: 0000000000000046
[ 4850.454310] RBP: ffff8edcbfc83f80 R08: 0000000000000000 R09: 0000000000000000
[ 4850.456461] R10: ffff8edcbfc83f98 R11: 0000000000000000 R12: 0000000000000000
[ 4850.458462] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
[ 4850.460539] FS: 00007fe811b59c80(0000) GS:ffff8edcbfc80000(0000) knlGS:0000000000000000
[ 4850.462835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4850.464618] CR2: 00007fe810e198a0 CR3: 00000001cd334001 CR4: 00000000000626e0
[ 4850.466610] Call Trace:
[ 4850.467401] <IRQ>
[ 4850.468183] ? __do_softirq+0x128/0x2b2
[ 4850.469357] do_flush_tlb_all+0x23/0x30
[ 4850.470510] ? do_flush_tlb_all+0x23/0x30
[ 4850.471785] flush_smp_call_function_queue+0x4c/0xf0
[ 4850.473336] generic_smp_call_function_single_interrupt+0x13/0x30
[ 4850.475067] smp_call_function_interrupt+0x36/0xd0
[ 4850.476565] call_function_interrupt+0x84/0x90
[ 4850.477865] </IRQ>
[ 4850.478573] RIP: 0033:0x7fe81195f398
[ 4850.479727] RSP: 002b:00007fff968f20a8 EFLAGS: 00000287 ORIG_RAX: ffffffffffffff03
[ 4850.481969] RAX: 000000000000000b RBX: 00007fe811b5d500 RCX: 0000000000000010
[ 4850.484034] RDX: 0000000000000000 RSI: 00007fe811b5d4d0 RDI: 00007fe81171ac8b
[ 4850.486032] RBP: 00007fe81171ac8b R08: 00007fe8114f1a18 R09: 00007fe811b6b428
[ 4850.488095] R10: 000000000000000b R11: 0000000000000000 R12: 00007fe811b5c130
[ 4850.490070] R13: 00007fe811b5c350 R14: 0000000000000000 R15: 0000000000000000
[ 4850.492110] Code: 48 83 c4 28 41 5a 5d 49 8d 62 f8 c3 9c 58 66 66 90 66 90 48 89 c7 fa 66 66 90 66 66 90 65 48 8b 05 4e 72 7b 55 48 89 c2 80 f2 80 <0f> 22 e2 0f 22 e0 57 9d 66 66 90 66 90 eb b7 e8 54 00 02 00 0f
[ 4850.497339] RIP: native_flush_tlb_global+0x88/0xa0 RSP: ffff8edcbfc83f50
[ 4850.499261] ---[ end trace d54d1c7bca906f73 ]---
[ 4850.500701] Kernel panic - not syncing: Fatal exception in interrupt
[ 4850.502615] Kernel Offset: 0x29800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 4850.505816] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 May 24 15:00 seq
 crw-rw---- 1 root audio 116, 33 May 24 15:00 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 18.04
Ec2AMI: ami-0000011e
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: compute1-8
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-22-generic root=UUID=3ca4ee87-6fd6-4b19-9b9d-14607e5674ae ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 4.15.0-22.24-generic 4.15.17
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-22-generic N/A
 linux-backports-modules-4.15.0-22-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic ec2-images
Uname: Linux 4.15.0-22-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1~cloud0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-bionic
dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1~cloud0:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr17.0.3:cvnQEMU:ct1:cvrpc-i440fx-bionic:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.version: 17.0.3
dmi.sys.vendor: OpenStack Foundation

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1773184

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[György Szombathelyi (gyurco)]

apport information

[György Szombathelyi (gyurco)]

apport information

[György Szombathelyi (gyurco)]

apport information

[György Szombathelyi (gyurco)]

apport information

[György Szombathelyi (gyurco)]

apport information

[György Szombathelyi (gyurco)]

apport information

[György Szombathelyi (gyurco)]

apport information

[György Szombathelyi (gyurco)]

apport information

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.17 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.17-rc6

[György Szombathelyi (gyurco)]

The problem is there with 4.17-rc6

ompute2 login: [ 5290.132368] general protection fault: 0000 [#1] SMP PTI
[ 5290.133501] Modules linked in: vxlan ip6_udp_tunnel udp_tunnel ebt_arp ebt_among ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6 xt_CT xt_mac xt_comment xt_physdev vhost_net vhost tap xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp kvm_intel ip6table_filter ip6_tables iptable_filter ebtable_filter ebtable_nat ebtables iptable_raw br_netfilter bridge stp llc nls_iso8859_1 kvm irqbypass joydev input_leds mac_hid serio_raw qemu_fw_cfg sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
[ 5290.153839] libcrc32c raid1 raid0 multipath linear crct10dif_pclmul hid_generic crc32_pclmul ghash_clmulni_intel pcbc aesni_intel usbhid aes_x86_64 crypto_simd cryptd glue_helper cirrus ttm hid drm_kms_helper psmouse syscopyarea sysfillrect sysimgblt fb_sys_fops virtio_blk virtio_net drm i2c_piix4 floppy pata_acpi [last unloaded: kvm_intel]
[ 5290.162975] CPU: 0 PID: 13660 Comm: CPU 0/KVM Not tainted 4.17.0-041700rc6-generic #201805202330
[ 5290.165690] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.2-1ubuntu1~cloud0 04/01/2014
[ 5290.168450] RIP: 0010:native_write_cr4+0x4/0x10
[ 5290.169847] RSP: 0018:ffffc0028128bb70 EFLAGS: 00010006
[ 5290.171383] RAX: 00000000000626f0 RBX: 0000000000000046 RCX: ffff9b6b3fc00000
[ 5290.173624] RDX: ffff9b6b3fc14020 RSI: ffff9b6b3fc25040 RDI: 00000000000606f0
[ 5290.175694] RBP: ffffc0028128bb70 R08: 0000000000027040 R09: ffffffff82bb9eb9
[ 5290.177778] R10: fffffa1988bb2a00 R11: 0000000000000048 R12: 0000000000025040
[ 5290.179295] R13: 0000000000000000 R14: ffff9b6b31ec09a0 R15: ffff9b6b31ed5270
[ 5290.180780] FS: 00007fc542763700(0000) GS:ffff9b6b3fc00000(0000) knlGS:0000000000000000
[ 5290.183096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5290.184583] CR2: 00007fc53380cd50 CR3: 0000000193a0a003 CR4: 00000000000626f0
[ 5290.186146] Call Trace:
[ 5290.186753] hardware_disable+0xaa/0xc0 [kvm_intel]
[ 5290.188150] ? vcpu_stat_get_per_vm_open+0x20/0x20 [kvm]
[ 5290.190099] kvm_arch_hardware_disable+0x19/0x40 [kvm]
[ 5290.191639] hardware_disable_nolock+0x2b/0x30 [kvm]
[ 5290.193253] on_each_cpu+0x46/0x60
[ 5290.194075] hardware_disable_all_nolock+0x35/0x40 [kvm]
[ 5290.195800] hardware_disable_all+0x1a/0x30 [kvm]
[ 5290.197091] kvm_put_kvm+0x203/0x260 [kvm]
[ 5290.198370] kvm_vm_release+0x21/0x30 [kvm]
[ 5290.199481] __fput+0xea/0x220
[ 5290.200445] ____fput+0xe/0x10
[ 5290.201442] task_work_run+0x9d/0xc0
[ 5290.202547] do_exit+0x2ec/0xb50
[ 5290.203640] do_group_exit+0x43/0xb0
[ 5290.204677] get_signal+0x27f/0x5c0
[ 5290.205368] do_signal+0x37/0x730
[ 5290.206005] ? do_futex+0x325/0x500
[ 5290.206774] exit_to_usermode_loop+0x8a/0xd0
[ 5290.207664] do_syscall_64+0xfb/0x120
[ 5290.208345] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5290.209667] RIP: 003...

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a kernel version where you were not having this particular problem? This will help determine if the problem you are seeing is the result of a regression, and when this regression was introduced. If this is a regression, we can perform a kernel bisect to identify the commit that introduced the problem.

[György Szombathelyi (gyurco)]

It started right after I tried to run our OpenStack CI (so the nested operation) on Bionic (with OpenStack Queens). Previously I used Xenial (with kernel 4.4.0), that still works.
The host kernel is still from Xenial.

So:
Host with KVM->Guest with KVM->Nested Guest

Xenial (4.4) -> Bionic (4.15)-> any other - crash
Xenial (4.4) -> Xenial (4.4) -> any other - works

[György Szombathelyi (gyurco)]

In the above, the middle one that crashes (so the 1st level of guests).

[sean mooney (sean-k-mooney)]

just adding some more info.
i also deploy openstack rocky on a ubuntu 18.04 host

Linux cloud-5 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

with ubuntu 18.04 l1 guest running
Linux numa-migration-1 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

and when i lauch a cirros l2 guest i get very similar kvm_intel call traces.

if i upgrade the l1 guest kernel to the latest linux-virtual-hwe-18.04-edge kernel
which is currently

Linux numa-migration-2 4.18.0-15-generic #16~18.04.1-Ubuntu SMP Thu Feb 7 14:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

i still get the same crashes when i launch the l2 cirros image

i was originally wondering if this was some how related to the gust cpu model

so i tried seting the l1 and l2 guest cpu_mode to host-passthough
to maximies the feature that were availabel tot he guest

i then also tried to limit the l2 guest to the kvm64 cpu-model so that the l2 guest
used the minium set of kvm feature but this did not seam to have an benifical
effect.

the host is useign

Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz cpus which i have used in the past with
ubuntu 16.04 and nested virt without issue so this seams to be related to the 18.04 kenel in some way.

the fact that i am seeing the same issue on a ubuntu 18.04 host with an 18.04 l1 guest

and that gyurco is only seeing this with an 18.04 l1 guest makes me think that this is likely an issue with kvm in the l1 guests kernel not the hosts kernel.

im going to try and confrim that with a centos 7 l1 guest tommorow but since i would like to be
able to do python 3.6/3.7 testing in my local settup i would prefer to use a Ubuntu 18.04 l1 guest in the long run so i hope this can be resolved.

[György Szombathelyi (gyurco)]

Just to note: since one of the kernel upgrades, this issue is not present anymore. I'm not sure which version fixed it.