nested kvm doesn't work

[Rakesh Ginjupalli]

Create VM on Azure using Canonical UbuntuServer 14.04.5-LTS latest using size Standard E4s v3

Enable private-ppa canonical-kernel-esm

Install kernel linux-azure, reboot VM, kernel version 4.15.0-1084-azure

Install qemu-kvm, download https://eosgnestedstorage.blob.core.windows.net/images/nested-ubuntu-4.15.0-23-generic.qcow2

Launch nested VM using command qemu-system-x86_64 -smp 2 -m 2048 -hda /mnt/resource/nested.qcow2 -display none -device e1000,netdev=user.0 -netdev user,id=user.0,hostfwd=tcp::60022-:22 -enable-kvm -daemonize

Expected ssh -p 60022 nesteduser@localhost will ask password to login

We can't connect to it, see below call trace in dmesg
           qemu-kvm version doesn't change after kernel upgraded.
           2.0.0+dfsg-2ubuntu1.46
[ 53.420589] hv_balloon: Max. dynamic memory size: 32768 MB
[ 60.928772] L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.
[ 60.940895] kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable
[ 60.967454] BUG: unable to handle kernel paging request at 00000000000056b8
[ 60.971140] IP: vmx_vcpu_run+0x3ed/0xbc0 [kvm_intel]
[ 60.971140] PGD 0 P4D 0
[ 60.971140] Oops: 0000 [#1] SMP PTI
[ 60.971140] Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner iptable_security xt_conntrack nf_conntrack libcrc32c ip_tables x_tables udf crc_itu_t dm_crypt joydev hid_generic kvm_intel kvm irqbypass hid_hyperv hid hyperv_keyboard hv_balloon serio_raw crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd hyperv_fb cfbfillrect pata_acpi cfbimgblt cfbcopyarea hv_netvsc hv_utils
[ 60.971140] CPU: 2 PID: 1903 Comm: qemu-system-x86 Not tainted 4.15.0-1084-azure #94~14.04.1-Ubuntu
[ 60.971140] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007 06/02/2017
[ 60.971140] RIP: 0010:vmx_vcpu_run+0x3ed/0xbc0 [kvm_intel]
[ 60.971140] RSP: 0018:ffffa44847a53cc0 EFLAGS: 00010002
[ 60.971140] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 61.020070] RDX: 0000000000006c14 RSI: 0000000000000000 RDI: 0000000000000000
[ 61.020070] RBP: ffffa44847a53d10 R08: 0000000000000000 R09: 0000000000000000
[ 61.020070] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 61.020070] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 61.020070] FS: 00007f753bba4700(0000) GS:ffff8fefdd680000(0000) knlGS:0000000000000000
[ 61.020070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 61.020070] CR2: 00000000000056b8 CR3: 0000000888c42005 CR4: 00000000003726e0
[ 61.020070] Call Trace:
[ 61.020070] vcpu_enter_guest+0x909/0x12a0 [kvm]
[ 61.020070] kvm_arch_vcpu_ioctl_run+0x2d8/0x3f0 [kvm]
[ 61.020070] kvm_vcpu_ioctl+0x2a8/0x630 [kvm]
[ 61.020070] ? dequeue_signal+0x3e/0x160
[ 61.020070] ? do_sigtimedwait+0xbd/0x210
[ 61.020070] ? copy_siginfo_to_user+0x12e/0x2d0
[ 61.020070] do_vfs_ioctl+0x9b/0x5f0
[ 61.020070] ? SyS_futex+0x71/0x150
[ 61.020070] SyS_ioctl+0x79/0x90
[ 61.020070] ? _copy_to_user+0x26/0x40
[ 61.020070] do_syscall_64+0x80/0x1e0
[ 61.020070] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ 61.020070] RIP: 0033:0x7f7545f34ea7
[ 61.020070] RSP: 002b:00007f753bba3bd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 61.020070] RAX: ffffffffffffffda RBX: 00007f754bbed000 RCX: 00007f7545f34ea7
[ 61.020070] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000000c
[ 61.020070] RBP: 000055c68fd7f160 R08: 0000000000000000 R09: 0000000000000002
[ 61.020070] R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
[ 61.020070] R13: 0000000000000000 R14: 00007f753bba49c0 R15: 000055c68ee809a0
[ 61.020070] Code: 31 ed 45 31 f6 45 31 ff 0f 20 d0 48 89 81 e0 02 00 00 31 c0 31 db 31 c9 31 d2 31 f6 31 ff 31 ed 5d 5a f6 05 0c c0 01 00 10 74 1b <48> 8b 81 b8 56 00 00 48 8b 40 40 48 8b 80 08 08 00 00 f6 c4 01
[ 61.020070] RIP: vmx_vcpu_run+0x3ed/0xbc0 [kvm_intel] RSP: ffffa44847a53cc0
[ 61.020070] CR2: 00000000000056b8
[ 61.020070] ---[ end trace ab52be3a09d557f5 ]---

[Marcelo Cerri (mhcerri)]

Hi, Do you know if the same issue also occurs on Xenial?

[Rakesh Ginjupalli (linuxelf001)]

It doesn't occur on Xenial.

[Marcelo Cerri (mhcerri)]

I tried to reproduce the issue with the latest 4.15.0-1089 kernel and the virtual machine started without issues.

This bug might be the same as the one that was reported in bug 1881072, which was fixed in -1089.

Do you mind validating this issue again now with the 4.15.0-1089?

[lilideng (lilideng)]

Hi Marcelo, this issue is fixed in kernel version 4.15.0-1091-azure from private-ppa canonical-kernel-esm.

[lilideng (lilideng)]

The issue still exist on 4.15.0-1092-azure, probably I just verify the basic nested vm scenario against 4.15.0-1091-azure, will double check.

[Marcelo Cerri (mhcerri)]

Hi, Lili. I think this new problem is caused by the older qemu version that we have in trusty. Does it work without the -cpu host argument?

[lilideng (lilideng)]

Hi Marcelo, it works without -cpu host