Nested virt fails with 'failed to set MSR' causing QEMU to abort

[Matthew Ruffell]

[Impact]

When using an instance on Azure Cloud, nested kvm virtualisation fails with the following error message:

qemu-system-x86_64: error: failed to set MSR 0x48b to 0x11582e00000000
qemu-system-x86_64: /build/qemu-FC5BvZ/qemu-4.2/target/i386/kvm.c:2680: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

It appears that some systems can expose particular features via CPUID, but can lack the corresponding VMX control for that feature. When this happens, such as with MSR 0x48b on Azure Cloud, attempts to enable the feature fail and QEMU aborts.

[Testcase]

Create an instance on Azure with the latest Focal image, install QEMU, and prepare to launch a VM:

$ sudo -s
# apt install qemu-kvm bridge-utils
# BR_NAME="br0"
# BR_ADDR="192.168.1.10"
# ip link add "$BR_NAME" type bridge
# ip addr add "$BR_ADDR"/24 dev "$BR_NAME"
# ip link set "$BR_NAME" up
# tap_name="tap1"
# br_name=$BR_NAME
# ip tuntap add $tap_name mode tap user $(whoami) multi_queue
# ip link set $tap_name up
# ip link set $tap_name master $br_name

Download a guest VM:

# wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img

Launch the VM:
# /usr/bin/qemu-system-x86_64 -cpu host -smp 2 -m 2048 -hda focal-server-cloudimg-amd64-disk-kvm.img -device virtio-net-pci,netdev=net0,mac=52:54:00:f5:be:83 -netdev user,id=net0,hostfwd=tcp::60022-:22 -device virtio-net-pci,netdev=net1,mac=52:54:00:c6:29:dc,mq=on,vectors=10 -netdev tap,id=net1,ifname=tap1,script=no,vhost=on,queues=4 -display none -enable-kvm -daemonize

With an unpatched QEMU, VM creation will fail with:
qemu-system-x86_64: error: failed to set MSR 0x48b to 0x11582e00000000
qemu-system-x86_64: /build/qemu-FC5BvZ/qemu-4.2/target/i386/kvm.c:2680: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

A test package for focal is available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf284806-test

The patched QEMU from the above ppa fixes the problem, and the VM launches successfully.

[Regression Potential]

The code adds a check to see if MSR_IA32_VMX_PROCBASED_CTLS2 is present, and if it is, then we skip over checking of particular CPUIDs and subsequent enabling of VMX's, since the CPUID might exist and the VMX won't.

If a regression were to occur, some systems might be missing VMX's being enabled if these checks were incorrectly skipped, leading to particular features missing for a VM. In the worse case scenario, the VM may not start due to missing or incorrectly set MSRs.

The code is simple and targeted, and I believe the chance for regression is very low.

[Other Info]

The commit that fixes the problem is:

Commit: 4a910e1f6ab4155ec8b24c49b2585cc486916985
From: Vitaly Kuznetsov <email address hidden>
Date: Tue, 31 Mar 2020 18:27:52 +0200
Subject: target/i386: do not set unsupported VMX secondary execution controls
Link: https://github.com/qemu/qemu/commit/4a910e1f6ab4155ec8b24c49b2585cc486916985

This fixes 048c951 ("target/i386: work around KVM_GET_MSRS bug for secondary execution controls"), which was introduced in QEMU 4.2, meaning only Focal and Groovy require fixing.

[Matthew Ruffell (mruffell)]

Attached is a debdiff for focal.