issues with secondary VMX execution controls

[Christian Ehrhardt ]

[Impact]

In qemu 4.2 was a change [1] meant to improve the handling of MSRs vs CPUID.
It was later identified [2] as an issue and fixed.
This has to be backported to Focal to resolve that issue on several platforms.

An example where this occurs is:
- Azure instances with nested virt
- GCP instances with nested virt

We have seen a bunch of qemu named CPU types that can expose similar behavior when used on chips that pretend to be of some type e.g. Skylake but miss some of their features to be settable.
It isn't entirely sure thou that this will be fixed by the same - yet worth to mention.

The impact is that qemu 4.2 as in Ubuntu 20.04 doesn't work on those platforms bailing out.

[1]: https://github.com/qemu/qemu/commit/048c95163b472ed737a2f0dca4f4e23a82ac2f8a
[2]: https://github.com/qemu/qemu/commit/4a910e1f6ab4155ec8b24c49b2585cc486916985

[Test Case]

 * Get a GCP or Azure instance with nested virtualization enabled
 * Spawn a KVM guest on it e.g. by using uvtool-libvirt using a named type
   matching the cpu
   e.g. if the host reports as skylake use such a type.
   You can use `virsh domcapabilities` to check what the host is
   detected as.

[Regression Potential]

 * It is a bit hard to guess, but it should not make things worse. But if I'd expect one then the
   VMX subfeatures could change on cases not intended to. Yet we should have one of two cases:
   a) the common one is that the host can set this and has done so, it will continue as before
   b) host was unable to set these and failed, this should now work with the fix in place
   Both seem ok to me.

[Other Info]

 * there might be a local (non cloud) way to reproduce but I don't know it yet

[Christian Ehrhardt  (paelzer)]

Test fix started to build in PPA (for Focal):
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4081/

It would be great if one with GCP or Azure instances affected could give this a try.

[Christian Ehrhardt  (paelzer)]

FYI there is an older build in the PPA, only >=qemu - 1:4.2-3ubuntu6.3~ppa2 will have the proposed fix for this issue.

[Nell Jerram (neil-jerram)]

Christian, would it be easy for you to build this for Bionic as well? I run tests with GCP instances as the OpenStack hypervisors, and have hit this bug, so am keen to test your fix.

(http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015343.html)

[Christian Ehrhardt  (paelzer)]

I have tested this with a GCE instance with nested enabled
here the domcapabilities for the type:
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Skylake-Client-IBRS</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='ssbd'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='disable' name='mpx'/>
      <feature policy='disable' name='xsavec'/>
      <feature policy='disable' name='xgetbv1'/>
    </mode>

I set the guest (on Bionic) to use host-model:
   <cpu mode="host-model"/>

Due to that it got on first execution the model generated as the above reported type:

  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='disable' name='mpx'/>
    <feature policy='disable' name='xsavec'/>
    <feature policy='disable' name='xgetbv1'/>
  </cpu>

Guest starts fine, no related errors in /var/log/libvirt/qemu/testguest.log

After an update to focal the type is now reported as unusable
      <model usable='no'>Skylake-Client-IBRS</model>

The guest would now be detected as, it thinks this definition is now closer:

    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Broadwell-IBRS</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='vme'/>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='f16c'/>
      <feature policy='require' name='rdrand'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='arat'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='ssbd'/>
      <feature policy='require' name='xsaveopt'/>
      <feature policy='require' name='abm'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='rsba'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
    </mode>

No matter if I use host-model of the former Skylake type - the guest now starts with the reported crash:

$ sudo tail -f /var/log/libvirt/qemu/testguest.log
...
error: Failed to start domain testguest
error: internal error: process exited while connecting to monitor: 2020-06-09T13:51:27.110925Z 2020-06-09T13:51:27.111798Z qemu-system-x86_64: error: failed to set MSR 0x48b to 0x159ff00000000
qemu-system-x86_64: /build/qemu-7aKH5L/qemu-4.2/target/i386/kvm.c:2680: kvm...

[Christian Ehrhardt  (paelzer)]

@Neil - it isn't too easy, but I was able to confirm myself in the meantime and it works well.

The usual flow is that I bring this fix into the affected Groovy and Focal and then the cloud-archive would rebase onto those versions.
But I'll add a ubuntu-cloud-archive task.
Maybe they can pre-build you something based on the PPA that is linked here - which you then can test on Bionic as well - just so we can be sure that the different older VMX controls on the bionic kernel will work the same way.
@James / Corey - could you do so?

[Christian Ehrhardt  (paelzer)]

FYI I'm waiting for the fix to bug 1878973 to be committed and then plan to roll both together into groovy.

[Christian Ehrhardt  (paelzer)]

FYI: Uploaded to Groovy

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:4.2-3ubuntu9

---------------
qemu (1:4.2-3ubuntu9) groovy; urgency=medium

  * debian/patches/ubuntu/lp-1878973-*: fix assert in qemu-guest-agent that
    crashes it on shutdown (LP: #1878973)
  * d/p/ubuntu/lp-1882774-*: fix issues with VMX subfeatures on systems not
    supporting to set them (LP: #1882774)

 -- Christian Ehrhardt <email address hidden> Tue, 02 Jun 2020 10:42:49 +0200

[Christian Ehrhardt  (paelzer)]

Complete in groovy, preparing for SRUs - SRU-Template ready, MPs are up.

@SRU Team - already ahead as hint please release qemu 1:4.2-3ubuntu6.2/1:4.0+dfsg-0ubuntu9.7 of bug 1805256 which is ready, before accepting this one.

[Christian Ehrhardt  (paelzer)]

Ok, the other SRU is released and the MP reviewed.
Uploaded this one to F-unapproved

[Matthew Ruffell (mruffell)]

Hi Christian, I have tested the exact same patch on Azure over on bug 1884470 and I can confirm that it fixes the issue. I will be eagerly awaiting your upload to hit -proposed.

[Łukasz Zemczak (sil2100)]

Hello Christian, or anyone else affected,

Accepted qemu into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:4.2-3ubuntu6.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:4.2-3ubuntu6.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

livecd-rootfs/2.664.2 (amd64)
systemd/245.4-4ubuntu3.1 (ppc64el)
edk2/0~20191122.bd85bf54-2ubuntu3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Christian Ehrhardt  (paelzer)]

I've started to look into the autpkgtest fails, so far all look like unreliable tests not triggered by the upload. I have re-triggered accordingly and will investigate in case one stays unresolved.

[Christian Ehrhardt  (paelzer)]

Prior to the fix starting a host-model guest on GCE:

ubuntu@nested-vm:~$ virsh start focal
error: Failed to start domain focal
error: internal error: process exited while connecting to monitor: 2020-06-23T09:54:47.062832Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-06-23T09:54:47.062847Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-06-23T09:54:47.062862Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-ept-execonly [bit 0]
2020-06-23T09:54:47.062875Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-eptad [bit 21]
2020-06-23T09:54:47.062891Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(491H).vmx-eptp-switching [bit 0]
2020-06-23T09:54:47.063762Z qemu-system-x86_64: error: failed to set MSR 0x48b to 0x159ff00000000
qemu-system-x86_64: /build/qemu-FC5BvZ/qemu-4.2/target/i386/kvm.c:2680: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

Updating to focal-proposed worked nicely, retesting the case ...

ubuntu@nested-vm:~$ virsh start focal
Domain focal started

The guest is up and running

ubuntu@nested-vm:~$ virsh list
 Id Name State
-----------------------
 3 focal running

The guest log is now without errors:
2020-06-23 09:57:29.845+0000: starting up libvirt version: 6.0.0, package: 0ubuntu8.1 (Christian Ehrhardt <email address hidden> Wed, 20 May 2020 06:59:57 +0200), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.3, kernel: 5.4.0-1015-gcp, hostname: nested-vm.c.prjparide.internal
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-3-focal \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-focal/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-focal/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-focal/.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=focal,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-focal/master-key.aes \
-machine pc-q35-focal,accel=kvm,usb=off,dump-guest-core=off \
-cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,abm=on,ibpb=on,amd-ssbd=on,rsba=on,skip-l1dfl-vmentry=on \
-m 512 \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 2e2a0717-613b-4f9c-91eb-59f93f5ecddb \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=30,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-devic...

[Matthew Ruffell (mruffell)]

Also verifying this on Azure Cloud.

I enabled -proposed and installed qemu 4.2-3ubuntu6.3 on a Focal instance.

$ sudo apt-cache policy qemu-kvm | grep Installed
  Installed: 1:4.2-3ubuntu6.3

From there I executed the following:

$ sudo -s
# apt install qemu-kvm bridge-utils
# BR_NAME="br0"
# BR_ADDR="192.168.1.10"
# ip link add "$BR_NAME" type bridge
# ip addr add "$BR_ADDR"/24 dev "$BR_NAME"
# ip link set "$BR_NAME" up
# tap_name="tap1"
# br_name=$BR_NAME
# ip tuntap add $tap_name mode tap user $(whoami) multi_queue
# ip link set $tap_name up
# ip link set $tap_name master $br_name

Download a guest VM:

# wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img

Launch the VM:
# /usr/bin/qemu-system-x86_64 -cpu host -smp 2 -m 2048 -hda focal-server-cloudimg-amd64-disk-kvm.img -device virtio-net-pci,netdev=net0,mac=52:54:00:f5:be:83 -netdev user,id=net0,hostfwd=tcp::60022-:22 -device virtio-net-pci,netdev=net1,mac=52:54:00:c6:29:dc,mq=on,vectors=10 -netdev tap,id=net1,ifname=tap1,script=no,vhost=on,queues=4 -display none -enable-kvm -daemonize

QEMU successfully starts without error, and checking ps and logs shows the QEMU VM is alive and well:

2500 ? Sl 0:31 /usr/bin/qemu-system-x86_64 -cpu host -smp 2 -m 2048 -hda focal-server-cloudimg-amd64-disk-kvm.img -device virtio-net-pci,netdev=net0,mac=52:54:00:f5:be:83 -netdev user,id=

I can ping the VM and ssh in.

I am happy to also say this bug is verified.

[Christian Ehrhardt  (paelzer)]

Thank you Matthew!

[Christian Ehrhardt  (paelzer)]

All autppkgtest errors resolved and verification done - ready once it is long enough in proposed without an issue found.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:4.2-3ubuntu6.3

---------------
qemu (1:4.2-3ubuntu6.3) focal; urgency=medium

  * debian/patches/ubuntu/lp-1878973-*: fix assert in qemu-guest-agent that
    crashes it on shutdown (LP: #1878973)
  * d/p/ubuntu/lp-1882774-*: fix issues with VMX subfeatures on systems not
    supporting to set them (LP: #1882774)

 -- Christian Ehrhardt <email address hidden> Tue, 02 Jun 2020 10:42:49 +0200

[Chris MacNaughton (chris.macnaughton)]

The fix for the cloud-archive looks to have successfully backported from the distro changes

[Nell Jerram (neil-jerram)]

Thanks to everyone who contributed to this fix. I confirm that it works for me in testing with Ussuri and Ubuntu Bionic on vmx-enabled GCP VMs; with fixed qemu packages coming from cloud-archive:ussuri.