qemu-system-x86 nested virtualisation is broken on AMD system

[Andrew Spiers]

I am reporting this bug against qemu-system-x86=1:2.2+dfsg-5expubuntu9.3~cloud0
in the Ubuntu cloud archive.

# lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04
# apt-cache policy qemu-system-x86
qemu-system-x86:
  Installed: 1:2.3+dfsg-5ubuntu4~cloud0
  Candidate: 1:2.3+dfsg-5ubuntu4~cloud0

What is happening:
Version 1:2.2+dfsg-5expubuntu9.3~cloud0 of qemu-system and qemu-system-x86_64
seems to be dropping the svm (AMD nested virtualization) flag from virtual
machines, despite it being specified in the CPU type. So nested virtualisation
has stoppped working.

What I expect to happen:
svm CPU flag to be passed through to guests.

Version 1:2.3+dfsg-5ubuntu4~cloud0 also drops the svm flag.

Version 2.0.0+dfsg-2ubuntu1.18 passes the CPU flag through fine.

This has been tested with libvirt-bin version 1.2.12-0ubuntu14. The processor
is Version: AMD Opteron(TM) Processor 6238.

# cat /sys/module/kvm_amd/parameters/nested
1
# modinfo kvm_amd | grep nested
parm: nested:int

The qemu command line includes the following cpu argument::

    -cpu Opteron_G4,+perfctr_nb,+perfctr_core,+topoext,+nodeid_msr,+lwp,+wdt,+skinit,+ibs,+osvw,+cr8legacy,+extapic,+c
mp_legacy,+fxsr_opt,+mmxext,+osxsave,+monitor,+ht,+vme

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug.

Could you please show the vm xml (virsh dumpxml domain_name) and the full qemu/kvm command (ps -auxw | egrep -e '(qemu|kvm)' )?

[Andrew Spiers (3-andrew)]

Hello, here is the extra information you have asked for.
Instance 37aa aka kvm id 677 is my currently running test instance, but I have given you
the full ps output you have asked for. 18429 is the PID for this instance.

<domain type='kvm' id='677'>
  <name>instance-000037aa</name>
  <uuid>12396aff-3f84-4ceb-9791-d20cf656fb35</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2015.1.1-a41-g26267dc-trusty"/>
      <nova:name>qemu9</nova:name>
      <nova:creationTime>2015-09-10 06:14:59</nova:creationTime>
      <nova:flavor name="m1.small">
        <nova:memory>4096</nova:memory>
        <nova:disk>10</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>30</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="76a0c7d2dcfc46a68cf1c5f074b79583"><email address hidden></nova:user>
        <nova:project uuid="87a5bb66157d4b42800c131bffafc184">pt-489</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="cfb066e2-9b8f-468a-bf29-e9b9336947af"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>2015.1.1-a41-g26267dc-trusty</entry>
      <entry name='serial'>534d4349-0002-5290-2500-529025005ccf</entry>
      <entry name='uuid'>12396aff-3f84-4ceb-9791-d20cf656fb35</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/nova/instances/12396aff-3f84-4ceb-9791-d20cf656fb35/disk'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/nova/instances/12396aff-3f84-4ceb-9791-d20cf656fb35/disk.local'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <a...

[Andrew Spiers (3-andrew)]

Sorry, I changed the status of this bug by accident.
But should it still be 'incomplete'?

[Serge Hallyn (serge-hallyn)]

no, since you've provided the information it should be reset to New, thanks :)

 status: new

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michael Mallon (durinix)]

Adding vm xml (virsh dumpxml domain_name) and the full qemu/kvm command (ps -auxw | egrep -e '(qemu|kvm)' ) that I'm seeing along with the host and VM /proc/cpuinfo

[Stefan Bader (smb)]

Did I understand this right to be a combination of roughly 14.04 (Trusty) as base with (probably both through the cloud archive) a libvirt version that appears to closest match with 15.04 (Vivid) and qemu at 15.10 (Wily) level?

Unfortunately I cannot recreate the same results as I only got a fam 10h model 09h host. For that I get the svm flag inside the guest on native Trusty, Vivid, and Wily. However nested is actually broken for all as cpuid function 8000000a seems not implemented in any qemu version (2.0.0+dfsg-2ubuntu1.18 ... 1:2.3+dfsg-5ubuntu6).

I have to check what the G4 type (from fam 16h model 02h) would yield. Just also wondering whether in the case of not dropping svm the kvm module actually loads or also fails on cpuid 8000000Ah.

[Stefan Bader (smb)]

So this quite likely might be a "feature" that got introduced with qemu 2.2:

commit 75d373ef9729bd22fbc46bfd8dcd158cbf6d9777
Author: Eduardo Habkost <email address hidden>
Date: Fri Oct 3 16:39:51 2014 -0300

    target-i386: Disable SVM by default in KVM mode

    Make SVM be disabled by default on all CPU models when in KVM mode.
    Nested SVM is enabled by default in the KVM kernel module, but it is
    probably less stable than nested VMX (which is already disabled by
    default).

    Add a new compat function, x86_cpu_compat_kvm_no_autodisable(), to keep
    compatibility on previous machine-types.

I saw both machine types (pc-i440fx-trusty and pc-i440fx-utopic) mentioned in some xml configs. Not sure which is which. Might be worth checking the config in the currently not working case and if that is not trusty, try to replace it with that.

[Stefan Bader (smb)]

Managed to verify that switching between pc-i440fx-trusty and -utopic will cause the svm flag to be dropped (as long as it is not forcefully enabled in the libvirt config).

And my special failure seems to be caused by another oddness in qemu: while a Opteron_G4 is too recent and I would have to disable things, the Opteron_G3 which normally gets picked as a base is far too old. (the G4 is based on 62xx but G3 is based on 23xx, while I got a 61xx). The better match for my case seems "phenom" which at least sets the right cpuid_level and xlevel.