qemu should not enable KSM on nested guests

[Chris J Arges]

This issue has been split from bug 1413540.

After discussing this bug it seems that we should be smarter with our upstart script such that nested guests inside KVM, Xen, VMWare, etc do _not_ enable KSM by default as this rarely would be optimal. Better solutions would be to use ksmtuned to manage this, but since this tool is not in debian/ubuntu and not as actively maintained we should first fix the upstart script.

[Serge Hallyn (serge-hallyn)]

So did we come up with a good way to detect not being on bare metal?

In fact ksm gets enabled by /etc/init/qemu-kvm which is only installed on a subset of architectures so using virt-what may be a possibliity, however I'd still prefer not to add virt-what as a dependency if we can come up with a better way.

[Chris J Arges (arges)]

Working on it. I'll post a 'detect' script for review first.

[Chris J Arges (arges)]

#!/bin/bash
# 2015 Chris J Arges <email address hidden>
# Detect if we are running inside KVM
NESTED_VM=0
VM_STRINGS="KVM QEMU VMware VirtualBox Xen"
VM_DETECT=$(dmesg | egrep -e '(Hypervisor detected|Booting paravirtualized kernel)')
VM_DMIDECODE=$(sudo dmidecode | egrep -i 'manufacturer|product|vendor')
for vm_string in $VM_STRINGS; do
  if [[ ${VM_DETECT}${VM_DMIDECODE} == *"${vm_string}"* ]]; then
    NESTED_VM=1; break;
  fi
done
echo "NESTED_VM = $NESTED_VM"

[Ryan Harper (raharper)]

One method that's pretty solid for QEMU (save folks who pass in custom DMI
table values to qemu) is the BIOS data available via dmidecode (or
/sysfs/dmi); would need to look at Power and arm for equivalent (likely
some device tree bits in /sysfs). Openstack exports a BIOS manufacture of
Openstack, the default QEMU bios exposes QEMU and SeaBIOS. These would be
good indicators for the the majority of default installs. Obviously it can
be worked around since the invoker can specify whole DMI table values (or
pass a complete dmi table blob from the host into the guest) but we don't
have to be foolproof.

On Fri, Jan 23, 2015 at 5:21 PM, Serge Hallyn <email address hidden>
wrote:

> So did we come up with a good way to detect not being on bare metal?
>
> In fact ksm gets enabled by /etc/init/qemu-kvm which is only installed
> on a subset of architectures so using virt-what may be a possibliity,
> however I'd still prefer not to add virt-what as a dependency if we can
> come up with a better way.
>
>
> ** Changed in: qemu (Ubuntu)
> Status: New => Triaged
>
> ** Changed in: qemu (Ubuntu)
> Importance: Undecided => High
>
> --
> You received this bug notification because you are subscribed to qemu in
> Ubuntu.
> https://bugs.launchpad.net/bugs/1414153
>
> Title:
> qemu should not enable KSM on nested guests
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1414153/+subscriptions
>

[Chris J Arges (arges)]

I could amend my script in #3 to include Openstack SeaBIOS as well. And obviously testing this in VMs across virtualization platforms and arches would help as well. I've only tested on x86/KVM, x86/EC2.

[Ryan Harper (raharper)]

On Fri, Jan 23, 2015 at 10:16 PM, Chris J Arges <email address hidden>
wrote:

> #!/bin/bash
> # 2015 Chris J Arges <email address hidden>
> # Detect if we are running inside KVM
> NESTED_VM=0
> VM_STRINGS="KVM QEMU VMware VirtualBox Xen"
> VM_DETECT=$(dmesg | egrep -e '(Hypervisor detected|Booting paravirtualized
> kernel)')
> VM_DMIDECODE=$(sudo dmidecode | egrep -i 'manufacturer|product|vendor')
>

dmidecide is x86 only,

I've got access to a power8 kvm, I'll see what I can see there.

And if someone has some ARM64 system, that'd be helpful too.

for vm_string in $VM_STRINGS; do
> if [[ ${VM_DETECT}${VM_DMIDECODE} == *"${vm_string}"* ]]; then
> NESTED_VM=1; break;
> fi
> done
> echo "NESTED_VM = $NESTED_VM"
>
> --
> You received this bug notification because you are subscribed to qemu in
> Ubuntu.
> https://bugs.launchpad.net/bugs/1414153
>
> Title:
> qemu should not enable KSM on nested guests
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1414153/+subscriptions
>

[Chris J Arges (arges)]

I'll look for a more agnostic solution. We most likely don't want to run KSM in L1 by default in any guest not just one that will host nested VMs.

[Ryan Harper (raharper)]

There's nothing preventing nested guests architectually IIUC. I don't know
that anyone has implemented it yet but seems reasonable to thing ahead and
avoid x86isms where possible.

On Sat, Jan 24, 2015 at 3:27 PM, Chris J Arges <email address hidden>
wrote:

> Ryan,
> I don't think Power8/kvm nor ARM/kvm allow for nested guests, so do we
> need to worry about in those cases?
> --chris
>
> --
> You received this bug notification because you are subscribed to qemu in
> Ubuntu.
> https://bugs.launchpad.net/bugs/1414153
>
> Title:
> qemu should not enable KSM on nested guests
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1414153/+subscriptions
>

[Chris J Arges (arges)]

FWIW, here is the systemd-virt-detect code used to detect if we are running on a virt platform.
https://github.com/systemd/systemd/blob/master/src/shared/virt.c

This would be the proper way to detect when using systemd, but we still need a bashy way of doing this with upstart. I'll look at merging some of the ideas here.

[Ryan Harper (raharper)]

Shame that virt.c isn't a standalone tool that could be reused.

On Thu, Jan 29, 2015 at 9:54 AM, Chris J Arges <email address hidden>
wrote:

> FWIW, here is the systemd-virt-detect code used to detect if we are
> running on a virt platform.
> https://github.com/systemd/systemd/blob/master/src/shared/virt.c
>
> This would be the proper way to detect when using systemd, but we still
> need a bashy way of doing this with upstart. I'll look at merging some
> of the ideas here.
>
> --
> You received this bug notification because you are subscribed to qemu in
> Ubuntu.
> https://bugs.launchpad.net/bugs/1414153
>
> Title:
> qemu should not enable KSM on nested guests
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1414153/+subscriptions
>

[Chris J Arges (arges)]

Ok so that particular tool is packaged with the 'systemd' package. For example:
ubuntu@vivid:~$ systemd-detect-virt -v
kvm

And it will return 0 if we're on a virt platform. This would be nice to use on vivid, but we don't really have a systemd script yet. So a few options:
1) Dep on systemd package, and run this program to check for virt in the upstart or init script.
2) Ensure upstart has the right bash equivalent and just check like that.
3) Properly add a systemd conf file to debian, and amend that properly...

--chris

[Chris J Arges (arges)]

After testing in a power8 KVM VM, I added the following to systemd to allow it to detect if its in a VM:

https://github.com/systemd/systemd/commit/d831deb512ab1d11aab156f69620db506c554170

[Chris J Arges (arges)]

Attached is a method to do this on x86 only. This abuses the postinst to check if we are on a guest and then edit /etc/default/qemu-kvm to enable or disable KSM.