qemu-kvm: ksmd should have a less CPU-hogging default config

[lequeux1]

Binary package hint: qemu-kvm

Lucid 64 bits

Expected: have qemu-kvm propose adquate default values
Found: using kvm, running 2 VMs , ksmd constantly eats 20 to 30%CPU

Lucid script used to tune ksm (/etc/init.d/qemu-kvm , parms in /etc/init/qemu-kvm.conf) doesnt provides adequate values IMHO

-> Suggest modify qemu-kvm.conf
 either to deactivate ksm
 or have 200 for SLEEP_MILLISECONDS

Similar issue found on proxmox (debian) proposing default values:
http://forum.proxmox.com/threads/3279-Correct-way-to-KSM-tunig-and-manage-2-6-32?p=18415#post18415

System:
# lsb_release -rd
Description: Ubuntu 10.04 LTS
Release: 10.04

#uname -a
Linux freuan0l000075 2.6.32-22-generic #33-Ubuntu SMP Wed Apr 28 13:28:05 UTC 2010 x86_64 GNU/Linux

Package: qemu-kvm 0.12.3+noroms-0ubuntu9

[Dustin Kirkland  (kirkland)]

Hi there-

You can actually set it to whatever you want, and have it persist across boots. Just modify the values in the file /etc/default/qemu-kvm.

We're just using the default, upstream kernel value for SLEEP_MILLISECONDS, which is 20. It does seem a bit aggressive, but it's the upstream default. We should have a discussion with them to see if it makes sense to change it.

However, I'm going to mark this won't-fix against the qemu-kvm package, since we have an easy configuration file where you can tune this value to your liking.

[Rakotomandimby Mihamina (tech-infogerance)]

See also https://bugzilla.redhat.com/show_bug.cgi?id=541230

[MNLipp (mnl)]

I don't think "Won't Fix" is appropriate. If you look at at "kernel/Documentation/vm/ksm.txt" you find that the values for "pages_to_scan" and "sleep_millisecs" are documentated as "chosen for demonstration purposes".

So "upstream" obviously does not consider these values appropriate for normal operation. Therefore I think ksm should either be distributed as disabled or with a less aggressive configuration.

[Andy (andy-xillean)]

Is this going to be fixed?

[Dustin Kirkland  (kirkland)]

As stated above, I personally don't plan on fixing it in Ubuntu.

You can set that value to whatever you want in the config file.

Dustin

[Dave Walker (davewalker)]

This does seem to be a valid bug, some serious CPU time is being dedicated to ksmd. Whilst I haven't seen it in Lucid, i have experienced it in Maverick. Smoser witnessed ~17% CPU utilization for ksmd.

Nominating for Maverick.

[Thierry Carrez (ttx)]

17% sounds a bit strong... I'd very much like to see a memory gain / CPU usage chart for various values, or more discussion around the acceptable value upstream. Any idea what the other qemu-kvm distributions are using as a default ?

[Nullzone (nullzone)]

Hello Thierry,

  I will try to help showing some information gathered from my lab server as you requested:

- Enviroment:

 10 x Ubuntu 10.04 x86_64 with several services (SMTP, MySQL, Oracle, DB2, TeamSpeak servers, Tomcats, bind ... etc).
 1 x Windows 7 x86_64
 3 x Debian Lenny x86_64 (iscsitarget, pacemaker cluster testing)

 RAM schema (assigned to the VMs): 3 vms 128 MB, 6 vms 256 MB, 2 vms 512 MB, 1 vm 768 MB, 2 vms 1GB
 SMP: 5 vms with 2 SMP (num vCPU) configured, rest 1 SMP

- Hardware:

 1 x Intel(R) Core(TM)2 Quad CPU @ 2.66GHz
 8 GB RAM DDR2 800 Mhz / 25 ns

 ---------------------------------------------

* Summary:

Usefull information:
  CPU % usage goes stable, also memory is examined and released after the 3rd full_scans
  CPU % usage is specially high between 1st and 2nd scan (when ksm is merging pages at the same time that it is doing the second scan)
  Memory starts being released after the 1st scan is completly and the 2nd is in progress (so past some minutes)

KSM default pages_to_scan:100 sleep_millisecs:20 CPU: ~11,67% Time_for_a_full_scan: 3m35.209s
KSM pages_to_scan:60 sleep_millisecs:40 CPU: ~3,44% Time_for_a_full_scan: 11m50.734s
KSM pages_to_scan:100 sleep_millisecs:50 CPU: ~4,21% Time_for_a_full_scan: 8m55.533s
KSM pages_to_scan:60 sleep_millisecs:60 CPU: ~2,00% Time_for_a_full_scan: 17m46.094s
KSM pages_to_scan:100 sleep_millisecs:120 CPU: ~1,67% Time_for_a_full_scan: 21m26.285s

Mem Optimization in all the tests (there before or after):
  RAM available (reminder): 8 GB
  Memory optimizacition thanks to ksm: 14-16% full available memory (~ 1.3 / 1.1 Gb of more free memory)

Conclusions:
  ksm is good but need some tweaks to doesnt become a problem more than a help
  I'd consider to off ksm in /etc/default/qemu-kvm for desktop versions of Ubuntu (availabe but off by default)
  I'd consider to switch default value of sleep_millisecs: 120 even a bit higher in the ubuntu server version. More conservative...

 -------------------------------------------

 root@core:/sys/kernel/mm/ksm# echo 2 > /sys/kernel/mm/ksm/run
 root@core:/sys/kernel/mm/ksm# sleep 30
 root@core:/sys/kernel/mm/ksm# landscape-sysinfo
   System load: 1.02 IP address for lo: xxx
   Usage of /home: 66.1% of 2.68TB IP address for eth1: xxx
   Memory usage: 60% IP address for eth1:0: xxx
   Swap usage: 0% IP address for bond0: xxx
   Processes: 199 IP address for br0: xxx
   Users logged in: 1
 root@core:/sys/kernel/mm/ksm# echo 1 > /sys/kernel/mm/ksm/run ; time /tmp/tmp.bash
 real 3m35.209s
 user 0m0.070s
 sys 0m0.130s
 [... /tmp/tmp.bash
     #!/bin/bash

     while true
     do
       SALIDA=`cat /sys/kernel/mm/ksm/full_scans`
       if [ $SALIDA == "1" ]
       then
          break
       else
          sleep 5
       fi
     done
 .... yeee... terrible but enough for a script done in 10 secs...]
 [...waiting for a pair of aditional scans...]
 root@core:/sys/kernel/mm/ksm# cat full_scans
 3
 root@core:/sys/kernel/mm/ksm# ps axuw | grep ksm | grep -v grep
 root 53 10.7 0.0 0 0 ? SN 17:51 12:08 \_ [ksmd...

[Thierry Carrez (ttx)]

@Nullzone: that's some very good input, thanks. Looking at your conclusions:

> ksm is good but need some tweaks to doesnt become a problem more than a help
Right, if there is a saner default we should use it, and if there is no sane default we should consider disabling it by default, with strong pointers on how to enable it with the right sleep_millisecs value.

> I'd consider to off ksm in /etc/default/qemu-kvm for desktop versions of Ubuntu (availabe but off by default)
That's difficult to do, since packages have no way to tell if they are on a "server".

> I'd consider to switch default value of sleep_millisecs: 120 even a bit higher in the ubuntu server version. More conservative...
+1 on choosing a less aggressive setting. Ideally upstream would also change its default setting.

CCing Serge Hallyn since he now owns that stack, for his input.

[Serge Hallyn (serge-hallyn)]

Thanks for the info Nullzone! Based on your data, I'm going to spend a
bit of time running with:

# To disable qemu-kvm's page merging feature, set KSM_ENABLED=0 and
# sudo restart qemu-kvm

 KSM_ENABLED=1
 #SLEEP_MILLISECS=2000
 SLEEP_MILLISECS=50
 PAGES_TO_SCAN=100

in my /etc/default/qemu-kvm, and

  if [ -w /sys/kernel/mm/ksm/pages_to_scan ]; then
   if [ -n "$PAGES_TO_SCAN" ]; then
    echo "$PAGES_TO_SCAN" > /sys/kernel/mm/ksm/pages_to_scan
   fi
  fi

added to my /etc/init/qemu-kvm.conf

I think upping sleep_millisec for Natty is a good idea. But not
for Lucid+Maverick, it's just too late, and after all these are
trivially changeable and really ought to be customized anyway.

I also think (obviously) that adding KSM_PAGES_TO_SCAN to the list of
variables which /etc/init/qemu-kvm can read seems worthwhile. However
again I'm not sure whether I want to ask for a feature freeze exception
for maverick for this. I personally think it's better to wait for Natty.
Thierry, do you concur, or should I try for maverick?

[Thierry Carrez (ttx)]

I agree we should not change the behavior too much now that FeatureFreeze is passed. Adjusting tasks in consequence.

[thearthur (arthur-ulfeldt)]

In maverick this bug is consuming 20% cpu 100% of the time. it really kills the battery life.

[Derek Simkowiak (ubuntu-cool-st)]

I'm getting burned by this too, and I disagree with Mr. Kirkland's assertion that Ubuntu should just use the upstream default.

There are many, many Ubuntu packages which alter the "upstream default" to make Ubuntu a more user-friendly experience. One could even argue that Ubuntu's best feature is that it changes the "upstream defaults" of the world to be something that actually works for human beings.

Please, set this to a reasonable value (200ms?). As has been pointed out, the "upsteam default" is not even considered a reasonable value by the upstream maintainers.

Ethically speaking, this bug is causing more CPU power consumption (and thus, global warming) for absolutely no reason whatsoever.

[Toby Corkindale (tjc-wintrmute)]

I'm disappointed in Ubuntu's decision not to fix this..
I find the alternative sleep time of 200ms to be far more sensible, rather than having ksmd burn thru CPU cycles constantly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu-kvm - 0.14.0+noroms-0ubuntu2

---------------
qemu-kvm (0.14.0+noroms-0ubuntu2) natty; urgency=low

  * debian/qemu-kvm.default: per popular request, reduce the ksm sleep
    milliseconds to 200ms, LP: #578930
 -- Dustin Kirkland <email address hidden> Wed, 09 Mar 2011 11:21:40 +0000