kvm-qemu, libvirt-bin: kernel panic if using memory balloon

[Thomas Schweikle]

Set up an ubuntu-server, install kvm-qemu, libvirt-bin and necessary tools.
Create various virtual machines running various Linux distributions. Make sure all VM together allocate more memory than the real host has, if they use all memory given to them. Set the minimal given memory to a value if all VM allocate memory the hosts memory isn't exceeded.
Make sure there is enough swap defined!

For example:
Host 8192MiB; define 14 VM minimal memory 512MiB, maximal memory 2048MiB.
If all hosts take minimal memory 7168MiB are allocated, leaving 1024MiB for the host.
But all VM are allowed to allocate as much as 28672MiB --- far more than the host can provide.
Start all available VM at once.
At startup all seems fine, until the host begins to swap. It's going to slow down everything, but nothing else happens. After a while most of the VM receive a kernel panic, because of out of memory conditions.

Second test:
Host 8192MiB; define 14 VM minimal memory 2048MiB, maximal memory 2048MiB.
Start all VM successive. The first three are no problem. The forth forces the host to swap. The host starts to slow down significantly. Start all remaining VM. The host will be setremely slow, VMs may need an hour or so to boot, but no kernel panic.

As soon as memory balooning is enabled again: VMs receive kernel panik.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: libvirt-bin 0.9.2-4ubuntu15.1
Uname: Linux 3.1.8 x86_64
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
Date: Fri Jan 13 16:43:29 2012
InstallationMedia: Xubuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111012)
ProcEnviron:
 PATH=(custom, user)
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)

[Serge Hallyn (serge-hallyn)]

Thanks for taking the time to report this bug.

By 'make sure there is enough swap defined', do you mean that host memory + host swap is greater than than the sum of memory used by all guests?

[Thomas Schweikle (tps)]

Yes.
This way most memory used by guests can be swapped (inefficient, but works) out if necessary because others need RAM to run.

[Serge Hallyn (serge-hallyn)]

Thanks. I see now. The 'second test' pinpoints ballooning as the source of the bug. Got it.

I'll try to reproduce this when I get a chance. In the meantime, if it's at all possible for you to try and reproduce this with the current upstream git tree, that be helpful.

Since this only affects the case of extereme overprovisining, and "don't enable ballooning" appears to be a viable workaround, I will mark this low priority per priority guidelines.

[Serge Hallyn (serge-hallyn)]

I'm testing this in quantal, and not quite seeing what you mean. When I switch the 'current' memory (using virt-manager) from 512m to 1024m, /proc/meminfo in the guest shows the updated values; likewise when I lower it back to 512m, after a few seconds the guest shows the new values. Of course, that requires the virtio_balloon kernel module to be loaded. To be clear, libvirt does not monitor the host's free memory or automatically lower the 'currently allocated' memory amount. That would need to be done by a separate monitoring tool.

Note that times the kvm process will take more than 512M ram. 512M is what you are allocating to the guest, but not what you are limiting the qemu-kvm process to. Therefore, with 14 guests with 512M ram, you should expect at least 7420M allocated at times by the kvm processes (that's assuming 530M per kvm process, which is the most I've seen in my test today, with guests compiling a kernel image).

I've left 4 VMs (512M allocated, 2048 max memory on a 4G+4G laptop) running a kernel compile for awhile. The guests do not automatically balloon up, so the most they've taken is 530M each. Did your guests automatically balloon up so that /proc/meminfo showed > 512M?

Do you still see the problem you were reporting with precise or quantal?

[Launchpad Janitor (janitor)]

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]