KVM VMs loses their CPU (kvm84 backport/hardy)

[Yann Hamon]

Binary package hint: kvm

Using the KVM84 backport for hardy and the libvirt backport, it seems that VMs regularly just "lose" their CPU. Can't provide much more infos sadly, as the VM just stops writing to log files as soon as this happens.

What I can provide though is a munin graph of when it happens:

http://waste.mandragor.org/kvmcpu/munin.html

The downtime at 2 in the morning was a reboot of the host (therefore normal). At 6 o'clock though you can see the IRQs dropping and the CPU just "disappearing" from the graph. The other thing I don't understand is how munin continued working...

Rebooting the VM solves the issue.

[Soren Hansen (soren)]

Apart from you monitoring system giving odd output, what are the symptoms? As you say yourself, munin still runs, so the machine still seems to be functional?

[Yann Hamon (yannh)]

Hello Soren, no, VM is down, cant ssh, services don't work anymore.

[Soren Hansen (soren)]

On Mon, Nov 30, 2009 at 10:47:16AM -0000, Yann Hamon wrote:
> Hello Soren, no, VM is down, cant ssh, services don't work anymore.

Except munin.. ? could it just be networking that's failing? Do you see
anything on the VM's console?

[Yann Hamon (yannh)]

Mmmh no don't think it could be the network, as munin is still working. I didn't check the VM console, but it's a good idea, I will try this next time. I kind of suspect ACPI, so will disable ACPI on failing VMs next time too.

[Soren Hansen (soren)]

On Tue, Dec 01, 2009 at 01:00:54PM -0000, Yann Hamon wrote:
> Mmmh no don't think it could be the network, as munin is still working.
> I didn't check the VM console, but it's a good idea, I will try this
> next time. I kind of suspect ACPI, so will disable ACPI on failing VMs
> next time too.

Be aware that disabling ACPI can have very unfortunate side effects.

acpi is lower resolution timing source, but is not counter based, so it
will be better at preventing time slide in the guest. If the guest
kernel detects instability in the TSC, it will fall back to ACPI. If you
look at a few of your guests, I would be surprised if not at least 75%
had fallen back to ACPI.

Also, without ACPI you will not be able to cleanly shutdown the VM from
libvirt, but will have to ssh into the guest and issue a shutdown.

[Yann Hamon (yannh)]

Hello soren, about this bug: I found out after a while that it was happening only on my 64 bits hosts.
What happened is that inadvertently I installed ubuntu 32 bits on VMs with a 64bits CPU. I never noticed this on kvm62 as it was running fine. However when I upgraded to the backports all the VMs from the host regularly crashed, with 10 VMs I used to have at least one VM hanging a day.

When i found out this bitness issue, I moved all the VMs back to 32 bits. It seems to run fine now, haven't had a single crash since then. Haven't tried 64bits guests on 64bits VMs. Not sure if it's worth fixing - as the workaround seems ok, and it might affect only 32 bits OSes on 64bits CPUs, which I admit is a weird thing to do with VMs. If there is a "known issues" list somewhere, it would be worth adding it there though :)

Thanks for your time

[Soren Hansen (soren)]

On Tue, Dec 29, 2009 at 02:36:43PM -0000, Yann Hamon wrote:
> When i found out this bitness issue, I moved all the VMs back to 32
> bits.

I'm not sure I completely understand what you mean. What exactly did you
change?

[Yann Hamon (yannh)]

Hello Soren, I had a 64bits host running 32 bits guests on a 64bit virtual CPU. This didn't work.
Now I have a 64bits host running 32 bits guests on a 32bits virtual CPU. This works fine.

Just changed x86_64 to i686 in the XML definition of the VM.

[Yann Hamon (yannh)]

Sorry if that wasn't clear: when I wrote "guest" I meant "Operating systems".

[Launchpad Janitor (janitor)]

[Expired for kvm (Ubuntu) because there has been no activity for 60 days.]