CPU hot plug kills the guest

Bug #878422 reported by Simon Déziel on 2011-10-19
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Low
Unassigned
Oneiric
Undecided
Unassigned
qemu-kvm (Ubuntu)
High
Unassigned
Oneiric
Undecided
Unassigned

Bug Description

===============================================
SRU Justification:
1. Impact: libvirt-qemu guest crashes
2. Development fix: refuse attempt to hotplug a new cpu
3. Stable fix: same as development fix
4. Test case:
 a. create a vdisk: qemu-img create disk.img 1G
 b. start a qemu instance: kvm -vnc :1 -monitor stdio -hda disk.img
 c. in the monitor on stdio, add a cpu:
     cpu_set 2 online
   Without this patch, kvm will print an error and die. With the patch, it
   will continue to run.
5. Regression potential: none
===============================================

I'm using a Natty host and an Oneiric guest (both fully up to date) to do some KVM testing. Hot plugging a CPU like this :

virsh setvcpus oneiric --count 2 --live

generates the following messages in /var/log/libvirt/qemu/oneiric.log :

kvm: /build/buildd/qemu-kvm-0.14.0+noroms/hw/qdev.c:95: qdev_create_from_info: Assertion `bus->allow_hotplug' failed.
2011-10-19 14:32:53.124: shutting down

From there, obviously the guest dies.

Host details :
$ apt-cache policy qemu-kvm
qemu-kvm:
  Installed: 0.14.0+noroms-0ubuntu4.4
  Candidate: 0.14.0+noroms-0ubuntu4.4
  Version table:
 *** 0.14.0+noroms-0ubuntu4.4 0
        500 http://ca.archive.ubuntu.com/ubuntu/ natty-updates/main amd64 Packages
        500 http://ca.archive.ubuntu.com/ubuntu/ natty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     0.14.0+noroms-0ubuntu4 0
        500 http://ca.archive.ubuntu.com/ubuntu/ natty/main amd64 Packages
$ apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 0.8.8-1ubuntu6.6
  Candidate: 0.8.8-1ubuntu6.6
  Version table:
 *** 0.8.8-1ubuntu6.6 0
        500 http://ca.archive.ubuntu.com/ubuntu/ natty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.8.8-1ubuntu6.5 0
        500 http://ca.archive.ubuntu.com/ubuntu/ natty-updates/main amd64 Packages
        500 http://ca.archive.ubuntu.com/ubuntu/ natty-security/main amd64 Packages
     0.8.8-1ubuntu6 0
        500 http://ca.archive.ubuntu.com/ubuntu/ natty/main amd64 Packages

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: qemu-kvm 0.14.0+noroms-0ubuntu4.4
ProcVersionSignature: Ubuntu 2.6.38-12.51-generic 2.6.38.8
Uname: Linux 2.6.38-12-generic x86_64
Architecture: amd64
Date: Wed Oct 19 14:30:44 2011
KvmCmdLine: Error: command ['ps', '-C', 'kvm', '-F'] failed with exit code 1: UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
MachineType: LENOVO 2516CTO
ProcEnviron:
 LANGUAGE=en_US:en
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-2.6.38-12-generic root=/dev/mapper/vg0-root ro possible_cpus=4 swapaccount=0 nodelayacct pcie_aspm=force splash vt.handoff=7
SourcePackage: qemu-kvm
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/20/2011
dmi.bios.vendor: LENOVO
dmi.bios.version: 6IET78WW (1.38 )
dmi.board.name: 2516CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr6IET78WW(1.38):bd05/20/2011:svnLENOVO:pn2516CTO:pvrThinkPadT410:rvnLENOVO:rn2516CTO:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 2516CTO
dmi.product.version: ThinkPad T410
dmi.sys.vendor: LENOVO

Related branches

Simon Déziel (sdeziel) wrote :
Simon Déziel (sdeziel) wrote :

Here is the guest definition extracted from "virsh dumpxml oneiric"

Serge Hallyn (serge-hallyn) wrote :

Thanks for reporting this bug. I can reproduce this even with the latest oneiric version.

Marking priority low since the host affects the guest (rather than guest affecting host), and since it can be worked around.

Changed in qemu-kvm (Ubuntu):
importance: Undecided → Low
status: New → Confirmed
Sergey Svishchev (svs) wrote :

Simon, does the guest start normally after that?

I've seen this and had to reboot the host, because 'virsh start' did not work again ("error: Timed out during operation: cannot acquire state change lock")

Simon Déziel (sdeziel) wrote :

Sergey, here is the steps I've followed :

$ virsh start oneiric
Domain oneiric started

$ virsh setvcpus oneiric --count 2 --live

$ virsh list
 Id Name State
----------------------------------

At this point the guest is dead and I need to start it again :

$ virsh start oneiric
Domain oneiric started

$ virsh vcpucount oneiric
maximum config 2
maximum live 2
current config 1
current live 1

You see that the CPU that was hot plugged is not there anymore. The offline method works though :

$ virsh setvcpus oneiric --count 2 --config

Serge Hallyn (serge-hallyn) wrote :

Ok, sorry, Simon, for taking so long. As it turns out, cpu hotplug is simply not yet supported. It had been implemented in qemu-kvm, but near as I can tell it was dropped when it failed to get into qemu.

However, qemu should at least fail more gracefully, or, failing that, libvirt should be refusing to ask qemu to hotplug cpus. If we can't get this fixed upstream soon, then we should carry a patch for LTS.

Changed in qemu-kvm (Ubuntu):
importance: Low → High
Serge Hallyn (serge-hallyn) wrote :

This is not a bug in qemu.git upstream because cpu_set does not exist there. It is however a bug in qemu-kvm.git.

Serge Hallyn (serge-hallyn) wrote :

We are in soft freeze so I can't upload this now, but I've rolled a fix into the following debdiff.

Serge Hallyn (serge-hallyn) wrote :

Note that with the debdiff above applied to qemu, the virsh setvcpu command doesn't crash the VM any more, but it doesn't give any indication of the failure.

Changed in libvirt (Ubuntu):
status: New → Confirmed
importance: Undecided → Low
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu-kvm - 0.15.0+noroms-0ubuntu7

---------------
qemu-kvm (0.15.0+noroms-0ubuntu7) precise; urgency=low

  * debian/qemu-ifdown: don't use full paths for sbin/ifconfig, especially
    as those paths are wrong. (LP: #898234)
  * debian/qemu-kvm.default and debian/qemu-kvm.upstart: optionally load
    the vhost_net module.
  * debian/patches/dont-try-to-hotplug-cpu.patch: trying to hotplug a cpu
    crashes qemu. So just don't do it! (LP: #878422)
 -- Serge Hallyn <email address hidden> Wed, 30 Nov 2011 11:37:28 -0600

Changed in qemu-kvm (Ubuntu):
status: Confirmed → Fix Released
tags: added: patch
Changed in libvirt (Ubuntu):
status: Confirmed → Invalid
description: updated
description: updated

Hello Simon, or anyone else affected,

Accepted qemu-kvm into oneiric-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in qemu-kvm (Ubuntu Oneiric):
status: New → Fix Committed
tags: added: verification-needed
Simon Déziel (sdeziel) wrote :

Martin, I unfortunately cannot test right now as my laptop runs Natty and I cannot upgrade to Oneiric yet. I'll give that a try as soon as I switch to Oneiric. If this patch lands in natty-proposed, I'll be happy to test this immediately.

Serge Hallyn (serge-hallyn) wrote :

Tests fine for me on oneiric, fwiw.

tags: added: verification-done
removed: verification-needed
Serge Hallyn (serge-hallyn) wrote :

Verified on oneiric.

(qemu) cpu_set 2 online
cpu hotplug not supported
(qemu) quit

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu-kvm - 0.14.1+noroms-0ubuntu6.1

---------------
qemu-kvm (0.14.1+noroms-0ubuntu6.1) oneiric-proposed; urgency=low

  * debian/qemu-ifdown: don't use full paths for sbin/ifconfig, especially
    as those paths are wrong. (LP: #898234)
  * debian/patches/dont-try-to-hotplug-cpu.patch: trying to hotplug a cpu
    crashes qemu. So just don't do it! (LP: #878422)
 -- Serge Hallyn <email address hidden> Tue, 06 Dec 2011 14:57:13 -0600

Changed in qemu-kvm (Ubuntu Oneiric):
status: Fix Committed → Fix Released
Rolf Leggewie (r0lf) wrote :

oneiric has seen the end of its life and is no longer receiving any updates. Marking the oneiric task for this ticket as "Won't Fix".

Changed in libvirt (Ubuntu Oneiric):
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers