virt-manager freezes when applying hardware change

[Connor Imes]

Binary package hint: virt-manager

I've been trying to decrease my number of CPUs to a XP guest from 2 to 1, but when I go to Apply, virt-manager freezes. The change is not committed to the xml file in /etc/libvirt/qemu/
After closing the frozen windows, I cannot reconnect to qemu:///system from anywhere. I'm not sure if the problem is specifically with virt-manager, perhaps it is with libvirt-bin. Restarting kvm and/or libvirt-bin does not solve the problem, I must reboot.

ProblemType: Bug
Architecture: i386
DistroRelease: Ubuntu 9.04
NonfreeKernelModules: nvidia
Package: virt-manager 0.6.1-1ubuntu4
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: virt-manager
Uname: Linux 2.6.28-11-generic i686

[Michal (michal-redhat-bugs)]

Description of problem:
libvirtd hangs when an attempt to set the number of virtual CPUs for a shut-off KVM guest is made.

Version-Release number of selected component (if applicable):
libvirt-0.6.0-3.fc10.x86_64

How reproducible:
always

Steps to Reproduce:
1. testguest is a previously installed KVM guest domain. It is not currently running.
2. virsh setvcpus testguest 1

Actual results:
virsh hangs. I can kill it CTRL+C. If I then run "virsh", it hangs too. "/etc/init.d/libvirtd restart" brings it back to life.

Expected results:
The number of VCPUs for testguest should be set to the desired value and it should not take longer than a fraction of a second.

Additional info:
virsh setmem and setmaxmem operations work fine, they do not make libvirtd hang.

[Mark (mark-redhat-bugs)]

Confirmed, libvirtd stack trace looks like:

#0 0x00774416 in __kernel_vsyscall ()
#1 0x002dad99 in __lll_lock_wait () from /lib/libpthread.so.0
#2 0x002d6149 in _L_lock_89 () from /lib/libpthread.so.0
#3 0x002d5a52 in pthread_mutex_lock () from /lib/libpthread.so.0
#4 0x04a12bfd in virMutexLock (m=0x85087a8) at threads-pthread.c:51
#5 0x04a28fdd in virDomainObjLock (obj=0x85087a8) at domain_conf.c:3731
#6 0x04a2b065 in virDomainFindByUUID (doms=0x85042f8, uuid=0x8506a64 "�\005�\024�E��B�Z>\234�h��") at domain_conf.c:192
#7 0x08070b12 in qemudDomainGetMaxVcpus (dom=0x8506a50) at qemu_driver.c:2697
#8 0x08070cd6 in qemudDomainSetVcpus (dom=0x8506a50, nvcpus=2) at qemu_driver.c:2527
#9 0x04a22155 in virDomainSetVcpus (domain=0x8506a50, nvcpus=2) at libvirt.c:3981
#10 0x0805ce62 in remoteDispatchDomainSetVcpus (server=0x84fb5a0, client=0x8519980, conn=0x8506d08, rerr=0xb6b70218,
    args=0xb6b702d0, ret=0xb6b70290) at remote.c:1984
#11 0x08060664 in remoteDispatchClientRequest (server=0x84fb5a0, client=0x8519980, msg=0x855abe8) at remote.c:322
#12 0x0805537f in qemudWorker (data=0x8506a90) at qemud.c:1406
#13 0x002d451f in start_thread () from /lib/libpthread.so.0
#14 0x0020a04e in clone () from /lib/libc.so.6

[Daniel (daniel-redhat-bugs)]

Yep, stupid recursive call - one public API is calling into another public API causing it to try & re-acquire the lock it already holds. Some refactoring needed here.

[Daniel (daniel-redhat-bugs)]

Created attachment 334628
Fix recursive locking

[Daniel (daniel-redhat-bugs)]

Posted upstream

http://www.redhat.com/archives/libvir-list/2009-March/msg00195.html

[Marc (marc-redhat-bugs-1)]

*** Bug 489779 has been marked as a duplicate of this bug. ***

[Mark (mark-redhat-bugs)]

Fixed in rawhide with:

* Tue Mar 17 2009 Daniel P. Berrange <email address hidden> - 0.6.1-4.fc11
- Avoid deadlock in setting vCPU count

[Mark Darbyshire (markdarb)]

I'm getting the same problem on amd64. I'm using kvm version 1:84+dfsg-0ubuntu12 from jaunty-proposed, with virt-manager, libvirt-bin, etc all just being the packages from main.

[Connor Imes (ckimes)]

Mark, thank you for confirming this bug's presence. I will mark it so.

[Mark Darbyshire (markdarb)]

Is anyone able to suggest an alternative way to reduce the number of cores used in a virtual machine to 1? Because of Bug #228442 I'm currently having to put up with my virtual machine using 100% of the CPU, so it'd be very nice if I could reduce it down to 1 CPU (which, as I understand it, will stop the virtual machine from hogging more CPU than it needs). Thanks.

[Mark Darbyshire (markdarb)]

The same bug has been reported in Red Hat: https://bugzilla.redhat.com/show_bug.cgi?id=487013

A patch has been posted that fixes it: http://www.redhat.com/archives/libvir-list/2009-March/msg00195.html

"In the QEMu driver, the qemudDomainSetVcpus() entry point is calling into another entry point qemudDomainGetMaxVcpus(), this causes a deadlock due to recursive mutex locking. The qemudDomainSetVcpus() method should in fact be calling the internal method qemudGetMaxVCPUs(). This patch makes it do that, thus avoiding the recursive mutex locking."

I'm assuming that means this bug should be pretty easy to fix. Or is it more complicated than just fixing the code and building a new package?

[Connor Imes (ckimes)]

Mark, to answer your above question from yesterday, you should be able to change the number of cores by editing the xml file that describes the guest. It should be located at /etc/libvirt/qemu/<name>.xml
There should be a line like
   <vcpu>1</vcpu>
where you can change the number of CPUs the guest sees. Make your change and save, then either redefine the xml file (be sure not to undefine!)
   sudo virsh
      # define /etc/libvirt/qemu/<name>.xml
      # quit
substitute it the correct xml of course. Alternatively you should be able to just restart libvirtd, though I'm not sure what effect this may have on running guests:
   sudo /etc/init.d/libvirt-bin restart

The above has worked for me in the past, I hope it helps.
Thanks for finding the Red Hat bug, I'll attach it to this report so we can track it.

[Mark Darbyshire (markdarb)]

I wasn't able to find the file. It turns out for me it's ~/.libvirt/qemu/<name>.xml

Making the change there and restarting the virtual machine worked (it's only a virtual XP desktop, so it by no means needs to be runnig 24/7; neither does my computer for that matter).

Thanks for your help!

[Marc Deslauriers (mdeslaur)]

This has been fixed in more recent versions of libvirt such as the one in Karmic or Lucid. I am setting this bug as "Fix Released". Please feel free to re-open it if you can reproduce the issue in Lucid.

Thanks.