virsh destroy might also kill another running VM

[Thierry Carrez]

Jaunty KVM amd64 host running:
  libvirt 0.6.1-0ubuntu4
  kvm 1:84+dfsg-0ubuntu10

I'm starting multiple VMs (using virsh start or virt-manager)
Then calling "virsh destroy" on the first one in the list also destroys the second one:

ttx@cassini:~$ virsh list
Connecting to uri: qemu:///system
 Id Name State
----------------------------------
  5 intrepid-test running
  6 jaunty-devel running

ttx@cassini:~$ virsh destroy intrepid-test
Connecting to uri: qemu:///system
Domain intrepid-test destroyed

ttx@cassini:~$ virsh list
Connecting to uri: qemu:///system
 Id Name State
----------------------------------

ttx@cassini:~$

Running strace on the-one-that-shouldn't-be-killed reveals it receives SIGKILL.

Test scenarios:
Starting A, Starting B, virsh destroy A -> A and B get destroyed
Starting B, Starting A, virsh destroy B -> A and B get destroyed
Starting A, Starting B, virsh destroy B -> only B is destroyed
Starting A, Starting B, Starting C, virsh destroy A -> A and B get destroyed
Starting A, Starting B, Starting C, virsh destroy B -> B and C get destroyed

I can reproduce it on freshly created VMs stuck at "select boot device" BIOS stage (I hit F12) so it probably doesn't depend on the guest nature.

Using "Force off" from virt-manager also triggers the issue.

[Andrew Bolster (bolster)]

Possibly related: when attempting to reboot vm's from the Virtual Machine Manager windows causes this complaint:

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 528, in reboot_domain
    vm.reboot()
  File "/usr/share/virt-manager/virtManager/domain.py", line 504, in reboot
    self.vm.reboot(0)
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 392, in reboot
    if ret == -1: raise libvirtError ('virDomainReboot() failed', dom=self)
libvirtError: this function is not supported by the hypervisor: virDomainReboot

[Andrew Bolster (bolster)]

Forgot to mention, same arch, same version of kvm and libvirt 0.6.1-0ubuntu4, tested scenarios listed above with and without guest os's installed, all confirmed

[Thierry Carrez (ttx)]

Bolster: yes, I also experience that, however this is a different bug. Could you please open a separate bug about it ? Thanks !

[Pekka Jääskeläinen (pekka-jaaskelainen)]

I'm also experiencing this. This issue combined with a strange randomish guest freezing problem (with a Debian Lenny guest) makes KVM unusable in the Jaunty host for me.

[Will (will-redhat-bugs)]

Sometimes, using 'virsh destroy' or the 'Force off' button in virt-manager will cause multiple running VMs to be destroyed. Here's an example:

[wwoods@metroid ~]$ sudo virsh list --all
 Id Name State
----------------------------------
  1 Ubuntu_Jaunty running
  4 Fedora_10_clone running
  5 F10_2 running
  6 F10 running

[wwoods@metroid ~]$ sudo virsh destroy F10_2
Domain F10_2 destroyed

[wwoods@metroid ~]$ sudo virsh list --all
 Id Name State
----------------------------------
  1 Ubuntu_Jaunty running
  4 Fedora_10_clone running
  - F10 shut off
  - F10_2 shut off

Note that 'F10' is now also shut off, even though I didn't destroy it. This doesn't seem to happen every time, and it doesn't seem to be related to the names of the hosts being similar:

[wwoods@metroid ~]$ sudo virsh list --all
 Id Name State
----------------------------------
  1 Ubuntu_Jaunty running
  4 Fedora_10_clone running
  8 F10_2 running
  - F10 shut off

[wwoods@metroid ~]$ sudo virsh start F10
Domain F10 started

[wwoods@metroid ~]$ sudo virsh destroy Fedora_10_clone
Domain Fedora_10_clone destroyed

[wwoods@metroid ~]$ sudo virsh list --all
 Id Name State
----------------------------------
  1 Ubuntu_Jaunty running
 10 F10 running
  - F10_2 shut off
  - Fedora_10_clone shut off

There are no relevant messages in syslog, other than the expected ones (e.g. "kernel: virbr0: port 3(vnet2) entering disabled state" as the host comes down).

[Daniel (daniel-redhat-bugs)]

Can you run 'strace -f -p $PID-OF-LIBVIRTD -s 1000 -ff -o' and then try and reproduce the destroy problem.

Also, can you turn on full debug logging of libvirtd & capture the results http://libvirt.org/logging.html

[Mark (mark-redhat-bugs)]

Nasty. I've asked wwoods for a libvirtd log ala:

https://fedoraproject.org/wiki/Reporting_virtualization_bugs#libvirt

[Will (will-redhat-bugs)]

Created attachment 342908
very verbose log from libvirtd

This is the full log from libvirtd with log_level set to 1. It follows these basic steps:

service libvirtd restart
virsh list --all
 Id Name State
----------------------------------
  1 Ubuntu_Jaunty running
 13 F10 running
  - F10_2 shut off
  - F10_RAID shut off
  - F9 shut off
  - Fedora_10_clone shut off
  - Rawhide shut off

virsh start F10_2
virsh start Fedora_10_clone
virsh list --all
virsh destroy F10_2
virsh list --all
virsh start F10_2
virsh destroy F10
virsh list --all
virsh start F10
virsh list --all
virsh list --all
virsh start F10_2
virsh list --all
# problem is triggered here - F10 dies as well
virsh destroy F10_2
virsh list --all
 Id Name State
----------------------------------
  1 Ubuntu_Jaunty running
 15 Fedora_10_clone running
  - F10 shut off
  - F10_2 shut off
  - F10_RAID shut off
  - F9 shut off
  - Rawhide shut off

service libvirtd stop

Hope you can make some sense of it - it's 4MB uncompressed.

[Mark (mark-redhat-bugs)]

Interesting, you only issued a destroy for F10_2, but yet:

14:12:43.927: debug : virDomainDestroy:1750 : domain=0x7fb270001270
14:12:43.927: debug : qemudShutdownVMDaemon:1518 : Shutting down VM 'F10_2'

14:12:43.927: debug : virEventRemoveHandleImpl:165 : Remove handle 20
14:12:43.927: debug : virEventRemoveHandleImpl:172 : mark delete 11 24
14:12:43.983: debug : virEventRunOnce:544 : Poll got 1 event
14:12:43.983: debug : virEventDispatchHandles:416 : Skip deleted 24
14:12:43.983: debug : virEventDispatchHandles:425 : Dispatch 24 32 0x13f7c00
14:12:44.036: info : Setting SELinux context on '/var/lib/libvirt/images/F10_2.img' to 'system_u:object_r:virt_image_t:s0'
14:12:44.036: debug : virEventUpdateTimeoutImpl:233 : Updating timer 0 timeout with 0 ms freq
14:12:44.036: debug : qemudShutdownVMDaemon:1518 : Shutting down VM 'F10'

[Daniel (daniel-redhat-bugs)]

This is an event loop dispatcher bug. The destroy call is killing the QEMU process, so we then get a HANGUP event on the FD associated with the guest monitor. The callback for this is already marked as deleted in the event loop though, so it gets skiped, and then we mistakenly dispatch the next callback in the loop, causing us to think another VM has died, and trigger cleanup of that guest

[Daniel (daniel-redhat-bugs)]

*** Bug 499788 has been marked as a duplicate of this bug. ***

[Daniel (daniel-redhat-bugs)]

Created attachment 343109
Fix event loop handling of deletes & test functionality

THis patch fixes the event loop handling of deletes and adds a test case which validates the various important scenarios actually work

[Daniel (daniel-redhat-bugs)]

*** Bug 500089 has been marked as a duplicate of this bug. ***

[Jamie Strandboge (jdstrand)]

I too see this. Attached is the XML definitions for two affected machines.

[Jamie Strandboge (jdstrand)]

Interestingly, as Thierry pointed out, if I run 5 VMs sec-(dapper|hardy|intrepid|jaunty|karmic)-i386, and start them in the order I listed, and do 'virsh destroy sec-intrepid-i386', then only sec-jaunty-i386 gets additionally destroyed. dapper, hardy and karmic all stay up. I tried this several times and the two machines that are destroyed are always the one specified and the one started immediately after it.

[Fedora (fedora-redhat-bugs)]

libvirt-0.6.2-9.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libvirt-0.6.2-9.fc11

[Mark (mark-redhat-bugs)]

I've pushed libvirt-0.6.2-9.fc11 to updates-testing with Dan's fix. Please test and update the update's karma using the link above

* Thu May 21 2009 Mark McLoughlin <email address hidden> - 0.6.2-9.fc11
- Fix qemu argv detection with latest qemu (bug #501923)
- Fix XML attribute escaping (bug #499791)
- Fix serious event handling issues causing guests to be destroyed (bug #499698)

[Justin Huff (jjhuff)]

Upstream bug:
https://bugzilla.redhat.com/show_bug.cgi?id=499698

Looks like Fedora has a patch:
https://bugzilla.redhat.com/show_bug.cgi?id=499698#c7

[Woo (w-digmia)]

You seriously mean that importance of this bug is medium?

[Fedora (fedora-redhat-bugs)]

libvirt-0.6.2-9.fc11 has been pushed to the Fedora 11 testing repository. If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with
 su -c 'yum --enablerepo=updates-testing update libvirt'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-5311

[Fedora (fedora-redhat-bugs)]

libvirt-0.6.2-10.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libvirt-0.6.2-10.fc11

[Michal Ingeli (xyzz)]

Severity high, also according to upstream. This flaw makes libvirt practically unusable.

[Andrew Bolster (bolster)]

has anyone confirmed if the fedora patch noted above actually works?

2009/5/22 Michal Ingeli <email address hidden>

> Severity high, also according to upstream. This flaw makes libvirt
> practically unusable.
>
> --
> virsh destroy might also kill another running VM
> https://bugs.launchpad.net/bugs/362288
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[Andrew Bolster (bolster)]

I cannot find any feedback regarding the fedora fix noted by JH, has been put into unstabletesting but no responders. Has anyone else tried this? Will test tonight and feedback here and on bz

[Fedora (fedora-redhat-bugs)]

libvirt-0.6.2-10.fc11 has been pushed to the Fedora 11 testing repository. If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with
 su -c 'yum --enablerepo=updates-testing update libvirt'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-5441

[Fedora (fedora-redhat-bugs)]

libvirt-0.6.2-11.fc11 has been pushed to the Fedora 11 testing repository. If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with
 su -c 'yum --enablerepo=updates-testing update libvirt'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-5515

[Michal Ingeli (xyzz)]

I can confirm, patch worked.

Attaching ported patch from fedora, if it's not already done.

[Fedora (fedora-redhat-bugs)]

libvirt-0.6.2-11.fc11 has been pushed to the Fedora 11 stable repository. If problems still persist, please make note of it in this bug report.

[Fedora (fedora-redhat-bugs)]

libvirt-0.6.2-12.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libvirt-0.6.2-12.fc11

[Radovan Pútec (radko)]

this is not funny anymore. this issue should be fixed immediately.

[Joshua Timberman (jtimberman)]

I don't see any updates to the libvirt package. Is this package updated, or will it be soon?

[Simon Morvan (simon-icilalune)]

Any updates ?

[David Sedeño Fernandez (david-alderia)]

I'm trying to update the package with the patch, but debuild doesn't work. This patch affects other.:

 Patch event-loop-hang.diff does not apply

[Oliver Siegmar (osiegmar)]

Please note, that also a 'virsh save domain' destroys other domains. Any updates here?

[Bryan McLellan (btm)]

Michal Ingeli's <a href="http://launchpadlibrarian.net/27207697/lp-362288.patch">patch</a> did not resolve the issue for me after converted by hand to quilt (attached) and applied to the patch series in libvirt=0.6.1-0ubuntu5 on jaunty.

Destroyed all guests and libvirt
Started libvirt
Started three guests
Destroyed the first guest, which consequently destroyed the second.

Perhaps my conversion is bunk. Uploading libvirt=0.6.1-0ubuntu6~btm1 to <a href="https://launchpad.net/~btm/+archive/ppa/+sourcepub/669470/+listing-archive-extra">my ppa</a> if someone wants to take a look.

Michal, what source did you apply your patch to where the problem was resolved for you? As was <a href="https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/362288/comments/17">suggested by David</a> the patch doesn't apply cleanly to the ubuntu source for libvirt=0.6.1-0ubuntu5.

[Bryan McLellan (btm)]

Upstream patch [1] works against 0.6.1-0ubuntu5 on jaunty.

I'm attaching this patch converted to quilt sans the test code, it applies fine as the last in the series.

Testers should grab libvirt=0.6.1-0ubuntu6~btm2 from my ppa [2].

Destroyed all guests and libvirt
Started libvirt
Started three guests
Destroyed the first guest, no other guests were destroyed.

[1] https://bugzilla.redhat.com/attachment.cgi?id=343109
[2] https://launchpad.net/~btm/+archive/ppa

[Bryan McLellan (btm)]

Upstream patch [1] was checked into source control [2] in 72dc6d + 0a31be + 37ede4, which are in libvirt v0.6.4. Current version of libvirt in karmic is 0.6.4-1ubuntu2, so this bug should not be present in karmic.

[1] https://bugzilla.redhat.com/attachment.cgi?id=343109
[2] git://git.et.redhat.com/libvirt.git

[Oliver Siegmar (osiegmar)]

Bryan, I couldn't apply the event-handling-fix-rh499698-lp362288.patch against libvirt_0.6.1-0ubuntu5 -

$ quilt push
Applying patch event-handling-fix-rh499698-lp362288.patch
patching file libvirt-0.6.1/qemud/event.c
Hunk #12 FAILED at 429.
Hunk #13 succeeded at 542 (offset -1 lines).
Hunk #14 succeeded at 568 (offset -1 lines).
Hunk #15 succeeded at 612 (offset -1 lines).
Hunk #16 succeeded at 631 (offset -1 lines).
1 out of 16 hunks FAILED -- rejects in file libvirt-0.6.1/qemud/event.c
patching file libvirt-0.6.1/qemud/qemud.c
patching file libvirt-0.6.1/qemud/qemud.h
Patch event-handling-fix-rh499698-lp362288.patch does not apply (enforce with -f)

What am I missing?

[Dustin Kirkland  (kirkland)]

Excellent, thanks Bryan.

Ping me on IRC and I can walk you through the SRU process.

:-Dustin

[Bryan McLellan (btm)]

There are a lot of patches in this package, so order matters a bit.

apt-get source libvirt=0.6.1-0ubuntu5
wget http://launchpadlibrarian.net/29013416/event-handling-fix-rh499698-lp362288.patch
cd libvirt-0.6.1/
QUILT_PATCHES=debian/patches/ quilt push -a
QUILT_PATCHES=debian/patches/ quilt import ../event-handling-fix-rh499698-lp362288.patch
QUILT_PATCHES=debian/patches/ quilt push

[Oliver Siegmar (osiegmar)]

Thank's Bryan! Patch is installed and works great (tested save, restore and destroy) so far.

[Bryan McLellan (btm)]

Jaunty SRU Notes:

This bugs causes a second guest to be erroneously destroyed when the user destroys a guest. Depending on the function of the second guest, this can lead to unexpected data loss and service interruption and is inherently very destructive.

Because this bug has been fixed upstream in libvirt v0.6.4, which is in karmic, this bug does not apply to the current development release.

A quilt patch [1] has been prepared from the upstream patch [2] with the test code excluded for minimalistic purposes. This patch should applied last in the series in libvirt=0.6.1-0ubuntu5.

TEST CASE:

Create three guests, start them with 'virsh state GUEST' and verify that they are running with 'virsh list'
Destroy the first guest with 'virsh destroy GUEST'
Notice that two guests are in fact destroyed with 'virsh list'

This upstream patch is almost entirely sanity checks and would not likely cause regressions.

[1] http://launchpadlibrarian.net/29013416/event-handling-fix-rh499698-lp362288.patch
[2] https://bugzilla.redhat.com/attachment.cgi?id=343109

[Bryan McLellan (btm)]

Setting importance to critical because this is such a destructive bug. Awaiting Soren to continue the SRU process.

[Dustin Kirkland  (kirkland)]

Thierry,

You originally reported this bug... Can you verify that Bryan's
proposed patch fixes the issue for you?

If so, I think we can move this SRU along.

:-Dustin

[Thierry Carrez (ttx)]

Thanks Dustin for pushing this one.
Yes, I can confirm that Bryan's patched version fixes the issue for me. I tested libvirt=0.6.1-0ubuntu6~btm2 from Bryan's PPA. I also confirm that the problem is already fixed in karmic.

[Dustin Kirkland  (kirkland)]

Okay, I've uploaded the package to jaunty-proposed.

Martin, could you take a look at that and accept it to -proposed at your convenience?

There seems to be a lot of people experiencing this bug. Please, by all means, help us test this package when it lands in -proposed.

:-Dustin

[Martin Pitt (pitti)]

Accepted libvirt into jaunty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Bryan McLellan (btm)]

libvirt-bin=0.6.1-0ubuntu5.1 from jaunty-proposed passes the SRU test case and acts as expected for me.

[Thierry Carrez (ttx)]

Confirmed fixed as well in jaunty-proposed.

[Cody Herriges (ody-cat)]

Also confirming fixed with jaunty-proposed packages.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 0.6.1-0ubuntu5.1

---------------
libvirt (0.6.1-0ubuntu5.1) jaunty-proposed; urgency=low

  * debian/patches/event-handling-fix-rh499698-lp362288.patch: cherry pick
    from upstream; fixes critical, destructive bug whereby calling
    'virsh destroy foo' kills foo, plus another vm; LP: #362288

 -- Dustin Kirkland <email address hidden> Thu, 16 Jul 2009 18:45:28 -0500