Can't reboot kvm virtual machines using virsh

[Marc Tardif]

I'm running Jaunty for amd64 with kvm (1:84+dfsg-0ubuntu11) and libvirt0 (0.6.1-0ubuntu5), which are the latest versions as of today. I have defined a kvm virtual machine which can be started and destroyed correctly. However, this is what I get when I attempt to reboot with the virsh command:

% virsh -c qemu:///system "reboot foo"
Connecting to uri: qemu:///system
error: Failed to reboot domain foo
error: this function is not supported by the hypervisor: virDomainReboot

I can also observe a similar behavior when attempting to define a kvm virtual machine with the action <on_reboot>reboot</on_reboot>. Then, this is what I get when attempting to define with the virsh command:

% virsh -c qemu:///system "define libvirt_run.xml"
Connecting to uri: qemu:///system
error: Failed to define domain from libvirt_run.xml
error: internal error unknown lifecycle action reboot

In the above, note that defining the action <on_reboot>destroy</on_reboot> works correctly. Apparently, I've heard that the reboot action works with the corresponding packages from the ubuntu-virt PPA.

[Itamar (itamar-redhat-bugs)]

Description of problem:

I receive this error when I press shutdown button

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 523, in reboot_domain
    vm.reboot()
  File "/usr/share/virt-manager/virtManager/domain.py", line 569, in reboot
    self.vm.reboot(0)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 392, in reboot
    if ret == -1: raise libvirtError ('virDomainReboot() failed', dom=self)
libvirtError: this function is not supported by the hypervisor: virDomainReboot

Version-Release number of selected component (if applicable):

qemu-0.10-8.fc11.x86_64
virt-manager-0.7.0-4.fc11.x86_64

Expected results:

I like to be able to power off and reset a virtual machine using shutdown button, without receiving error message.

[Mark (mark-redhat-bugs)]

Yep, this is a known issue:

http://www.redhat.com/archives/libvir-list/2008-October/msg00168.html

Moving to upstream bugtracker

[Mark (mark-redhat-bugs)]

Ryan Harper posted a patch to add the qemu side of this:

  http://lists.gnu.org/archive/html/qemu-devel/2009-05/msg00450.html

[Mark (mark-redhat-bugs)]

Okay, moving back to Fedora/rawhide and putting on F12VirtTarget

[Daniel (daniel-redhat-bugs)]

*** Bug 505719 has been marked as a duplicate of this bug. ***

[Dustin Kirkland  (kirkland)]

Mathias Gug reported this same issue in IRC.

I'm marking Confirmed/High.

:-Dustin

[Mark (mark-redhat-bugs)]

AFAICT, the conclusion from upstream was that libvirt should do system_powerdown followed by polling via info status followed by system_reset

[Mark (mark-redhat-bugs)]

*** Bug 524779 has been marked as a duplicate of this bug. ***

[Andreas Ntaflos (daff)]

This bug doesn't seem to have gotten much attention since July last year but I can report that the issue still exists on Karmic using libvirt-bin 0.7.0-1ubuntu13.1:

$ virsh reboot foo1
Connecting to uri: qemu:///system
error: Failed to reboot domain foo1
error: this function is not supported by the hypervisor: virDomainReboot

The stack trace printed by virt-manager says the same:

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 523, in reboot_domain
    vm.reboot()
  File "/usr/share/virt-manager/virtManager/domain.py", line 554, in reboot
    self.vm.reboot(0)
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 398, in reboot
    if ret == -1: raise libvirtError ('virDomainReboot() failed', dom=self)
libvirtError: this function is not supported by the hypervisor: virDomainReboot

Anything we users can or should do here?

[Neil Wilson (neil-aldur)]

Still an issue on lucid. libvirt reports that 'reboot' is not supported by the hypervisor.

[Scott Moser (smoser)]

Per the redhat bug above, this is not a bug so much as a known missing feature.

[kamome (kamome)]

Which I consider a severe bug ;) (and I am affected as well)

[Soren Hansen (soren)]

I really don't consider this High importance at all[1], and I doubt this will get fixed any time soon, if ever. There's simply no portable way (that I can think of, at least) to reboot a fully virtualised operating system, and until there is, such a method cannot be exposed through the libvirt API. Think of it this way: There's no way to walk up to a physical machine and reboot the operating system on there without knowing which operating system it's running and subsequently knowing specifically how to reboot said operating system (if said operating system even /has/ a mechanism for that). That's basically what you're asking libvirt/kvm to be able to do.

[1]: https://wiki.ubuntu.com/Bugs/Importance

[Andrew Bolster (bolster)]

The severity is a matter opinion, but if several users feel it is severe to them, it qualifies as High, as per the provided link.

As for 'no portable way...to reboot...', most (if not all) OS's respect ACPI.

And, in a similar vein, if you can walk up to a physical machine, most physical machines will gracefully shutdown when the power button is pressed, so I don't see your point #6

[Andrew Bolster (bolster)]

Just to add, that concept of 'power down - reset' is apparently what was suggested on the fedora side [1], it just hasnt been implemented.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=496537#c5

[Neil Wilson (neil-aldur)]

Power off via ACPI, if it powers off, power it back on again else
report 'refuse to respond to ACPI shutdown'.

Simples

On 25 April 2010 14:23, Soren Hansen <email address hidden> wrote:
> I really don't consider this High importance at all[1], and I doubt this
> will get fixed any time soon, if ever. There's simply no portable way
> (that I can think of, at least) to reboot a fully virtualised operating
> system, and until there is, such a method cannot be exposed through the
> libvirt API. Think of it this way: There's no way to walk up to a
> physical machine and reboot the operating system on there without
> knowing which operating system it's running and subsequently knowing
> specifically how to reboot said operating system (if said operating
> system even /has/ a mechanism for that). That's basically what you're
> asking libvirt/kvm to be able to do.
>
> [1]: https://wiki.ubuntu.com/Bugs/Importance
>
> --
> Can't reboot kvm virtual machines using virsh
> https://bugs.launchpad.net/bugs/368962
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
Neil Wilson

[Soren Hansen (soren)]

On Sun, Apr 25, 2010 at 01:44:32PM -0000, Andrew Bolster wrote:
> The severity is a matter opinion, but if several users feel it is severe
> to them, it qualifies as High, as per the provided link.

To be perfectly honest, I have a hard time taking it seriously that it's
supposedly a severe impediment to anyone. It's /trivial/ to work around
if you know what OS you're running in the guest. Just do whatever you
usually do to shut it down and start it back up again a little while
later.

"Severe impact for a small number of users" is stuff like a few hundred
users of a particular piece of hardware being unable to boot, for
instance, not 5-6 people who feel mildly inconvenienced by something
that takes them all but a few seconds to work around.

In the grand scheme of things, the importance setting is supposed to
work as a tool for developers to decide which bugs to work on. Across
thousands of packages, it's really not very useful to have stuff like
this ranking the same as people's kernels hanging on boot, system
lock-ups, firefox going all weird on upgrades, or whatever.

> As for 'no portable way...to reboot...', most (if not all) OS's
> respect ACPI.

Certainly not all do. That's the problem. That's what "portable" means.
I'm not even sure Ubuntu supports it. I'm rarely near a machine with a
reset button, so can't check, but at least looking at the acpid package,
I don't see anything that causes the system to reboot in response to
/anything/, and specifically I don't see a handler for any sort of
"reset" event.

> And, in a similar vein, if you can walk up to a physical machine, most
> physical machines will gracefully shutdown when the power button is
> pressed, so I don't see your point #6

It's simple, really. "most" != "all", so it fails the "portable"
requirement.

So, imagine we make libvirt send the acpi shutdown event, wait until the
guest has shut down, and then start it again.. What if the OS never
shuts down when you issue the acpi event? Should the call just never
return? Or should it return immediately and just never succeed in
rebooting the guest?

It's really not as simple as you make it out to be, when you have to
care about edge cases.

--
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/

[Soren Hansen (soren)]

On Sun, Apr 25, 2010 at 02:00:11PM -0000, Neil Wilson wrote:
> Power off via ACPI, if it powers off, power it back on again else
> report 'refuse to respond to ACPI shutdown'.

So you would have the call hang for some amount of time until you decide
that by then it should have shut down and then return failure? How long
would you suggest? Remember, libvirt API calls are not asynchronous, so
this is a concern that needs to be addressed. We can't just have the
call tentatively return succesfully and then later call back into the
calling application telling it that it didn't work out.

For fully virtualised guests, I really believe you'd be better off with
a tool that knows about the guests in question and that can Do The Right
Thing. The Right Thing may be to send a shutdown, wait a while, if
nothing has happened, send a destroy, and then start up the guest again.
libvirt can't (and IMO shouldn't) decide what The Right Thing is for a
particular guest. Some guests may have shutdown sequences that take a
long time, which is perfectly reasonable. Others may not carry state at
all and just need to be destroyed and started again. In other words,
it's virtually impossible make a general decision about a timeout for
this that isn't either too long to be useful in the case of a guest that
has simply hung (and thus will not respond to a reboot request, if it
even has such a mechanism) or too short and causes data loss because a
guest didn't get to shut down properly.

This is the curse (and blessing, depending on your POV) of fully
virtualised virtual machines. Now, we could extend the domain definition
XML to have it define said timeout, that seems clumsy to me. I want to
provide virtual machines to people and not have to fiddle with my
configuration just because someone is doing a lot of stuff on shutdown
and don't want to lose data because I felt like setting a too short
timeout.

> Simples

Sorry, but I beg to differ. If not, I'd have done this a long, long time
ago. I have to deal with this (rebooting guests) on a reasonably regular
basis.

--
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/

[Soren Hansen (soren)]

Oh, by the way, as for "most OS's support ACPI", an Ubuntu Server installation does not do so out of the box. I consider that a bug, by the way, but nevertheless, that is the case and has been years.

So, if this feature depended on the guest OS handling ACPI events at all, it'd sadly fail on Ubuntu.

[Neil Wilson (neil-aldur)]

On 25 April 2010 16:20, Soren Hansen <email address hidden> wrote:
> So you would have the call hang for some amount of time until you decide
> that by then it should have shut down and then return failure? How long
> would you suggest? Remember, libvirt API calls are not asynchronous, so

Pick a number. it's a synchronous call. Where's the 80% case? Anybody
else can use the underlying primitives and poll a bit longer/shorter.

> this is a concern that needs to be addressed. We can't just have the
> call tentatively return succesfully and then later call back into the
> calling application telling it that it didn't work out.

It'll only return successful if a call to the libvirt status element
returns 'off' during the poll period and it was able to successfully
issue 'boot'.

>
> For fully virtualised guests, I really believe you'd be better off with
> a tool that knows about the guests in question and that can Do The Right

Excellent. Where is said tool? Exactly. So why not implement something
that is useful for most people.

>
>> Simples
>
> Sorry, but I beg to differ. If not, I'd have done this a long, long time
> ago. I have to deal with this (rebooting guests) on a reasonably regular
> basis.

Don't you think you are searching for perfection rather than
implementing something that is 'good enough' for the majority of uses?
Get it to log a warning if you're worried about bugs. At the moment
*everybody* has to implement 'shutdown, poll for a bit, startup' (or
would do if the acpi scripts worked). I've just written another one
this week, whereas a default one that hung around for a bit and worked
with a standard Ubuntu server would have been plenty good enough.

--
Neil Wilson

[Soren Hansen (soren)]

On Sun, Apr 25, 2010 at 04:35:25PM -0000, Neil Wilson wrote:
>> So you would have the call hang for some amount of time until you
>> decide that by then it should have shut down and then return failure?
>> How long would you suggest? Remember, libvirt API calls are not
>> asynchronous, so
> Pick a number. it's a synchronous call. Where's the 80% case? Anybody
> else can use the underlying primitives and poll a bit longer/shorter.

So you'd err out instead of falling back to forcibly killing the domain?
Hm. Interesting.

>> this is a concern that needs to be addressed. We can't just have the
>> call tentatively return succesfully and then later call back into the
>> calling application telling it that it didn't work out.
> It'll only return successful if a call to the libvirt status element
> returns 'off' during the poll period and it was able to successfully
> issue 'boot'.

So in the succesful case, the call will take as long as the VM takes to
shut down, and in the failure case, it'd take some amount of time longer
(and in the case of the VM not having ACPI enabled, it'll be
instantaneous). I don't know, really. I'm not super hot on the idea of
function calls that block for that long. Maybe I'm just too used to
writing async code these days. :)

>> For fully virtualised guests, I really believe you'd be better off
>> with a tool that knows about the guests in question and that can Do
>> The Right
> Excellent. Where is said tool?

That's the point. I don't know what's running in your VM's, so I can't
write the tool for you. Chances are that said tool wouldn't even be
using libvirt to do it (in which case libvirt obviously can't (or at
least very much shouldn't)) help you. If an OS doesn't support or
respond correctly to ACPI events, you may want to connect to it over SSH
and issue a command or maybe connect to it over VNC and send it a
ctrl-alt-delete.

>>> Simples
>> Sorry, but I beg to differ. If not, I'd have done this a long, long
>> time ago. I have to deal with this (rebooting guests) on a reasonably
>> regular basis.
> Don't you think you are searching for perfection rather than
> implementing something that is 'good enough' for the majority of uses?

I think I just have different measures of "good enough".

> Get it to log a warning if you're worried about bugs. At the moment
> *everybody* has to implement 'shutdown, poll for a bit, startup' (or
> would do if the acpi scripts worked). I've just written another one
> this week, whereas a default one that hung around for a bit and worked
> with a standard Ubuntu server would have been plenty good enough.

...and as I've said, what you're suggesting would not work with Ubuntu
Server out of the box, so it's a moot point. The only thing I can think
of that would work out of the box on Ubuntu Server is something that
would send ctrl-alt-delete to the guest.

--
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/

[Dustin (dustin-redhat-bugs)]

Any update on the status of this bug? The discussion on the Ubuntu dupe [1] of this bug has been lively, lately.

We're curious if libvirt might simply implement "reboot" by sending "ctrl-alt-delete" to the guest. It seems that most Linux and Windows operating systems will handle ctrl-alt-delete as a warm reboot fairly consistently.

[Dustin (dustin-redhat-bugs)]

Any update on the status of this bug? The discussion on the Ubuntu dupe [1] of this bug has been lively, lately.

We're curious if libvirt might simply implement "reboot" by sending "ctrl-alt-delete" to the guest. It seems that most Linux and Windows operating systems will handle ctrl-alt-delete as a warm reboot fairly consistently.

[1] https://bugs.launchpad.net/bugs/368962

[Daniel (daniel-redhat-bugs)]

I most/many Windows, Ctrl-Alt-Delete is a shortcut to bring up the login dialog, not reboot, so its not really viable.

[Dustin (dustin-redhat-bugs)]

Bummer. I guess it's really been that long since I used Windows. I swear ctrl-alt-delete used to reboot Windows...

[Daniel (daniel-redhat-bugs)]

Win 3.1 & Win 95/98/XP derivatives treat it as a reboot. Win NT and 2k, 2k3, 2k8, Vista, 7, derivatives treat it as a logon signal.

http://en.wikipedia.org/wiki/Control-Alt-Delete#Windows

[Andrew M (andrew-miniatureworldmaker)]

I understand this isn't severe but in the interests of trying to solve a problem would it be possible to define an element in the XML definition which is equivalent to saying 'This domain supports ACPI'. If this flag is set ( it can default to off) then those who know their domain supports acpi use reboot. Would this be an acceptable change ? If so i could try and implement it .I understand libvirt doesn't / can't have a dtd so the barrier for adding elements is low so long as people are comfortable that this would be a positive change.

Andrew

[Thierry Carrez (ttx)]

This is a known missing feature in libvirt/KVM and as such should be Wishlist.

[flickerfly (josiah-ritchie)]

Could someone point me to documentation on how to send an ACPI signal to a vm so I can script my own solution as has been suggested?

Andrew: It appears that the xml already has room for this: http://libvir.org/formatdomain.html#elementsFeatures

I have been able to shutdown by installing acpid package on the guest (not the acpi package) and proven that the shutdown capability can work. It took me awhile to figure that out so I'll just mention that here to save someone time.

[Neil Wilson (neil-aldur)]

virsh shutdown xxxx sends an ACPI shutdown to the VM.

Or you can use the ruby/python bindings if you prefer.

On 5 January 2011 15:36, flickerfly <email address hidden> wrote:
> Could someone point me to documentation on how to send an ACPI signal to
> a vm so I can script my own solution as has been suggested?
>
> Andrew: It appears that the xml already has room for this:
> http://libvir.org/formatdomain.html#elementsFeatures
>
> I have been able to shutdown by installing acpid package on the guest
> (not the acpi package) and proven that the shutdown capability can work.
> It took me awhile to figure that out so I'll just mention that here to
> save someone time.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/368962
>
> Title:
>  Can't reboot kvm virtual machines using virsh
>

--
Neil Wilson

[Daniel (daniel-redhat-bugs)]

commit 42f43592be3045bed5dabdaea65718f6d2c706e4
Author: Daniel P. Berrange <email address hidden>
Date: Wed Jun 15 17:49:58 2011 +0100

    Support reboots with the QEMU driver

    For controlled shutdown we issue a 'system_powerdown' command
    to the QEMU monitor. This triggers an ACPI event which (most)
    guest OS wire up to a controlled shutdown. There is no equiv
    ACPI event to trigger a controlled reboot. This patch attempts
    to fake a reboot.

[Tim Miller Dyck (timmillerdyck)]

As acpid is required in guest VMs for the Upstart pre-stop procedure to work, I'd like to suggest that acpid be included by default in Ubuntu Server virtual machine variants (or all Ubuntu Server default installs, for that matter). This will enable this feature to automatically work as expected in future versions of Ubuntu Server.

I submitted this suggestion here also: http://brainstorm.ubuntu.com/idea/28447/

Thanks,
Tim Miller Dyck

[Neil Wilson (neil-aldur)]

0.9.3 libvirt has native qemu/kvm reboot support baked in now.

[Soren Hansen (soren)]

Yup, but unless you've installed acpid, it'll do exactly nothing. Just FYI.

I'm honestly not sure whether Ubuntu server does that by default these days.

Sent from my phone. Pardon my brevity.
Den 02/09/2011 17.27 skrev "Neil Wilson" <email address hidden>:
> 0.9.3 libvirt has native qemu/kvm reboot support baked in now.
>
> --
> You received this bug notification because you are subscribed to libvirt
> in Ubuntu.
> https://bugs.launchpad.net/bugs/368962
>
> Title:
> Can't reboot kvm virtual machines using virsh
>
> To manage notifications about this bug go to:
>
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/368962/+subscriptions

[Bernhard M. Wiedemann (ubuntubmw)]

I found this nice workaround:
virsh send-key $domain KEY_RIGHTCTRL KEY_RIGHTALT KEY_DELETE

It also helps when VMs are stuck in BIOS

[Serge Hallyn (serge-hallyn)]

Marking this fix released. Guests now by default have acpid installed. If they do not, it is a bug in the client (as they leave no clean way for libvirt to request a clean reboot).