Should shut down domains on system shutdown

[Alexandre Kandalintsev]

===========================
SRU Justification
1. Impact: libvirt VMs must be manually shut down before host shutdown, else they will be corrupted.
2. How bug was addressed: the libvirt upstart job will now try to shut down all VMs before the host reboots or shuts down
3. patch: see the linked bzr tree (or https://code.launchpad.net/~serge-hallyn/ubuntu/oneiric/libvirt/fix-shutdown/+merge/70245/+preview-diff/+files/preview.diff)
4. TEST CASE: start a KVM vm under libvirt, then reboot the host.
5. Regression potential: shutting down a host can now be delayed until the VMs shut down or the shutdown attempts time out. This should be better than having VMs corrupted, and won't happen if VMs are not running; but could conceivably be seen as a regression.
===========================

I'm using autostart feature of libvirtd (my guest defined in /etc/libvirt/qemu/autostart/). It's starts normally, but if I reboot server then guest fs will be corrupted becouse its cant stop in time:

WARNING: / was not properly dismounted

We need to give more time to libvirtd to stop all guests.

I'm running jaunty amd64 with all updates.

[Anthony Liguori (anthony-codemonkey)]

This is not a KVM issue but rather a libvirt issue. libvirt should delay shutdown until the guests have fully shut down.

[Chuck Short (zulcss)]

Hi,

With which version of libvirt is this with?

Regards
chuck

[Alexandre Kandalintsev (spam-messir)]

I used jaunty version of libvirt. Now I'm on debian so I can't say you what exactly version it was.

Are you shure it's a nessesary information? Does libvirt rc-script have any ability to wait for virtual maschines at all?

[Dustin Kirkland  (kirkland)]

I can hack this into the upstart script. There will need to be a configurable timeout, though, on how long to delay shutdown. I suggest 5 minutes.

[Alexandre Kandalintsev (spam-messir)]

I think 5 minutes is very long for servers... I suggest two minutes.

[Alessandro Bono (a.bono)]

Hi

Any news for this bug? I want to convert my xen server to libvirt+kvm but this bug it's really serious for me, it's not possible to automate shutdown (ups or time based) or simply "press power button to clean shutdown" without important modification to libvirt upstart script. Any chance to update upstart script prior to lucid announce??

thanks

[Francesco Pretto (ceztko)]

I'm not sure it can be done hacking upstart job. When libvirt was a sysvinit script in ubuntu 9.10, I used this [1] script adding it in "stop" function. In Lucid, if I add the same script to /etc/init/libvirt-bin.conf with the stanza:

pre-stop exec shut-guests.sh 2>&1 > /var/log/shut-guests.log

This works if I manually invoke a stop of libvirt-bin in a running system with:
$ sudo stop libvirt-bin

but **doesn't** work when issuing a reboot or halt. It seems there's an aggressive killing of all kvm processes **before** the pre-stop stanza is execute, and libvirt doesn't find any running guests when the shut-guests.sh is executed. it's not clear to me why and If this is an exptected behavior. For sure, kvm processes are detached from libvirt process branch.

[1] http://www.linux-kvm.com/content/stop-script-running-vms-using-virsh

[Francesco Pretto (ceztko)]

Workaround:

Add the lines:

        /usr/local/bin/shut-guests.sh 2>&1 >> /var/log/shut-guests.log
        /sbin/initctl emit guests-shutted

just after "do_stop() {" in /etc/init.d/sendsings

*and* changing the line:

stop on (runlevel [!2345])

on file /etc/init/libvirt-bin.conf to:

stop on guests-shutted

Ugly but works, until libvirt/kvm/distributions crews will care more about data integrity in vm guests during shutdown (sigh).

[Christopher Hlubek (hlubek)]

This is a very important issue for server virtualization. If Ubuntu wants to be en par with Xen on other Distributions this bug has to be fixed!

[Soren Hansen (soren)]

Note to anyone implementing this (maybe I'll get around to it myself, but not right now):

It's important to remember that libvirt intentionally does not kill VM's on termination. This enables us to upgrade libvirt without interruping running VM's, so we want to only conditionally shut down VM's. What I've done on one of my Hardy boxes is to check $0 to see if the init script was invoked as K??libvirt-bin. If so, I assume we're being shut down and then I go and shut down the virtual machines. If invoked any other way, it just stops libvirt, not the VM's. To achieve something similar with upstart jobs, there's an environment variable that tells us why libvirt-bin is being stopped. This will reveal whether it's being shut down as a result of shutdown being called or whatnot.

[Francesco Pretto (ceztko)]

2010/5/5 Soren Hansen <email address hidden>:
>
> It's important to remember that libvirt intentionally does not kill VM's
> on termination. This enables us to upgrade libvirt without interruping
> running VM's, so we want to only conditionally shut down VM's.

Correct.

> What I've
> done on one of my Hardy boxes is to check $0 to see if the init script
> was invoked as K??libvirt-bin. If so, I assume we're being shut down and
> then I go and shut down the virtual machines. If invoked any other way,
> it just stops libvirt, not the VM's. To achieve something similar with
> upstart jobs, there's an environment variable that tells us why libvirt-
> bin is being stopped. This will reveal whether it's being shut down as a
> result of shutdown being called or whatnot.
>

But the problem is that the PIDs of the vm are not part of the libvirt-bin upstart job. Try to look my *ugly* workaround[1] :
/etc/init.d/sendsings is just parallelizing killing any process not in upstart pool of tracked ones. So, when upstart shutdown libvirt-bin, almost for sure libvirt-bin won't find any active guests because they have been killed before by /etc/init.d/sendsing (killing those vm processes seems very quick). While this is good for libvirt-bin independence from kvm guests processes, this make fine grain control of guests shutdown harder, and one may wonder if it's not kvm itelsef that should send ACPI shutdown signal to guests, and not libvirt.

I have another suspect: upstart may failing here. Doesn't libvirt-bin forks here to create kvm guests? Shouldn't "expect daemon" upstart stanza track daemons that fork more than once? If so, shouldn't libvirt-bin upstart job track these guests processes too? I have to ask in upstart mailing list.

[1] https://bugs.launchpad.net/ubuntu/+source/kvm/+bug/350936/comments/8

[Reinhard Tartler (siretart)]

How about adding a pre-stop script to /etc/init/libvirt-bin.conf that uses /usr/bin/virsh to iterate over all running guests to shut them down? Probably this should be done a) in parallel and b) supervised by reasonable timeouts. So technically, it might make sense to implement that pre-stop script in C or python to access libvirt directly, or extend /usr/bin/virsh to offer an appropriate command.

[Reinhard Tartler (siretart)]

ignore my previous comment, I should have read the whole thread in more detail.

[ossjunkie (ossjunkie)]

as this bug can corrupt guests i did an old school init script for my server, but that doesn't work as libvirt-bin got always stopped before (even at S01). so i removed all bashism like arrays and tried to create an upstart script. the "core logic" is done, but i need some help regarding upstart integration:

-how can we ensure in upstart that the script got run on shutdown & reboots but always before libvirt-bin and qemu-kvm got stopped?

-should we use return or exit in upstart scripts?

feedback highly welcome :)

BTW better output to screen and a log file wouldn't hurt

[Soren Hansen (soren)]

On Wed, Jun 23, 2010 at 03:53:18PM -0000, ossjunkie wrote:
> -how can we ensure in upstart that the script got run on shutdown &
> reboots but always before libvirt-bin and qemu-kvm got stopped?

I /think/ this should do the trick:

start on shutdown and stopping libvirt-bin

IIUIC, libvirt-bin will block its stop procedure until your script is
done.

--
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/

[ossjunkie (ossjunkie)]

thanks for the hint, but after trying i found that "start on (runlevel [06] and stopping libvirt-bin)" should do the trick. but i still got problems as i can use libvirt at first in the upstart job, but when testing the timeout loop it seems that libvirt-bin still stopped before the script finished.

why does libvirt-bin still stop before the libvirt-shutdown-guests job has finished the script?
do i have to modify the libvirt-bin job to wait for libvirt-shutdown-guests?
does upstart jobs also receive some sort of kill signal?
does upstart in general allow to delay the shutdown just by a loop in the script of a upstart job?

here is my current version. upstart experts required urgently ;)

BTW i have no problems when testing it with "initctl start libvirt-shutdown-guests", so any testing needs to done on real shutdown ;(

[Soren Hansen (soren)]

On Thu, Jun 24, 2010 at 01:38:04PM -0000, ossjunkie wrote:
> thanks for the hint, but after trying i found that "start on (runlevel
> [06] and stopping libvirt-bin)" should do the trick. but i still got
> problems as i can use libvirt at first in the upstart job, but when
> testing the timeout loop it seems that libvirt-bin still stopped before
> the script finished.

That shouldn't happen. You're doing the right thing with upstart. I
double checked with one of the experts. :) Something else seems to be
killing libvirt-bin. Try putting a a post-stop thing in the libvirt-bin
job to log when it's being killed, just to double check.

> why does libvirt-bin still stop before the libvirt-shutdown-guests job
> has finished the script?

Good question. Upstart shouldn't be killing it, but something else
might. I can't imagine what, though.

> do i have to modify the libvirt-bin job to wait for
> libvirt-shutdown-guests?

Nope.

> does upstart jobs also receive some sort of kill signal? does upstart
> in general allow to delay the shutdown just by a loop in the script of
> a upstart job?

Yes.

> BTW i have no problems when testing it with "initctl start libvirt-
> shutdown-guests", so any testing needs to done on real shutdown ;(

> ** Attachment added: "libvirt-shutdown-guests.conf"
> http://launchpadlibrarian.net/50850625/libvirt-shutdown-guests.conf

The script still has some debugging stuff in it. That would need to go
away before we can include this in the package.

Thanks for your efforts so far! This has been a problem for a looong
time.

--
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/

[Francesco Pretto (ceztko)]

2010/6/24 ossjunkie <email address hidden>:
> why does libvirt-bin still stop before the libvirt-shutdown-guests job has finished the script?

Because kvm guests processes aren't tracked by the upstart job and
they are killed immediately by /etc/init.d/sendsings. This approach
have pros (independence of libvirt from guests) and cons (this
problem) so there's no easy fix. In my previous post [1] there's an
ugly but safe workaround. If you want a definitive solution try to bug
libvirt devs by pointing this exact problem (/etc/init.d/sendsings
killing guests before libvirt-bin upstart job finishes).

[1] https://bugs.launchpad.net/ubuntu/+source/kvm/+bug/350936/comments/8

[Francesco Pretto (ceztko)]

2010/6/24 Soren Hansen <email address hidden>:
>> why does libvirt-bin still stop before the libvirt-shutdown-guests job
>> has finished the script?
>
> Good question. Upstart shouldn't be killing it, but something else
> might. I can't imagine what, though.
>

I'm not sure Upstart isn't supposed to kill them. Let's suppose
upstart is able to track all pids forked for livbvirt-bin job start
(and it doesn't do so, even if the "expect daemon" stanza is used:
this may be an upstart bug): there would be a pid for libvirt-bin and
many pids for kvm guests. Suppose you want to kill the libvirt-bin
job: how can you tell upstart to kill just libvirt-bin and not kvm
guests? I don't think upstart jobs are so flexible.

[Soren Hansen (soren)]

On Thu, Jun 24, 2010 at 08:04:48PM -0000, Francesco Pretto wrote:
>>> why does libvirt-bin still stop before the libvirt-shutdown-guests
>>> job has finished the script?
>> Good question. Upstart shouldn't be killing it, but something else
>> might. I can't imagine what, though.
> I'm not sure Upstart isn't supposed to kill them.

Not "them" (as in the kvm guests), but libvirtd itself. Besides, of
course upstart is meant to kill it (or shut it down by some other
means), but it shouldn't be doing it until all tasks that are triggered
on "stopping libvirt-bin" have completed.

> Let's suppose upstart is able to track all pids forked for
> livbvirt-bin job start (and it doesn't do so, even if the "expect
> daemon" stanza is used: this may be an upstart bug): there would be a
> pid for libvirt-bin and many pids for kvm guests. Suppose you want to
> kill the libvirt-bin job: how can you tell upstart to kill just
> libvirt-bin and not kvm guests? I don't think upstart jobs are so
> flexible.

I'm reasonably sure upstart only tracks one pid per upstart job, and as
such has no notion of the started kvm guests.

--
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/

[Soren Hansen (soren)]

Why are you not doing this in the pre-stop part of the libvirt-bin job,
by the way?

[ossjunkie (ossjunkie)]

@soren: as i simple don't know how to trigger it only on shutdown, as the premise was it should run on regular stop. but i know it may ease the situation and would like to do so. do you know some solution here. maybe we could do a simple condition that checks for an enviromental stuff thats only present on shutdown/reboot and is a reliable source. and yes i left some ugly debugging stuff in, but because it isn't ready so far ;)

regarding the killing by /etc/init.d/sendsings we could easily test when the VMs got killed and how to prevent when libvirtd would be still accessable. so i would say we should get the upstart things right first and then go on with that. maybe we could try to track and bound the pids of the guest somehow to the upstart job (in case we keep it seperate) and prevent it that way.

[Soren Hansen (soren)]

On Fri, Jun 25, 2010 at 10:32:07AM -0000, ossjunkie wrote:
> @soren: as i simple don't know how to trigger it only on shutdown, as
> the premise was it should run on regular stop. but i know it may ease
> the situation and would like to do so. do you know some solution here.
> maybe we could do a simple condition that checks for an enviromental
> stuff thats only present on shutdown/reboot and is a reliable source.
> and yes i left some ugly debugging stuff in, but because it isn't
> ready so far ;)

I actually had a modified version of libvirt's init script that did what
you're doing. It checked whether $0 was called K??libvirt-bin, in which
case it was being called as part of a runlevel change. The equivalent in
the world of Upstart is probably something like the UPSTART_EVENTS
environment variable. It will not be set if you're running "stop
libvirt-bin", but will be set if the libvirt-bin job is being stopped as
a result of an Upstart event.

> regarding the killing by /etc/init.d/sendsings we could easily test
> when the VMs got killed and how to prevent when libvirtd would be
> still accessable. so i would say we should get the upstart things
> right first and then go on with that. maybe we could try to track and
> bound the pids of the guest somehow to the upstart job (in case we
> keep it seperate) and prevent it that way.

libvirt makes sure qemu creates pidfiles for kvm processes anyway. It
shouldn't be hard at all to make sendsigs omit them from its killing
spree.

--
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/

[tomcrus (tomcrus)]

As I'm currently setting up an intranet mail-server for my company (based on lucid / ebox) in some kvm-guests and there for have exactly the same problem. Reading all previous posts - and because I need kvm-guest shutdown as well - I will use something like this:

Add following lines in /etc/init.d/sendsigs just before the sync line:

  # avoid every running kvm (started thru libvirt-bin) from being killed
  for pidfile in /var/run/libvirt/qemu/*.pid; do
    OMITPIDS="${OMITPIDS:+$OMITPIDS }-o $(cat $pidfile)"
  done

I think this should prevent kvm-guests from being killed on shutdown.
Then in /etc/init/libvirt-bin.conf I would try to add following lines:

  kill timeout 360

  pre-stop script
    # only shutdown all guests if stopping of task is because of
    # runlevels 0 (halt) or 6 (reboot)
    if [ "$RUNLEVEL" = "0" -o "$RUNLEVEL" = "6" ]; then
      TIMEOUT=300
      VIRSH_CONNECT="qemu:///system"

      function list_running_vms(){
        virsh --connect $VIRSH_CONNECT list | grep running | awk '{ print $2 }'
      }

      # send shutdown to each running vm (they must handle acpi-power-button!)
      list_running_vms | while read vm; do
        virsh --connect $VIRSH_CONNECT shutdown $vm
      done

      # wait until timeout has reached or no running are vms left
      END_TIME=$(date -d "$TIMEOUT seconds" +%s)
      while [ $END_TIME -lt $(date +%s) ]; do
        test -z "$(list_running_vms)" && break
        sleep 1
      done

      # if any vms left running destroy them now
      if [ -n "$(list_running_vms)" ]; then
        list_running_vms | while read vm; do
          virsh --connect $VIRSH_CONNECT destroy $vm
        done
      fi
      sleep 3
    fi
  end script

I don't know yet if this works, but I will test as soon as possible. Maybe if somebody else is in the mood to do so please report if it succeeded...

[ossjunkie (ossjunkie)]

i reworked the changes by tomcrus (thanks for the elegance ;) and removed all reintroduced bashism. so here is the current of the "core logic" to shutdown guests.

[ossjunkie (ossjunkie)]

i have also tested the hack to /etc/init.d/sendsigs

[ossjunkie (ossjunkie)]

but i am still hanging on the upstart integration when using pre-stop in libvirt-bin. while the condition regarding the runlevel by tomcrus works great, it seems we are not able to simply delay the shutdown process with a simple sleep. in my case i only managed to get the pre-stop killed or make the reboot hang forever. i also tried "exec sleep". there is also some stuff in /etc/init.d/sendsigs that might be of interest.

for anyone who like to do the puzzle just make the timeout work in pre-stop and we should have the solution by placing the core script into it.

[Reinhard Tartler (siretart)]

instead of destroying vms, wouldn't be a more elegant solution to "save"
the state of the VM in a statefile?

On bootup, upstart would check for such statefiles and restore the
VMs. This would work even for VMs that don't react to ACPI events
properly and saves the VM's uptimes.

[ossjunkie (ossjunkie)]

@Rheinhard: sure it would be, but this should be the next step, as we still have shutdown not working yet. so let's stay to the KISS principle till we got the upstart integration.

[ossjunkie (ossjunkie)]

as i really needed a solution i did a dirty workaround for servers:

move /sbin/shutdown to /sbin/shutdown.real
place the attached script to /sbin/shutdown and make it executable

you can do the same for /sbin/reboot to have it run even on "reboot, halt, poweroff --force". you only need to change REALCMD to /sbin/reboot.real

this way we could at least just run "reboot" and "shutdown" again without worring about the guests.

[Sergey Svishchev (svs)]

This is implemented in libvirt 0.8.2:

https://bugzilla.redhat.com/show_bug.cgi?id=444273

[Valentijn Sessink (valentijn)]

I'm still not sure, but doesn't the /etc/init.d/sendsigs script, combined with the Ubuntu 10.04 /etc/init/rc.conf, make up for a giant race condition? Where a shell script tries to find out which processes it should not kill? This is, at least, what I'm making of it, when running libvirt on a server that has no other services running (i.e. the sendsigs script is run almost right away).

[Francesco Pretto (ceztko)]

I have seen the commit that implements this functionality and it's, as
usual, a sysvinit script. Unfortunately, I think this is a use case
where upstart can't do anything yet and the only solution (without
moving back to a sysvinit script) is doing an ugly workaround in
/etc/init.d/sendsigs. I've tried to subscribe Scott James Remnant
(maintainer of upstart: thanks!): this is not a bug in upstart (maybe
a feature lack) but I hope he have something to share with us about
this problem.

2010/8/6 Sergey Svishchev <email address hidden>:
> This is implemented in libvirt 0.8.2:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=444273
>
> ** Bug watch added: Red Hat Bugzilla #444273
>   https://bugzilla.redhat.com/show_bug.cgi?id=444273
>
>

[Valentijn Sessink (valentijn)]

~$ dpkg -S /etc/init/rc.conf
upstart: /etc/init/rc.conf

While rc.conf has nothing to do with shutting down virtual machines, it does have the consequence of running the sendsigs script in parallel with the upstart scripting. I'm not sure that is the right way to proceed.

[Andy (andy-xillean)]

So does this mean that KVM is not properly supported on Ubuntu 10.04 Server LTS? The ability to automatically shutdown VMs when someone logs into a production host server and types /sbin/shutdown does not exist on a server operating sytstem flagged LTS that is supposed to be for production? This can't be. There must be something that is being overlooked here.

I tried creating a simple traditional sys5 init script to test this out after the script I wrote failed to shutdown the guests on reboot or shutdown but works manually from the command line.

[code]

!/bin/bash
### BEGIN INIT INFO
# Provides: virsh test
# Required-Start:
# Required-Stop:
# Default-Start:
# Default-Stop: 0 1 6
# Short-Description: Gracefull shutdown of all KVM guests.
# Description: Shutdown KVM guests on host shutdown
### END INIT INFO

/usr/bin/virsh list >>/var/log/kvmguest.log 2>&1

[/code]

this works when i run it manually.
I created a link from /etc/init.d/testscript to /etc/rc6.d/K5testscript.
reboot server.

and I get this message in the log
error: unable to connect to '/var/run/libvirt/libvirt-sock': Connection refused
error: failed to connect to the hypervisor

So ofcourse the guests are not shutdown gracefully on host shutdown. A very basic feature of running a virtual machine server. Wow. Just WOW!

[Valentijn Sessink (valentijn)]

libvirt-bin gets shutdown by Upstart, so when you try to shutdown from an init script, chances are that libvirt has shutdown already. So if you want to do this, you need to change /etc/init/libvirt-bin.conf as well, and have it wait for the VM's to shut down.

[Andy (andy-xillean)]

Ok I create a pre-stop script in /etc/init/libvirt-bin.conf
and it is running the script however its not waiting for the script to complete. Everything shutsdown extremely fast in about 3 seconds and I am back to the BIOS screen.
So the it starts shutting down the guest os but the host contiues to shutdown no waiting for the guest to complete shutting down?
I even added sleep 20 in the pre-stop script in /etc/init/libvirt-bin.conf and it completely ignored the sleep command or because sendsigs may be running in parallel the system shutsdown anyways regardless of any sleep command. If this is the case then thats just brilliant. So how does one fix this if this is indeed the case?

modified the /etc/init.d/sendsigs file as stated above in comment #24 by tomcrus and it just hangs the host on reboot forcing a hard reset.

Does anyone know if Canonical uses KVM internally on Ubuntu 10.04 LTS. If so how have they solved this problem? Is there a paid solution?

Thanks.

[Andy (andy-xillean)]

Ok so I came up with a workaround that seems to be working for me. The guests shutdown completely. Not sure what this breaks if anything. There still needs to be a permanent fix for this bug but here is what I did.

(1)
Edit /etc/init/libvirt-bin.conf
I added this under the "pre-start script" section.

pre-stop script
        /etc/init.d/kvmguests stop
end script

(2)
Edit /etc/init.d/sendsigs

Search in /etc/init.d/sendsigs for "do_stop () {" and add these lines right below it.

kvm_shutdown_timeout=30
while /bin/ls /var/run/libvirt/qemu/*.pid>/dev/null 2>&1 && [ $kvm_shutdown_timeout -gt 0 ]
        do
        sleep 1
        kvm_shutdown_timeout=$(expr $kvm_shutdown_timeout - 1)
done

(3)
Create sys-v init script kvmguests and put it in /etc/init.d and make it executable.
Here is mine attached to this post. It can also be used to manually stop the guests.

**Notes: The kvm_shutdown_timeout=30 in /etc/init.d/sendsigs can be tuned to your environment.

[Andy (andy-xillean)]

Update**
In my comment #38 above step (2) I replaced the following modification in /etc/init.d/sendsigs from.

while /bin/ls /var/run/libvirt/qemu/*.pid>/dev/null 2>&1 && [ $kvm_shutdown_timeout -gt 0 ]

with:

while /usr/bin/pgrep -cx kvm > /dev/null 2>&1 && [ $kvm_shutdown_timeout -gt 0 ]

I have rebooted the host several times and all 2 guests shutdown cleanly.

[Reinhard Tartler (siretart)]

Andy,

I didn't try your suggestion (yet), but is it really necessary to create
the kvmguests init script? Can't you just place the script inside the
pre-stop script section of /etc/init/libvirt-bin.conf?

Moreover, I've noticed that your kvmguests script uses upstart's 'start'
command to ensure that libvirtd is actually running. is that really
necessary? AFAIUI this can be dropped if integrated in libvirt-bin.conf

As for your change to /etc/init.d/sendsigs, wouldn't it be less
intrusive to make /etc/init/libvirt-bin.conf or probably libvirtd itself
drop or copy the pid files into /lib/init/rw/sendsigs.omit.d? AFAIUI,
the upstart job should be executed before the sendsigs init script, or
is there another race here?

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4

[Andy (andy-xillean)]

Reinhard, to answer you questions.
I created the init script first before I ran into this problem
The reason being was 1) to minimize modification of system files and 2) to be able to shutdown guests on non-upstart versions of Linux. and 3) to enable stopping all guests manually without having to stop libvirtd

As far as the section that checks if libvirtd is running. I had and inststance on shutdown where libvirtd was not running
for some reason. and hence it could not stop the guests. Dont know why that was so I stuck that in there just to make sure it is before shutting down the vm. It may not be necessary at this point.

I thought of using the omit feature in the sendsigs script. I tried to test this by manully putting the pids of vms in file called
/lib/init/rw/sendsigs.omit. This did nothing. Maybe I didn't do something correctly there to test this.

Reinhard said: "wouldn't it be less intrusive to make /etc/init/libvirt-bin.conf or probably libvirtd itself
drop or copy the pid files into /lib/init/rw/sendsigs.omit.d? AFAIUI,
the upstart job should be executed before the sendsigs init script, or
is there another race here?"

I agree. If it would work. I think the less intrusive the better actually. having to keep up with modified system files on every 10.04 system i setup can be a drag.

[Alastair Sherringham (asherringham)]

I've been banging my head on this for over a day, pretty sick of the reboot/test cycle. Nothing seemed to work - and then I was pointed at this bug on IRC in #ubuntu-server.

I think it is essential that a KVM host attempts a clean shutdown of any guests on host reboot. It was quite disconcerting to find KVM/libvirt did not have this functionality in Ubuntu Server 10.04, especially since I am now in production. This isn't production quality really.

Thanks to all above for the discussion and especially the suggestions. I have used Francesco's suggestion in comment #8 and this seems to work. It is very unsatisfactory to hack system scripts though (libvirt-bin.conf and sendsigs). I hope something cleaner can be done - and since 10.04 is LTS, a back-port.

Thanks.

[tomcrus (tomcrus)]

Sorry for not writing that long, there was lot to do at work.

My solution now for this problem is using a script that gets called by molly-guard, as testing other script-stuff was really annoying! As Molly-Guard installs it self as shutdown-command and executes each script inside /etc/molly-guard/run.d/ just before doing the real shutdown, it seems the perfect way for me on our server-system.
I know it's just another workaround, but it's the least intrusive way and works very well because shutting down KVMs is done before all the other upstart/sysv-scripting stuff.

Greets
tomcrus

[John Morrissey (jwm)]

The attached file updates libvirt-bin's upstart job to gracefully shut down VMs when the system shuts down. You can alter the shutdown timeout (SHUTDOWN_TIMEOUT, default: 300s) and list of virsh(1) URIs to use (URIS, default: "qemu:///system") in /etc/default/libvirt-bin.

You'll also need the omit-kvm-vm-pids init script (attachment forthcoming), which copies the KVM pidfiles to /lib/init/rw/sendsigs.omit.d, preventing /etc/init.d/sendsigs from killing KVM VM processes before the updated libvirt-bin upstart job shuts them down.

Finally, this modified upstart job requires the fix for https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/639940. Otherwise, the libvirt-bin pre-stop script isn't guaranteed to finish successfully, since sendsigs races the child processes executed during the course of the job script.

With all of these changes, VMs shut down gracefully with the physical host. I would have implemented suspend-on-shutdown (a la the rc script from libvirt 0.8.2), but lucid has an older libvirt that doesn't support 'virsh managedsave'. Shutting down domains gets us *a lot* of the way there, anyway. In fact, I run some largeish KVM hosts, and saving 30-60GB of memory images to disk before shutdown strikes me as less useful than shutting the domains down.

sudo service stop libvirt-bin
sudo cp libvirt-bin.conf /etc/init/
sudo kill -HUP 1
sudo service start libvirt-bin
sudo cp omit-kvm-vm-pids /etc/init.d/
sudo update-rc.d omit-kvm-vm-pids defaults
sudo cp -f sendsigs /etc/init.d/ # get fixed sendsigs from LP#639940

[Francesco Pretto (ceztko)]

2010/9/17 John Morrissey <email address hidden>:
> The attached file updates libvirt-bin's upstart job to gracefully shut
> down VMs when the system shuts down. You can alter the shutdown timeout
> (SHUTDOWN_TIMEOUT, default: 300s) and list of virsh(1) URIs to use
> (URIS, default: "qemu:///system") in /etc/default/libvirt-bin.
>

While I am beginning to hate upstart because of its deficiencies
(#494141, #406397), I don't think adding a sysvinit script to fix an
upstart job is the best workaround. Just handle pid omits in sendsigs
as a special case for libvirt, as done for other problems.

[John Morrissey (jwm)]

@francesco, #46
The packaging is cleaner and easier with the rc script running before sendsigs (which is also an rc script). That way, the initscripts package/sendsigs doesn't need to carry around special behavior for libvirt-bin and any other packages that don't need this type of fix.

[John Morrissey (jwm)]

Er, s/don't need/need/

[Alastair Sherringham (asherringham)]

John (#44),

Thanks for your efforts. Looks like you really spent some time on this and if it's working for you now, that's great.

I also banged my head on this - upstart, the conf, init, send-sigs etc. and it's a real PITA :-) I couldn't get it working (guest FS always needed an fsck for an unclean shutdown). In the end, I installed "molly-guard" - and now think that this is a better solution for me - and something I should install on my servers anyway (whether KVM VM guests or not). It's simpler and more understandable to me, plus perhaps more maintainable.

[Valentijn Sessink (valentijn)]

While there's many solutions now to shutdown VM's, here's my 2 cents. Instead of a libvirt-bin.conf script, I made a separate script (/usr/local/sbin/shutdown-libvirt-hosts.sh). This script waits 120 seconds for all VM's to shut down. Then I'm running this script from two places; in libvirt-bin.conf:
pre-stop script
        /usr/local/sbin/shutdown-libvirt-hosts.sh
end script

And also (as a link) from /etc/rc0.d/K10shutdown-libvirt-hosts (and from rc6.d alike)

This way, the "sendsigs" script will only run if all VMs have terminated.?field.comment=While there's many solutions now to shutdown VM's, here's my 2 cents. Instead of a libvirt-bin.conf script, I made a separate script (/usr/local/sbin/shutdown-libvirt-hosts.sh). This script waits 120 seconds for all VM's to shut down. Then I'm running this script from two places; in libvirt-bin.conf:
pre-stop script
        /usr/local/sbin/shutdown-libvirt-hosts.sh
end script

And also (as a link) from /etc/rc0.d/K10shutdown-libvirt-hosts (and from rc6.d alike)

With a K10shutdown-libvirt-hosts script in place, the "sendsigs" script will only run after this script has ended, which is when all VMs have terminated.

A few remarks:
- shutting down the VM's twice is not a problem; to the VM, it just looks like pressing the power button twice.
- to gracefully shut down, the VM's need the "acpid" package installed.
- if the shutdown script searches for the word "running" (like mine does, but John Morrissey's script is affected, too), it must set LANG=C at the top, otherwise virsh will run in the local language.

[John Morrissey (jwm)]

On Wed, Sep 22, 2010 at 09:47:47AM -0000, Valentijn Sessink wrote:
> - if the shutdown script searches for the word "running" (like mine does,
> - but John Morrissey's script is affected, too), it must set LANG=C at the
> - top, otherwise virsh will run in the local language.

Good catch. I've updated libvirt-bin.conf; let's see if LP accepts
attachments via e-mail.

john
--
John Morrissey _o /\ ---- __o
<email address hidden> _-< \_ / \ ---- < \,
www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

[Andy (andy-xillean)]

Does anyone know what Canonical's *official* position on shutting down KVM VMs on host shutdown with Upstart?
Does Canonical run Ubuntu 10.04 Server with KVM in production?
Do they shutdown VMs? How did this slip by for an LTS release?
Is this up to their standards as Production Quality?
Do they recommend deploying KVM on Ubuntu Server 10.04?
Shouldn't it be a very basic feature to shutdown VMs on a supported platform when
the host goes through an unattended unscheduled shutdown?

It is a dismal situation when at this point there is still no official simple way from Canonical to
gracefully shutdown KVM VMs if the host has to go down for any reason such as
a UPS issued shutdown command.

[Francesco Pretto (ceztko)]

Clap! :)

2010/9/29 Andy <email address hidden>:
> It is a dismal situation when at this point there is still no official simple way from Canonical to
> gracefully shutdown KVM VMs if the host has to go down for any reason such as
> a UPS issued shutdown command.
>

[Nathan Crawford (njcrawford)]

@John Morrissey - Nice work on getting this working! I hope you don't mind, I've taken the liberty of packaging all your changes and putting them in a PPA for easier testing.
One question on omit-kvm-vm-pids from #51, what does the '##*/' mean toward the end of line 26?

[John Morrissey (jwm)]

On Fri, Oct 01, 2010 at 03:47:07AM -0000, Nathan Crawford wrote:
> @John Morrissey - Nice work on getting this working! I hope you don't
> mind, I've taken the liberty of packaging all your changes and putting
> them in a PPA for easier testing.

Thanks, and packaging the changes is no problem at all.

FWIW, you might want to mark /etc/default/libvirt-bin as a conffile in the
libvirt packaging. Also, it might make more sense to package
omit-kvm-vm-pids as part of libvirt instead of a separate package; makes it
harder to forget to install it.

> One question on omit-kvm-vm-pids from #51, what does the '##*/' mean
> toward the end of line 26?

It's a shell construct that removes text at the beginning of a parameter. In
this case, the net effect is to remove the leading pathname, so all that's
left is the filename ('##' is longest match, and the pattern is '*/'). Since
it's a shell builtin, it's faster than calling basename(1), although speed
is arguably important here. :-)

Looking at it again, that for loop should probably have a guard so it
gracefully handles the case where no QEMU VMs are running:

for pidfile in /var/run/libvirt/qemu/*.pid; do
 if [ "$pidfile" = '/var/run/libvirt/qemu/*.pid' ]; then
  break
 fi
 if ! cp -f "$pidfile" "/lib/init/rw/sendsigs.omit.d/libvirt-bin-${pidfile##*/}"; then
  status=1
 fi
done

john
--
John Morrissey _o /\ ---- __o
<email address hidden> _-< \_ / \ ---- < \,
www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

[Nathan Crawford (njcrawford)]

> FWIW, you might want to mark /etc/default/libvirt-bin as a conffile in the
> libvirt packaging.
I think it already is? (dpkg --status libvirt-bin says it is... but I've been changing things all over the place so I might have confused it)

> Also, it might make more sense to package
> omit-kvm-vm-pids as part of libvirt instead of a separate package; makes it
> harder to forget to install it.
I'll look into this, it would be nicer. There's already a libvirt-bin.init in the packaging though, so for now I just added a dependency on omit-kvm-vm-pids.

Thanks for the '##*/' explanation, google was not at all helpful with that particular search. :)

I made another change in addition to yours above. The /lib/init/rw/sendsigs.omit.d/ folder does not exist on my system, so I changed it to /var/run/sendsigs.omit.d/ as mentioned by this bug:
https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/541512

Also, are these changes based on Lucid packages? The Lucid machine I'm using for testing has a kernel panic on shutdown with the packages from my PPA installed.

[John Morrissey (jwm)]

On Sat, Oct 02, 2010 at 02:54:47AM -0000, Nathan Crawford wrote:
> > FWIW, you might want to mark /etc/default/libvirt-bin as a conffile in
> > the libvirt packaging.
>
> I think it already is? (dpkg --status libvirt-bin says it is... but I've
> been changing things all over the place so I might have confused it)

Yes, my fault. debhelper probably adds it as a conffile automatically.

> Also, are these changes based on Lucid packages? The Lucid machine I'm
> using for testing has a kernel panic on shutdown with the packages from
> my PPA installed.

Yes, I'm running these changes on a number of lucid machines. Maybe the
changed behavior triggers something in the kernel that's causing a panic?
I can't really say without seeing the full panic output.

john
--
John Morrissey _o /\ ---- __o
<email address hidden> _-< \_ / \ ---- < \,
www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

[Nathan Crawford (njcrawford)]

Sorry, I forgot to report back about the panics. They were caused by a corrupted XFS filesystem on that server. I haven't had time to do any further testing with my packages since then - I just hacked the shutdown command to do what I needed until I have time to thoroughly test them.

[Valentijn Sessink (valentijn)]

Hmm. If libvirt shuts down the virtual machines it monitors, then these virtual machines will also be shut down when updating libvirt.

[Francesco Pretto (ceztko)]

Don't think so. It's part of the independence of libvirt from the the
actual virtualization technology (KVM, ...) to permit updates without
shutdown of guests.

2010/11/4 Valentijn Sessink <email address hidden>:
> Hmm. If libvirt shuts down the virtual machines it monitors, then these
> virtual machines will also be shut down when updating libvirt.
>

[Valentijn Sessink (valentijn)]

The proposals in this report (mine, John Morrissey's) all have a "pre-stop" shutdown of virtual machines in their Upstart conf-file.

When you update libvirt-bin, dpkg will stop libvirt-bin before upgrading. As a result — if you have implemented the pre-stop shutdown — your virtual machines will all stop.

[John Morrissey (jwm)]

On Thu, Nov 04, 2010 at 11:43:10AM -0000, Valentijn Sessink wrote:
> The proposals in this report (mine, John Morrissey's) all have a "pre-
> stop" shutdown of virtual machines in their Upstart conf-file.
>
> When you update libvirt-bin, dpkg will stop libvirt-bin before
> upgrading. As a result — if you have implemented the pre-stop shutdown —
> your virtual machines will all stop.

Sorry, that's incorrect.

The libvirt-bin upstart job I wrote will only shut down VMs if the physical
machine is entering runlevels 0 or 6 (i.e., shutdown or reboot):

--
pre-stop script
 if [ -z "$RUNLEVEL" ]; then
  exit 0
 fi
 if [ "$RUNLEVEL" -ne 0 ] && [ "$RUNLEVEL" -ne 6 ]; then
  exit 0
 fi
--

john
--
John Morrissey _o /\ ---- __o
<email address hidden> _-< \_ / \ ---- < \,
www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

[Marshall Kennard (marshall-ahoy)]

I made the following changes to John's scripts for Lucid:

modified omit-kvm-vm-pids as follows:
    changed directory: /lib/init/rw/sendsigs.omit.d to /var/run/sendsigs.omit.d
    check for no qemu pids
modified sendsigs as follows:
    changed directory: /lib/init/rw/sendsigs.omit.d to /var/run/sendsigs.omit.d
    added quotes on local variable assignments
    example: local omitpids=$1
                          local omitpids="$1"
    also added a 10 second delay at the end to allow guests to shutdown
    otherwise the umount for the root filesystem fails and the system never shuts down

See attached updated scripts.

Installed as follows:

sudo service libvirt-bin stop
sudo cp libvirt-bin.conf /etc/init/
sudo kill -HUP 1
sudo service libvirt-bin start
sudo cp omit-kvm-vm-pids /etc/init.d/
sudo update-rc.d omit-kvm-vm-pids start 18 0 6 .
sudo cp sendsigs /etc/init.d

[Marshall Kennard (marshall-ahoy)]

revised omit-kvm-vm-pids

[Martin Rusko (rusko)]

On John Morrissey's libvirt-bin.conf file ... there is construct like:

 delay=$SHUTDOWN_TIMEOUT
 while [ $delay -gt 0 ]; do
  for uri in $URIS; do
   if ! run_virsh -c "$uri" list | awk '$3 == "running" {exit 1}'; then
    # VMs at this URI are still running. Wait, then
    # start at the beginning looking for running VMs.
    sleep 1
    delay=$(($delay - 1))
    continue 2
   fi
  done
 done

... what if no guest is running anymore? We will never enter if...then condition and thus never decrement $delay variable anymore. It looks like we stay in that while...done loop forever.

[Martin Rusko (rusko)]

Well placed 'break' command should make libvirt-bin.conf functioning correctly.

[Lars Hansson (romabysen)]

The same problem exist in 10.10 and John's posted workarounds are not working there.
The only way I have managed to have the VM's shutdown is on halt/reboot is by installing molly-guard. Not a wholly satisfactory solution but at least it works.

[Clint Byrum (clint-fewbar)]

Wow this one has been around a while.

The simplest way to run a script before stopping, only on shutdown, but not on restarts, is to use the UPSTART_EVENTS.
 environment variable.

pre-stop script
  if [ x$UPSTART_EVENTS = xrunlevel ] ; then
    shutdown_guests
  fi
end script

[Clint Byrum (clint-fewbar)]

Oops, just noticed, that should be $UPSTART_STOP_EVENTS

[Orticio Jlgtgutisu (jlgutisu3)]

¿? Lenguage spanisch ??????????????????

> Date: Wed, 2 Feb 2011 00:22:03 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 350936] Re: Should shut down domains on system shutdown
>
> Oops, just noticed, that should be $UPSTART_STOP_EVENTS
>
> --
> You received this bug notification because you are subscribed to Ubuntu
> ubuntu-10.04.2.
> https://bugs.launchpad.net/bugs/350936
>
> Title:
> Should shut down domains on system shutdown
>
> Status in “kvm” package in Ubuntu:
> Invalid
> Status in “libvirt” package in Ubuntu:
> Triaged
>
> Bug description:
> I'm using autostart feature of libvirtd (my guest defined in
> /etc/libvirt/qemu/autostart/). It's starts normally, but if I reboot
> server then guest fs will be corrupted becouse its cant stop in time:
>
> WARNING: / was not properly dismounted
>
> We need to give more time to libvirtd to stop all guests.
>
>
> I'm running jaunty amd64 with all updates.
>
>

[Orticio Jlgtgutisu (jlgutisu3)]

¿? LENGUAGE SPANISCH ????????????????

> Date: Tue, 1 Feb 2011 22:35:00 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 350936] Re: Should shut down domains on system shutdown
>
> Wow this one has been around a while.
>
> The simplest way to run a script before stopping, only on shutdown, but not on restarts, is to use the UPSTART_EVENTS.
> environment variable.
>
> pre-stop script
> if [ x$UPSTART_EVENTS = xrunlevel ] ; then
> shutdown_guests
> fi
> end script
>
> --
> You received this bug notification because you are subscribed to Ubuntu
> ubuntu-10.04.2.
> https://bugs.launchpad.net/bugs/350936
>
> Title:
> Should shut down domains on system shutdown
>
> Status in “kvm” package in Ubuntu:
> Invalid
> Status in “libvirt” package in Ubuntu:
> Triaged
>
> Bug description:
> I'm using autostart feature of libvirtd (my guest defined in
> /etc/libvirt/qemu/autostart/). It's starts normally, but if I reboot
> server then guest fs will be corrupted becouse its cant stop in time:
>
> WARNING: / was not properly dismounted
>
> We need to give more time to libvirtd to stop all guests.
>
>
> I'm running jaunty amd64 with all updates.
>
>

[Andreas Ntaflos (daff)]

Orticio, this is an English-speaking forum. Please stop asking people to speak Spanish.

Translated by Google:

Este es un foro de habla Inglés. Por favor, deje de pedir a la gente a hablar español.

[gollum53 (smid)]

Hi,
I have installed the latest mentioned files here (sendsigs, omit-kvm-vm-pid, libvirt-bin.conf with 'break' command suggested by Martin Rusko) on our Lucid 64bit. servers, and it did not help. When doing a reboot, the system does not wait for the domains to perform a clean shutdown...is there anyone here with working configuration? What could be wrong? Can you please post your files?
Thanx
Roman Smid

[Martin Rusko (rusko)]

Hi Roman,

actually I ended up with molly-guard solution. It is suboptimal solution because it stops your virtual machines only when shutdown/restart is initiated manually (and only from remote ssh session). It won't apply if UPS initiates shutdown or if power button is pressed shortly, etc.

In my case "the server" (well, shall I still call it this way?) is desktop PC with no screen, no UPS, accessible only via SSH. Therefore molly-guard solutions is satisfactory for me. Attached is file, which you can copy into /etc/molly-guard/run.d/ directory if you decide to use it this way.

Cheers,
Martin

[Clint Byrum (clint-fewbar)]

On Fri, 2011-03-18 at 11:27 +0000, gollum53 wrote:
> Hi,
> I have installed the latest mentioned files here (sendsigs, omit-kvm-vm-pid, libvirt-bin.conf with 'break' command suggested by Martin Rusko) on our Lucid 64bit. servers, and it did not help. When doing a reboot, the system does not wait for the domains to perform a clean shutdown...is there anyone here with working configuration? What could be wrong? Can you please post your files?
> Thanx
> Roman Smid
>

I'd be interested to hear if adding this to libvirt-bin.conf would work:

pre-stop script
  if [ x$UPSTART_EVENTS = xrunlevel ] ; then
    shutdown_guests
  fi
end script

Obviously the 'shutdown_guests' command needs to be a single command
which stops the guests and waits for them to die.

[Lars Hansson (romabysen)]

The problem with the upstart solution is that it doesn't work since
Upstart will kill the virsh instances that you spawn in
shutdown_guests.
So far the only somewhat satisfactory solution is molly-guard.

[John Morrissey (jwm)]

On Fri, Mar 18, 2011 at 05:13:55PM -0000, Lars Hansson wrote:
> The problem with the upstart solution is that it doesn't work since
> Upstart will kill the virsh instances that you spawn in
> shutdown_guests.

That's only the case if you don't follow the instructions to patch sendsigs
to avoid that bug:

> #44: John Morrissey wrote on 2010-09-17:
> Finally, this modified upstart job requires the fix for
> https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/639940. Otherwise,
> the libvirt-bin pre-stop script isn't guaranteed to finish successfully,
> since sendsigs races the child processes executed during the course of the
> job script.

john
--
John Morrissey _o /\ ---- __o
<email address hidden> _-< \_ / \ ---- < \,
www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

[Clint Byrum (clint-fewbar)]

On Fri, 2011-03-18 at 17:13 +0000, Lars Hansson wrote:
> The problem with the upstart solution is that it doesn't work since
> Upstart will kill the virsh instances that you spawn in
> shutdown_guests.
> So far the only somewhat satisfactory solution is molly-guard.
>

Right, so you would need shutdown_guests to block until everything is in
fact shut down:

while box in `virsh list --all | grep '^ -'|grep -qv 'shut off'` ; do
  virsh shutdown all
  sleep 1
done

Note that currently libvirt-bin.conf has

stop on runlevel [!2345]

There is a bug in sendsigs that does not wait for these upstart jobs to
die before moving on to the end of the shutdown... so really, we also
need to fix that bug before the technique above will work.

[Serge Hallyn (serge-hallyn)]

@Clint,

is there an open bug for sendsigs? Can you mark this bug as blocked on that one?

[gollum53 (smid)]

Hi,
the proposed solution (patched sendsigs, libvirt-bin.conf, omit-kvm-vm-pids) still does not work for me. I have done several tests with sendsigs and found out that the corresponding pids of running virtuals are successfully included in the omitpids array in sendsigs. Hovever, the virtuals are still being killed instantly upon reboot...:( Any ideas?
Thanx
Roman Smid

[tuttle (sven-launchpad-anderson)]

This is really a show-stopper and spoils the otherwise solid solution to use Ubuntu and KVM for virtualization.

Why did nobody propose yet to shut down the VMs by the qemu-kvm script? I think "stop qemu-kvm" should try to shutdown all running kvms and kill them after a timeout, independently of libvirt. This would also work for instance, if libvirtd has been stopped before for other reasons and then the system is rebooted. With the monitor (which is accessible after libvirtd has been stopped) you can send a "system_powerdown" command.

At the moment "stop qemu-kvm" just removes the kvm kernel modules. Maybe this is the explanation, why the kvm processes get killed? (Well, normally not, the modprobe -r should not succeed, as long as the module is in use. I cannot try it at the moment.)

Sven

[Serge Hallyn (serge-hallyn)]

Hi Clint,

Assigned this to you in the hopes you would reply to comment #79 :)

[Clint Byrum (clint-fewbar)]

Shutdown is definitely on the radar for heavy testing this cycle, and I hope to have a good solution for this bug soon. The bug in sendsigs should hopefully become moot as we transition the shutdown to a more upstart-aware solution.

[Sebastian Marsching (sebastian-marsching)]

The problem of shutting down the virtual machines consists of two parts:

If a init-script is used, libvirtd might be killed, before the shutdown command has been sent to the virtual machines. If the pre-stop script in the libvirt-bin upstart job is used or a new upstart job starting on stopping libvirt-bin is used, the computer is rebooted, the VMs are killed by the sendsigs script. Even if the sendsigs script is modified to not kill the VMs, the reboot or halt script might run before the libvirt-bin upstart job has been stopped.

I use the following approach, which works quite well for me:
I added a libvirt-shutdown-domains upstart job, which is a task starting on stopping libvirt-bin, thus running before libvirt-bin is stopped. This makes sure that libvirt-bin is not stopped, before the VMs are shutdown. In fact, I also included a runlevel check in the libvirt-shutdown-domains script, so that the VMs are only shutdown, when libvirtd is stopped because of a runlevel change. This allows for libvirtd to be restarted (e.g. when libvirt-bin is upgraded) without having to shutdown all VMs.
I also added a init-script running before the sendsigs script, which waits for libvirt-bin to be stopped (or a timeout, whichever occurs first). Thus, sendsigs will not kill VMs and the machine will not be rebooted or powered off, before the VMs have been stopped.

The scripts I wrote for this can be found at http://sebastian.marsching.com/wiki/Linux/KVM#Shutdown_virtual_machines_on_host_system_shutdown and some more explanations can be found at http://sebastian.marsching.com/blog/archives/112-KVM-and-Graceful-Shutdown-on-Ubuntu.html.

The solution I created is not perfect though because of two issues:
1. The python-libvirt package is needed, because I use a Python script to shutdown the virtual machines. You might want to use a shell script using virsh for that, thus eliminating this dependency.
2. The timeout, after which the shutdown will proceed anyway, is configured at two different places: The init-script and the Python script called by the upstart job. You might want to put both in the same place (e.g. /etc/default/libvirt-bin).

[Peter Thomassen (mail-peter-thomassen)]

Thank you, Sebastian, it works more awesome and came just in time.

[Francesco Pretto (ceztko)]

Is this fixed in ubuntu 10.04.3?

[dbendlin (diego-bendlin)]

Hello Guys,

I wanned to share my workarroud on this issue, Im using debian squeeze as OS for a couple of virtualization hosts, each one of them holding a number of vm's.

What I need is that when shuting down my virtualization host (from cli, by cron, by ups low batt or by pressing the power button (ACPI)), the host hast to shutdown each vm before going down.

I used one of the scripts from earlier posts:
1.- I copied it to /usr/local/bin
2.- Changed ownership (chown root:root /usr/local/bin/libvirt-shutdown-domains)
3.- Chandeg permitions (chmod 775 /usr/local/bin/libvirt-shutdown-domains)
4.- Edited /etc/init.d/libvirt-bin, adding a reference to this file as the secound line in the stop section of the init file

Before actually executing the libvirt-bin stop logic, it calls the first script witch shutdown each vm's, when that script ends (after every vm is gracefully halted), the libvirt-bin init script continues.

[dbendlin (diego-bendlin)]

Here's my /etc/init.d/libvirt-bin file

[Serge Hallyn (serge-hallyn)]

Thanks to ossjunkie for your original script, most of which is in my updated libvirt job.

I'm requesting a review by Spamaps for upstart oddities, but the script in the linked tree is working for me.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 0.9.2-4ubuntu7

---------------
libvirt (0.9.2-4ubuntu7) oneiric; urgency=low

  * libvirt-bin.upstart: add a pre-stop script to shut down running VMs
    before the host shuts down. (LP: #350936)
 -- Serge Hallyn <email address hidden> Tue, 02 Aug 2011 19:49:40 -0500

[Andreas Ntaflos (daff)]

So is there any way the proper fix will make it into Lucid? Or has that ship sailed?

In the meantime I have applied Sebastian Marsching's method which works quite well. Thanks for that, Sebastian.

[Clint Byrum (clint-fewbar)]

Andreas, yes, this should make it into lucid soon enough. We'll let Serge's fix bake for about a week in oneiric, and then backport to lucid. I don't know if we'll do the same for maverick and natty, though its not out of the question (its just a matter of whether its important enough to users).

[Andreas Ntaflos (daff)]

Clint, great to hear, thanks. We have many Lucid servers in production, but no Maverick or Natty, so the fix getting into Lucid is most important for us. But if time and resources allow it, backporting the fix to Maverick and Natty would certainly be nice.

[Chris Halse Rogers (raof)]

Hello exe, or anyone else affected,

Accepted libvirt into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martin Pitt (pitti)]

Hello exe, or anyone else affected,

Accepted libvirt into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martin Pitt (pitti)]

Hello exe, or anyone else affected,

Accepted libvirt into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Nathan Crawford (njcrawford)]

Tested on 10.04 64bit. Before installing the proposed package, guests' /var/log/dmesg contained this after every time the host was shutdown (either with shutdown command or by pressing power button):
[ 7.316252] EXT3-fs: INFO: recovery required on readonly filesystem.
[ 7.316256] EXT3-fs: write access will be enabled during recovery.
[ 36.242794] kjournald starting. Commit interval 5 seconds
[ 36.242815] EXT3-fs: recovery complete.

And after installing proposed package, guest dmesg contains:
[ 7.101435] kjournald starting. Commit interval 5 seconds
[ 7.101455] EXT3-fs: mounted filesystem with ordered data mode.

It gets a thumbs up from me!

[Saman Behnam (sbehnam73)]

OK I've wrote something, that works on Lucid 10.04-3 without patching or backporting!!!!
Just check the attachment!

Deshalb koennen Pinguine nicht fliegen. Was nicht fliegt kann nicht abstuerzen.

have fun

[Saman Behnam (sbehnam73)]

sorry
the first one had errors!
here is a corrected version 1.0-1

[Martin Pitt (pitti)]

Resetting verification tags for missing maverick testing.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 0.7.5-5ubuntu27.17

---------------
libvirt (0.7.5-5ubuntu27.17) lucid-proposed; urgency=low

  * debian/libvirt-bin.upstart: add a pre-stop script to shut down running VMs
    before the host shuts down. (LP: #350936)
  * debian/libvirt-bin.default: add a comment that this file is not actually
    used at startup. (LP: #823638)
 -- Serge Hallyn <email address hidden> Mon, 15 Aug 2011 15:20:37 -0500

[Chris Halse Rogers (raof)]

Is anyone available to test this for natty and/or maverick?

[nutznboltz (nutznboltz-deactivatedaccount)]

I could test on Natty later today.

Adding link to pending SRU page for my convenience
http://people.canonical.com/~ubuntu-archive/pending-sru.html

[Serge Hallyn (serge-hallyn)]

Verified on maverick-proposed.

[nutznboltz (nutznboltz-deactivatedaccount)]

Verified on natty-proposed.

$ dpkg -l | grep libvirt | awk '{print $2 " " $3}'
libvirt-bin 0.8.8-1ubuntu6.6
libvirt0 0.8.8-1ubuntu6.6
python-libvirt 0.8.8-1ubuntu6.6
$ lsb_release -ds
Ubuntu 11.04
$ uname -srvi
Linux 2.6.38-13-generic #52-Ubuntu SMP Tue Nov 8 16:53:51 UTC 2011 x86_64
$ cat /var/log/libvirt/shutdownlog.log
libvirt: libvirt-bin: entering pre-stop at Mon Nov 14 17:37:47 EST 2011
libvirt: libvirt-bin: attempting clean shutdown of test-vm at Mon Nov 14 17:37:47 EST 2011
libvirt: libvirt-bin: exiting pre-stop at Mon Nov 14 17:37:53 EST 2011

[nutznboltz (nutznboltz-deactivatedaccount)]

In comment #104 Serge wrote that he did the verification on Maverick. Based on that I'll update the tags.

[Clint Byrum (clint-fewbar)]

nutznboltz, typically we require somebody who wasn't the developer uploading the fix to verify it (Serge did the fix). However, with maverick in its sunset (< 6 months of life left) I think we should accept the fact the the surrounding releases, lucid and natty worked fine with the same fix.

[nutznboltz (nutznboltz-deactivatedaccount)]

clint-fewbar, Thanks, I'll keep that in mind.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 0.8.3-1ubuntu19.4

---------------
libvirt (0.8.3-1ubuntu19.4) maverick-proposed; urgency=low

  * New version of debian/patches/lxc-use-own-ptyfns.patch. Previous version
    failed to build.

libvirt (0.8.3-1ubuntu19.3) maverick-proposed; urgency=low

  * lxc_controller: use our own unlocpt+grantpt rather than glibc's, which
    can't handle opening a pty in a devpts not mounted at /dev/pts.
    (LP: #863629)
 -- Serge Hallyn <email address hidden> Tue, 15 Nov 2011 08:06:57 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 0.8.8-1ubuntu6.7

---------------
libvirt (0.8.8-1ubuntu6.7) natty-proposed; urgency=low

  * lxc_controller: use our own unlocpt+grantpt rather than glibc's, which
    can't handle opening a pty in a devpts not mounted at /dev/pts.
    (LP: #863629)
 -- Serge Hallyn <email address hidden> Tue, 01 Nov 2011 18:00:51 +0000

[Rolf Leggewie (r0lf)]

natty has seen the end of its life and is no longer receiving any updates. Marking the natty task for this ticket as "Won't Fix".