starting kvm guests with libvirt on virthost startup doesn't wait till all dependencies are available

[Lars]

Hallo!

One of our kvm guests has its system on a SAN (AoE) device.
This aoe device is available about 2 minutes after startup.
The libvirt startup script tries to start the guest earlier.
The guest dies with:

qemu: could not open disk image /dev/etherd/e9.2: No such file or directory

Is it possible to test for this dependency or insert a wait command?

Thanks
Lars

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: libvirt-bin 0.7.5-5ubuntu27
ProcVersionSignature: Ubuntu 2.6.32-22.36-server 2.6.32.11+drm33.2
Uname: Linux 2.6.32-22-server x86_64
Architecture: amd64
Date: Thu Jun 17 08:59:01 2010
ExecutablePath: /usr/sbin/libvirtd
InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
ProcAttrCurrent: /usr/sbin/libvirtd (enforce)
ProcEnviron: PATH=(custom, no user)
SourcePackage: libvirt

[Serge Hallyn (serge-hallyn)]

Hi,

thanks for the report. I assume you have libvirt set up to start these VMs
automatically, and what you are asking is for the startup scripts to wait for
two minutes after boot before starting the VMs?

There are three ways I would consider going about this.

One is to simply mark the VMs to not start up automatically. You could start
them manually, or create another job in /etc/init/ which waits 2 minutes and
then starts the VMS.

Another is to edit /etc/init/libvirt-bin.conf to make libvirt wait to start up
altogether. If the aoe mounts have their own upstart job which works
correctly, then you can make it depend on that, or you can simply add a
'prestart exec sleep 3m' to amke it wait 3 minutes until after boot. You
might want to talk to upstart folks for the best way to do this.

Another is to patch libvirt to add the capability to wait to auto-start VMs.
You can discuss this with upstream at <email address hidden>.

The best solution is the second one, particularly if there is an aoe upstart
job which emits a completion event when the mounts are available.

[Lars (lars-taeuber)]

Hi!

Yes, the vms is configured to be started automatically by libvirtd.

The aoetool start script only loads the module. I don't mount any aoe device, because it is used directly by the vm.

You're right the best solution would be to enhance libvirtd to check all prerequisites for each vm before starting it.
But sadly I had no time to wait for this to be realised. And I'm not skilled enough to do this myself.

So my solution was to enhance the upstart script in this way:

pre-start script
        mkdir -p /var/run/libvirt
        # Clean up a pidfile that might be left around
        rm -f /var/run/libvirtd.pid

        AOE=$(aoe-stat | wc -l)
        until [ $AOE -gt 0 ]
        do
            sleep 1
            AOE=$(aoe-stat | wc -l)
        done
end script

This pre-start script waits till at least one aoe device is accessible.

Thanks and kind regards
Lars

[Serge Hallyn (serge-hallyn)]

Hi Lars,

glad you were able to work around it. Script looks good.

I'm going to set the status of this bug to Wontfix as the aoe-stat
check is not appropriate for general usage. If you get time to
talk to upstream about strategies for a generic solution, I
suspect there are other people (with other types of filesystems)
who also could benefit.

[Lars (lars-taeuber)]

Hallo Serge,

Serge Hallyn <email address hidden> schrieb:
> Hi Lars,
>
> glad you were able to work around it. Script looks good.

thanks.

>
> I'm going to set the status of this bug to Wontfix as the aoe-stat
> check is not appropriate for general usage.

This seems very reasonable.

> If you get time to
> talk to upstream about strategies for a generic solution, I
> suspect there are other people (with other types of filesystems)
> who also could benefit.

I think the correct solution would be an integration of such a check directly into the libvirt tools.

Best regards
Lars

>
>
> ** Changed in: libvirt (Ubuntu)
> Status: Incomplete => Won't Fix
>
> --
> starting kvm guests with libvirt on virthost startup doesn't wait till all dependencies are available
> https://bugs.launchpad.net/bugs/595388
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in “libvirt” package in Ubuntu: Won't Fix
>
> Bug description:
> Hallo!
>
> One of our kvm guests has its system on a SAN (AoE) device.
> This aoe device is available about 2 minutes after startup.
> The libvirt startup script tries to start the guest earlier.
> The guest dies with:
>
> qemu: could not open disk image /dev/etherd/e9.2: No such file or directory
>
> Is it possible to test for this dependency or insert a wait command?
>
> Thanks
> Lars
>
> ProblemType: Bug
> DistroRelease: Ubuntu 10.04
> Package: libvirt-bin 0.7.5-5ubuntu27
> ProcVersionSignature: Ubuntu 2.6.32-22.36-server 2.6.32.11+drm33.2
> Uname: Linux 2.6.32-22-server x86_64
> Architecture: amd64
> Date: Thu Jun 17 08:59:01 2010
> ExecutablePath: /usr/sbin/libvirtd
> InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
> ProcAttrCurrent: /usr/sbin/libvirtd (enforce)
> ProcEnviron: PATH=(custom, no user)
> SourcePackage: libvirt
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/595388/+subscribe

--
Lars Täuber <email address hidden>