libvirtd upstart job should handle host reboot/shutdown

[Dmitry Teselkin]

The upstart job must give the running VMs an opportunity to handle the host shutdown (flush the outstanding writes to the disk).
Otherwise the following happens during the host reboot/shutdown
 - upstarts stops libvirtd (SIGTERM, pause, SIGKILL, unless pre-stop and stop actions defined), qemu processes powering the VMs are still running
 - upstart stops other services
 - upstart stops the remaining processes, including qemu ones: SIGTERM, pause, SIGKILL

The problem is that the pause (a few seconds) is way too short for flushing the data, especially if there are several VMs running,
so the host reboot/shutdown can easily corrupt VMs' filesystems.

Suggested fix: gracefully shutdown domains in the pre-stop script (similarly to libvirt-bin package shipped by Canonical)

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Roman Podoliaka (rpodolyaka)]

Dima, actually Canonical ships a separate upstart script for that - libvirt-guests

[Ivan Suzdal (isuzdal)]

Frankly speaking, we are shipping libvirt-guests service (even if it's not upstart job but sysv). Perharps this issue related to puppet manifests.

[Albert Syriy (asyriy)]

Actually since libvirtd and libvirt-guests services have been separated
(enabling don't power off domains/hosts when libvirtd service is rebooting)
I don't see any reason to tie up libvirtd and libvirt-guests back. Do you?

Albert Syriy,

Software Engineer,
Mirantis

On Fri, May 13, 2016 at 3:59 PM, Ivan Suzdal <email address hidden> wrote:

> Frankly speaking, we are shipping libvirt-guests service (even if it's
> not upstart job but sysv). Perharps this issue related to puppet
> manifests.
>
> --
> You received this bug notification because you are a member of MOS
> Linux, which is a bug assignee.
> https://bugs.launchpad.net/bugs/1581129
>
> Title:
> libvirtd upstart job should handle host reboot/shutdown
>
> Status in Mirantis OpenStack:
> Confirmed
> Status in Mirantis OpenStack 10.0.x series:
> Confirmed
> Status in Mirantis OpenStack 9.0.x series:
> Confirmed
>
> Bug description:
> The upstart job must give the running VMs an opportunity to handle the
> host shutdown (flush the outstanding writes to the disk).
> Otherwise the following happens during the host reboot/shutdown
> - upstarts stops libvirtd (SIGTERM, pause, SIGKILL, unless pre-stop and
> stop actions defined), qemu processes powering the VMs are still running
> - upstart stops other services
> - upstart stops the remaining processes, including qemu ones: SIGTERM,
> pause, SIGKILL
>
> The problem is that the pause (a few seconds) is way too short for
> flushing the data, especially if there are several VMs running,
> so the host reboot/shutdown can easily corrupt VMs' filesystems.
>
> Suggested fix: gracefully shutdown domains in the pre-stop script
> (similarly to libvirt-bin package shipped by Canonical)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mos/+bug/1581129/+subscriptions
>

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/20849

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20849
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: bff99ef3a5a029d97f7798c563a7f90d2a6dda3f
Author: Ivan Suzdal <email address hidden>
Date: Wed May 18 16:17:34 2016

Shutdown guests from libvirtd upstart job.

As in upstream version, we need to gently shut down
virtual guests when system is going to
reboot or shutdown.

Change-Id: I320a9c91e6d96cbcda84552b694d2a59fc482e40
Closes-Bug: #1581129

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/21409

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/21409
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 39217c5f29d1f90643aac527380ec8358ea6bd8c
Author: Ivan Suzdal <email address hidden>
Date: Tue May 31 10:17:55 2016

Shutdown guests from libvirtd upstart job.

As in upstream version, we need to gently shut down
virtual guests when system is going to
reboot or shutdown.

Change-Id: I320a9c91e6d96cbcda84552b694d2a59fc482e40
Closes-Bug: #1581129
(cherry picked from commit bff99ef3a5a029d97f7798c563a7f90d2a6dda3f)

[Alexander Gubanov (ogubanov)]

Verified on MOS 9.0 (build 445) - fixed!
Proof:

root@node-5:~# grep -A 12 'pre-stop script' /etc/init/libvirtd.conf
pre-stop script
# shutdown guests if system is going to reboot or shutdown
    if [ -z "$RUNLEVEL" ]; then
        exit 0
    fi

    if [ "$RUNLEVEL" -ne 0 ] && [ "$RUNLEVEL" -ne 1 ] && [ "$RUNLEVEL" -ne 6 ]; then
        exit 0
    fi

    /usr/lib/libvirt/libvirt-guests.sh stop
end script