package libvirt-bin=1.3.1-1ubuntu1 fails to install due to new virtlockd initd script

[Mikhail S Medvedev]

On Ubuntu Xenial, new release of libvirt-bin fails to cleanly install:

root@xenial-server-2016-01-18-ppc64el:~# apt-get install libvirt-bin
--snip--
Setting up libvirt-bin (1.3.1-1ubuntu1) ...
Adding group `libvirtd' (GID 119) ...
Done.
 * Starting libvirt logging daemon virtlogd [ OK ]
Starting virtlockd daemon: /etc/init.d/virtlockd: 46: /etc/init.d/virtlockd: daemon: not found

invoke-rc.d: initscript virtlockd, action "start" failed.
dpkg: error processing package libvirt-bin (--configure):
 subprocess installed post-installation script returned error exit status 127
Setting up libxml2-utils (2.9.3+dfsg1-1) ...
Setting up pm-utils (1.4.1-16) ...
Processing triggers for libc-bin (2.21-0ubuntu6) ...
Processing triggers for systemd (229-1ubuntu2) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for dbus (1.10.6-1ubuntu2) ...
Errors were encountered while processing:
 libvirt-bin
E: Sub-process /usr/bin/dpkg returned an error code (1)

After installing 'daemon' package, the different error would show up indicating that init.d script uses parameters
that daemon can not handle:

Setting up libvirt-bin (1.3.1-1ubuntu1) ...
 * Starting libvirt logging daemon virtlogd [ OK ]
Starting virtlockd daemon: daemon: unrecognized option '--check'
usage: daemon [options] [--] [cmd arg...]

root@xenial-server-2016-01-18-ppc64el:~# daemon --version
daemon-0.6.4

I could not figure out exactly what version of daemon it expects. The failing source package: https://launchpad.net/ubuntu/xenial/ppc64el/libvirt-bin/1.3.1-1ubuntu1

Offending portion of /etc/init.d/virtlockd:

start() {
    echo -n "Starting $SERVICE daemon: "
    daemon --pidfile $PIDFILE --check $SERVICE $PROCESS --daemon $VIRTLOCKD_ARGS
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$SERVICE
}

Additional Info:

root@xenial-server-2016-01-18-ppc64el:~# lsb_release -rd
Description: Ubuntu Xenial Xerus (development branch)
Release: 16.04

root@xenial-server-2016-01-18-ppc64el:~# apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 1.3.1-1ubuntu1
  Candidate: 1.3.1-1ubuntu1
  Version table:
 *** 1.3.1-1ubuntu1 500
        500 http://ports.ubuntu.com/ubuntu-ports xenial/main ppc64el Packages
        100 /var/lib/dpkg/status

[Mikhail S Medvedev (msmedved)]

This is still happening for for https://launchpad.net/ubuntu/+source/libvirt/1.3.1-1ubuntu2

[Serge Hallyn (serge-hallyn)]

Install (... really, upgrade) did not fail for me, but virtlogd frequently fails to start on boot for me, leading to mysterious failures starting VMs. Starting it by hand fixes it.

[Mikhail S Medvedev (msmedved)]

Serge, are you testing on Xenial? If so, could you share output of 'daemon --version' or 'dpkg -l daemon'?

The problem for me is primarily not that I care about virlockd running, but that because of failure to start it, apt install fails. Thus, if you depend on libvirt package to be installed without failure (e.g. when you run automated OpenStack tests in a VM), you are out of luck.

[Serge Hallyn (serge-hallyn)]

daemon is not installed here.

[Mikhail S Medvedev (msmedved)]

So without the daemon, I would expect you should not be able to do '/etc/init.d/virtlockd start'. Why apt-get install is not failing for you? What distro are you installing on? And can you confirm you are installing libvirt-bin=1.3.1-1ubuntu2?

[Serge Hallyn (serge-hallyn)]

Quoting Mikhail S Medvedev (<email address hidden>):
> So without the daemon, I would expect you should not be able to do
> '/etc/init.d/virtlockd start'. Why apt-get install is not failing for

systemctl start virtlockd works fine

now interestingly,

0 ✓ serge@sl ~ $ sudo systemctl stop virtlockd
[sudo] password for serge:
Warning: Stopping virtlockd.service, but it can still be activated by:

so it's supposed to be socket activated. So if it's not getting
started when libvirt needs it, well that's messed up.

> you? What distro are you installing on? And can you confirm you are
> installing libvirt-bin=1.3.1-1ubuntu2?

ii libvirt-bin 1.3.1-1ubuntu2 amd64 programs for the libvirt library

[Mikhail S Medvedev (msmedved)]

Tested on x86 Xenial VM, and libvirt-bin=1.3.1 did install without a problem.

So install of libvirt-bin only fails on ppc64el build of Xenial. I do not see a way to tag this bug as ppc64el.

Serge - I see the same output as you showed for systemctl commands when I run on x86.

[Stefan Bader (smb)]

Oddly sounds like on ppc64el the build creates no systemd files, so the sysvinit script gets used and I was not able to test this. Not sure whether its possible to try modifying the init.d script to have the full path assigned to the PROCESS variable. Running apt-get install -f probably overwrites this again...

[Mikhail S Medvedev (msmedved)]

The init.d/virlockd script initially fails due to 'daemon' being absent. So setting PROCESS to use full path has no effect (I have just tested it).

[Stefan Bader (smb)]

Thanks Mikhail, it was not clear to me whether absent means really not there or just failing to start because it is not found (missing path setting or needing full path for the process). I will try to find a ppc64el porter today to see what the build there actually does.

[Stefan Bader (smb)]

So finding a machine turns out a bit more complicated. Anyhow, I looked into the libvirt-bin binary package for ppc64le and weirdly it does also contain the /lib/systemd/system/virtlockd.[service,socket] files. So why is the init.d script being run? Also the virtlockd should be there /usr/sbin/virtlockd. Question is which way the scripts needs to be modified. Could be that PROCESS=virtlockd is ok but it needs to set and export a PATH environment which also contains /usr/sbin (like the virtlogd one does).
I must admit I had to fix other issues (found by lintian) in that script which I copied from the upstream source tree. Just testing is a problem as on x86 with systemd and sysvinit files around the systemd variant always wins.

[Mikhail S Medvedev (msmedved)]

I should have provided the postinst trace right away. Hope that helps to narrow it down:

root@xenial-server-2016-01-18-ppc64el:~# dpkg -D777 --configure libvirt-bin
  <snip>
D000040: ok 2 msgs >><<
D000040: checking Breaks
D000400: checking breaker apparmor:ppc64el virtbroken <none>
Setting up libvirt-bin (1.3.1-1ubuntu2) ...
D000002: fork/exec /var/lib/dpkg/info/libvirt-bin.postinst ( configure )
 * Starting libvirt logging daemon virtlogd [ OK ]
Starting virtlockd daemon: /etc/init.d/virtlockd: 46: /etc/init.d/virtlockd: daemon: not found

invoke-rc.d: initscript virtlockd, action "start" failed.
dpkg: error processing package libvirt-bin (--configure):
 subprocess installed post-installation script returned error exit status 127
D000001: ensure_diversions: same, skipping
Errors were encountered while processing:
 libvirt-bin

root@xenial-server-2016-01-18-ppc64el:~# sh -x /var/lib/dpkg/info/libvirt-bin.postinst configure
+ set -e
+ add_users_groups
+ getent group libvirtd
+ sed -e s/^.*:// -e s/,/ /g
+ getent group admin
+ sed -e s/^.*:// -e s/,/ /g
+ getent group sudo
+ adduser ubuntu libvirtd
+ getent group kvm
+ getent passwd libvirt-qemu
+ getent passwd libvirt-dnsmasq
+ add_statoverrides
+ ROOT_DIRS= /var/lib/libvirt/images/ /var/lib/libvirt/boot/ /var/cache/libvirt/
+ QEMU_DIRS= /var/lib/libvirt/qemu/ /var/cache/libvirt/qemu/
+ SANLOCK_DIR=/var/lib/libvirt/sanlock
+ QEMU_CONF=/etc/libvirt/qemu.conf
+ dpkg-statoverride --list /var/lib/libvirt/images/
+ [ -d /var/lib/libvirt/images/ ]
+ chown root:root /var/lib/libvirt/images/
+ chmod 0711 /var/lib/libvirt/images/
+ dpkg-statoverride --list /var/lib/libvirt/boot/
+ [ -d /var/lib/libvirt/boot/ ]
+ chown root:root /var/lib/libvirt/boot/
+ chmod 0711 /var/lib/libvirt/boot/
+ dpkg-statoverride --list /var/cache/libvirt/
+ [ -d /var/cache/libvirt/ ]
+ chown root:root /var/cache/libvirt/
+ chmod 0711 /var/cache/libvirt/
+ dpkg-statoverride --list /var/lib/libvirt/qemu/
+ [ -d /var/lib/libvirt/qemu/ ]
+ chown libvirt-qemu:kvm /var/lib/libvirt/qemu/
+ chmod 0750 /var/lib/libvirt/qemu/
+ dpkg-statoverride --list /var/cache/libvirt/qemu/
+ [ -d /var/cache/libvirt/qemu/ ]
+ chown libvirt-qemu:kvm /var/cache/libvirt/qemu/
+ chmod 0750 /var/cache/libvirt/qemu/
+ dpkg-statoverride --list /var/lib/libvirt/sanlock
+ [ -d /var/lib/libvirt/sanlock ]
+ chown root:root /var/lib/libvirt/sanlock
+ chmod 0700 /var/lib/libvirt/sanlock
+ dpkg-statoverride --list /etc/libvirt/qemu.conf
+ [ -f /etc/libvirt/qemu.conf ]
+ chown root:root /etc/libvirt/qemu.conf
+ chmod 0600 /etc/libvirt/qemu.conf
+ [ -n ]
+ dpkg --compare-versions lt 0.6.1-2
+ [ -e /etc/rc2.d/S20libvirt-bin ]
+ chown libvirt-qemu:kvm /var/lib/libvirt/qemu/channel/target
+ profile=/etc/apparmor.d/usr.sbin.libvirtd
+ [ -f /etc/apparmor.d/usr.sbin.libvirtd ]
+ aa-status --enabled
+ profile=/etc/apparmor.d/usr.lib.libvirt.virt-aa-helper
+ [ -f /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper ]
+ aa-status --enabled
+ touch /var/log/libvirt/qemu/.placeholder
+ touch /var/log/libvirt/uml/.placeho...

[Stefan Bader (smb)]

Doh! Yeah well, yes. I missed that when looking at it last night. Of course "daemon" as executable or function to start a daemon does not exist in Debian/Ubuntu. Still why it is run at all is odd. At least on x86 systemd is taking over so hardly that even when I try to run /etc/init.d/virtlockd start it gets converted into a systemctl call...
But I think I know now what to do about "daemon not found".

[Stefan Bader (smb)]

I'd really like to to verify this really works before uploading a new version, but getting hw access proves a bit hard. So if you could try the attached script as replacement for /etc/init.d/virtlockd

[Mikhail S Medvedev (msmedved)]

Stefan - the init.d script you provided fixes the package install on ppc64el Xenial! See full apt install log attached. Excerpt:

  Adding group `libvirtd' (GID 119) ...
  Done.
   * Starting libvirt logging daemon virtlogd [ OK ]
   * Starting libvirt locking daemon virtlockd [ OK ]
  libvirt-bin start/running, process 9713
  Setting up libvirt-bin dnsmasq configuration.

Thank you for working on this and for timely response! A new release with the fix would be greatly appreciated :)

This bug brought up the fact that there are differences between package install code paths for x86 and ppc64. Should that be a concern at this stage before 16.04 is out? We could investigate this further.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 1.3.1-1ubuntu4

---------------
libvirt (1.3.1-1ubuntu4) xenial; urgency=low

  * d/libvirt-bin.virtlockd.init: Replace by the version I had already
    prepared and was tested (LP: #1547208).
  * d/libvirt-bin.virtlogd.init: Fix up some left-over references to
    libvirtd.
  * d/control: Add provides libvirt-daemon for libvirt-bin (LP: #1551643)

 -- Stefan Bader <email address hidden> Tue, 01 Mar 2016 10:58:23 +0100

[strk (strk)]

It's still failing for me:
# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up libvirt-bin (1.3.1-1ubuntu10.5) ...
Job for libvirt-bin.service failed because the control process exited with error code. See "systemctl status libvirt-bin.service" and "journalctl -xe" for details.
invoke-rc.d: initscript libvirt-bin, action "restart" failed.
dpkg: error processing package libvirt-bin (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 libvirt-bin
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Mikhail S Medvedev (msmedved)]

strk, are you upgrading the package or installing it fresh? I've seen a couple of times libvirt package did fail to update because of service restart failure. I did work around it by 'pkill -f libvirtd' before the upgrade.