"systemctl enable corosync-qdevice.service" fails

[Don van der Haghen]

[Impact]

* corosync-qdevice.init does not contain default start run levels

* if corosync-qdevice is being started by the sysv generator, by any reason, it will fail to start automatically.

* As Focal is a LTS, I thought it would be better to have it fixed (even meaning an upgrade to those not relying in systemd sysv generator).

[Test Case]

* enable corosync-qdevice.init through systemd sysv generator instead of using the corosync-qdevice.service unit

[Regression Potential]

* little potential as it changes default init runlevels only

[Other Info]

 * Original Description:

"systemctl enable corosync-qdevice.service" fails:
Synchronizing state of corosync-qdevice.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable corosync-qdevice
update-rc.d: error: corosync-qdevice Default-Start contains no runlevels, aborting.

Reason is that an init.d script as well as a systemd service definition exists:
# dpkg -L corosync-qdevice
/lib/systemd/system/corosync-qdevice.service
/etc/init.d/corosync-qdevice

Therefore, using pcs to add a qdevice to the cluster fails:
pcs quorum device add model net host=qdevice-host algorithm=ffsplit
Log:
I, [2018-12-24T22:39:00.208158 #149] INFO -- : Running: systemctl enable corosync-qdevice.service
I, [2018-12-24T22:39:00.208322 #149] INFO -- : CIB USER: hacluster, groups:
I, [2018-12-24T22:39:00.463878 #149] INFO -- : Return Value: 1
E, [2018-12-24T22:39:00.464110 #149] ERROR -- : Enabling corosync-qdevice failed

Workaround is to manually remove the "/etc/init.d/corosync-qdevice" script.

Solution is to remove the "/etc/init.d/corosync-qdevice" file from the "corosync-qdevice" package.

[Don van der Haghen (donvdh)]

I forgot to mention, this bug report concerns:
Ubuntu 18.04
Package: corosync-qdevice
Version: 2.4.3-0ubuntu1

[Andreas Hasenack (ahasenack)]

Bug confirmed. It's an odd looking initscript, as it does not have the usual actions in itself.

[Don van der Haghen (donvdh)]

Attached debdiff, based on corosync (2.4.3-0ubuntu1) for Ubuntu 18.04/bionic removes the debian/corosync-qdevice.init script from the package. If already installed previously, it removes the /etc/init.d/corosync-qdevice script through the corosync-qdevice.postinst script upon package upgrade.

I also tried to reproduce the issue on Debian 9.7.0: On Debian the package also includes and installs the same script. However on Debian systemctl prefers to use systemd service definitions instead of the init scripts, therefore the issue doesn't occur on Debian.

I would like to kindly ask a sponsor to review the patch and let me know which changes would be required to get it accepted.

Thank you very much in advance.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "bionic-fix-for-corosync-qdevice-lp1809682.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Simon Quigley (tsimonq2)]

Hello Don, thank you for your work on this!

Since Bionic is a stable release, the description needs to be updated to use the SRU bug template, which can be found here: https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

Also, did this only affect Bionic? If it affects newer releases, we should get that handled there as well.

Unsubscribing the Ubuntu Sponsors Team, please resubscribe when that's done.

[Rafael David Tinoco (rafaeldtinoco)]

Actually removing the sysv script is not an acceptable fix for this error. I mean, it might come to a day where we will disable all sysv inits, but, for now, they are called by systemd through systemd generator mechanism (https://www.freedesktop.org/software/systemd/man/systemd-sysv-generator.html) and the fix for this bug is:

change /etc/init.d/corosync-qdevice header to:

### BEGIN INIT INFO
# Provides: corosync-qdevice
# Required-Start: $remote_fs $syslog corosync
# Required-Stop: $remote_fs $syslog corosync
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Corosync Qdevice daemon
# Description: Starts and stops Corosync Qdevice daemon.
### END INIT INFO

(adding 2 3 4 5 in Default-Start)

And updating rc*.d scripts:

(c)inaddy@temp:~$ sudo update-rc.d -f corosync-qdevice remove
(c)inaddy@temp:~$ sudo update-rc.d -f corosync-qdevice defaults

And then:

(c)inaddy@temp:~$ systemctl enable corosync-qdevice.service
Synchronizing state of corosync-qdevice.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink /etc/systemd/system/multi-user.target.wants/corosync-qdevice.service → /lib/systemd/system/corosync-qdevice.service.

With that, the service will be started again.

I'll provide a fix for this soon, after I finish the [main] Ubuntu HA packages first (that is why I moved this into "corosync-qdevice" and not "corosync" only).

Thank you.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Bert Driehuis (driehuis)]

Just as a side note, I'm not sure why sysvinit script is provided at all. The daemon seems to work just fine with the included systemd config file in /lib/systemd/system/corosync-qdevice.service. Unless there's a compelling reason to keep the sysv init script I'd ditch it, if only to cut on the potential for confusion.

[Rafael David Tinoco (rafaeldtinoco)]

Hello Bert,

We are not yet fully disabling sysv related scripts because there are systems that are not, yet, using systemd. Not for this case, but, one of these examples is windows WSL environment, where systemd is not supported yet. Just wanted to mention this to show there are some other things to be taken in consideration when dropping support, entirely, for sysv scripts, for example.

Just a quick example on corosync running inside WSL environment:

rafaeldtinoco@lenovo:~$ uname -a
Linux lenovo 4.4.0-18362-Microsoft #476-Microsoft Fri Nov 01 16:53:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

rafaeldtinoco@lenovo:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.10
Release: 19.10
Codename: eoan

$ sudo corosync -f
Mar 20 01:49:47 notice [MAIN ] Corosync Cluster Engine 3.0.1 starting up
Mar 20 01:49:47 info [MAIN ] Corosync built-in features: dbus monitoring watchdog augeas systemd xmlconf snmp pie relro bindnow
Mar 20 01:49:47 warning [MAIN ] Could not lock memory of service to avoid page faults: Function not implemented (38)
Mar 20 01:49:47 notice [TOTEM ] Initializing transport (UDP/IP Unicast).
Mar 20 01:49:47 warning [TOTEM ] Could not set traffic priority: Protocol not available (92)
Mar 20 01:49:47 notice [TOTEM ] The network interface [127.0.0.1] is now up.
Mar 20 01:49:47 notice [SERV ] Service engine loaded: corosync configuration map access [0]
Mar 20 01:49:47 info [QB ] server name: cmap
Mar 20 01:49:47 notice [SERV ] Service engine loaded: corosync configuration service [1]
Mar 20 01:49:47 info [QB ] server name: cfg
Mar 20 01:49:47 notice [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Mar 20 01:49:47 info [QB ] server name: cpg
Mar 20 01:49:47 notice [SERV ] Service engine loaded: corosync profile loading service [4]
Mar 20 01:49:47 notice [SERV ] Service engine loaded: corosync resource monitoring service [6]
Mar 20 01:49:47 warning [WD ] Watchdog not enabled by configuration
Mar 20 01:49:47 warning [WD ] resource load_15min missing a recovery key.
Mar 20 01:49:47 warning [WD ] resource memory_used missing a recovery key.
Mar 20 01:49:47 info [WD ] no resources configured.
Mar 20 01:49:47 notice [SERV ] Service engine loaded: corosync watchdog service [7]
Mar 20 01:49:47 notice [QUORUM] Using quorum provider corosync_votequorum
Mar 20 01:49:47 notice [QUORUM] This node is within the primary component and will provide service.
Mar 20 01:49:47 notice [QUORUM] Members[0]:
Mar 20 01:49:47 notice [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Mar 20 01:49:47 info [QB ] server name: votequorum
Mar 20 01:49:47 notice [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Mar 20 01:49:47 info [QB ] server name: quorum
Mar 20 01:49:47 notice [TOTEM ] adding new UDPU member {127.0.0.1}
Mar 20 01:49:47 notice [TOTEM ] A new membership (1:28) was formed. Members joined: 1
Mar 20 01:49:47 warning [CPG ] downlist left_list: 0 received
Mar 20 01:49:47 notice [QUORUM] Members[1]: 1
Mar 20 01:49:47 notice [MAIN ] Completed serv...

[Rafael David Tinoco (rafaeldtinoco)]

I'm closing up corosync / corosync-* bugs for Focal and will check sysv script tomorrow. For now, I can say that:

(c)rafaeldtinoco@clusterdev:~$ sudo systemctl status corosync-qdevice
● corosync-qdevice.service - Corosync Qdevice daemon
     Loaded: loaded (/lib/systemd/system/corosync-qdevice.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:corosync-qdevice

systemd is not defaulting to sysv generator anymore when systemd units are available, as you can see above, package corosync-qdevice installed defaults to systemd unit so having both simultaneously won't be a problem to new users (apart from having a tiny extra file at /etc/init.d/ directory).

[Rafael David Tinoco (rafaeldtinoco)]

In groovy:

(k)rafaeldtinoco@clusterg01:~$ sudo rm -rf /lib/systemd/system/corosync-qdevice.service
(k)rafaeldtinoco@clusterg01:~$ sudo rm -rf /etc/systemd/system/corosync-qdevice.service

(k)rafaeldtinoco@clusterg01:~$ sudo /etc/init.d/corosync-qdevice start

$ systemctl status corosync-qdevice
● corosync-qdevice.service - LSB: Corosync Qdevice daemon
     Loaded: loaded (/etc/init.d/corosync-qdevice; generated)
     Active: active (exited) since Sat 2020-07-11 03:21:23 UTC; 2s ago
       Docs: man:systemd-sysv-generator(8)
    Process: 62926 ExecStart=/etc/init.d/corosync-qdevice start (code=exited, status=0/SUCCESS)

Jul 11 03:21:23 clusterg01 systemd[1]: Starting LSB: Corosync Qdevice daemon...
Jul 11 03:21:23 clusterg01 corosync-qdevice[62926]: * Starting Corosync Qdevice daemon corosync-qdevice
Jul 11 03:21:23 clusterg01 systemd[1]: Started LSB: Corosync Qdevice daemon.

[Christoph Roeder (brightdroid)]

For me it's not fixed in focal:

# dpkg -l corosync*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-================-==============-============-===================================
ii corosync 3.0.3-2ubuntu2 amd64 cluster engine daemon and utilities
ii corosync-qdevice 3.0.0-4ubuntu1 amd64 cluster engine quorum device daemon

# systemctl enable corosync-qdevice.service
Synchronizing state of corosync-qdevice.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable corosync-qdevice
update-rc.d: error: corosync-qdevice Default-Start contains no runlevels, aborting.

[Christoph Roeder (brightdroid)]

workaround for now:

$ ln -st /etc/systemd/system/multi-user.target.wants/ /lib/systemd/system/corosync-qdevice.service

[Rafael David Tinoco (rafaeldtinoco)]

Hum, looks like systemd used your sysv scripts (perhaps because of upgrades) and it had issues because they had no runlevels by default... and your workaround was to use the systemd unit, which is the one used by default...

### BEGIN INIT INFO
# Provides: corosync-qdevice
# Required-Start: $remote_fs $syslog corosync
# Required-Stop: $remote_fs $syslog corosync
# Default-Start:
# Default-Stop: 0 1 6
# Short-Description: Corosync Qdevice daemon
# Description: Starts and stops Corosync Qdevice daemon.
### END INIT INFO

and indeed the sysv scripts don't have default start runlevels.

I should fix this.

[Rafael David Tinoco (rafaeldtinoco)]

Fixed in Debian:

commit 03f7f53
Author: Valentin Vidic <email address hidden>
Date: Tue Apr 7 20:19:07 2020

    Set Default-Start runlevels for corosync-qdevice (Closes: #932346)

and already present in Groovy and Focal.

Focal is missing this patch.

Bionic does not have corosync-qdevice package.

[Robie Basak (racb)]

Hello Don, or anyone else affected,

Accepted corosync-qdevice into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/corosync-qdevice/3.0.0-4ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Rafael David Tinoco (rafaeldtinoco)]

All good with published package. Good to go.

[Łukasz Zemczak (sil2100)]

Please be sure to include the version number of the tested package, as we should not be releasing updates without this information provided!

[Launchpad Janitor (janitor)]

This bug was fixed in the package corosync-qdevice - 3.0.0-4ubuntu1.1

---------------
corosync-qdevice (3.0.0-4ubuntu1.1) focal; urgency=medium

  * Set Default-Start runlevels for corosync-qdevice (LP: #1809682)

 -- Rafael David Tinoco <email address hidden> Tue, 08 Sep 2020 17:36:04 +0000

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for corosync-qdevice has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.