glusterfs-server 3.13.2-1build1 installation fails in Bionic because starting up the glustereventsd.service times out

[Szilard Cserey]

[Impact]

glusterfs-server is not "installable" in Bionic/18.04 LTS.

[Test case]

1. Run Bionic
2. Install glusterfs-server

--
apt-get install glusterfs-server

...

Job for glustereventsd.service failed because a timeout was exceeded.
See "systemctl status glustereventsd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript glustereventsd, action "start" failed.
● glustereventsd.service - LSB: Gluster Events Server
   Loaded: loaded (/etc/init.d/glustereventsd; generated)
   Active: failed (Result: timeout) since Mon 2019-05-27 09:52:15 UTC; 15ms ago
     Docs: man:systemd-sysv-generator(8)
  Process: 17909 ExecStart=/etc/init.d/glustereventsd start (code=killed, signal=TERM)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/glustereventsd.service
           ├─17927 /usr/bin/python /usr/sbin/glustereventsd -p /var/run/glustereventsd.pid
           └─17931 /usr/bin/python /usr/sbin/glustereventsd -p /var/run/glustereventsd.pid

May 27 09:47:14 vm1 systemd[1]: Starting LSB: Gluster Events Server...
May 27 09:47:14 vm1 glustereventsd[17909]: * Starting glustereventsd service glustereventsd
dpkg: error processing package glusterfs-server (--configure):
 installed glusterfs-server package post-installation script subprocess returned error exit status 1
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for systemd (237-3ubuntu10.15) ...
Errors were encountered while processing:
 glusterfs-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

dpkg -l | grep gluster
ii glusterfs-client 3.13.2-1build1 amd64 clustered file-system (client package)
ii glusterfs-common 3.13.2-1build1 amd64 GlusterFS common libraries and translator modules
iF glusterfs-server 3.13.2-1build1 amd64 clustered file-system (server package)

systemctl status glustereventsd.service
● glustereventsd.service - LSB: Gluster Events Server
   Loaded: loaded (/etc/init.d/glustereventsd; generated)
   Active: failed (Result: timeout) since Mon 2019-05-27 09:52:15 UTC; 2min 9s ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/glustereventsd.service
           ├─17927 /usr/bin/python /usr/sbin/glustereventsd -p /var/run/glustereventsd.pid
           └─17931 /usr/bin/python /usr/sbin/glustereventsd -p /var/run/glustereventsd.pid

May 27 09:47:14 vm1 systemd[1]: Starting LSB: Gluster Events Server...
May 27 09:47:14 vm1 glustereventsd[17909]: * Starting glustereventsd service glustereventsd
May 27 09:52:15 vm1 systemd[1]: glustereventsd.service: Start operation timed out. Terminating.
May 27 09:52:15 vm1 systemd[1]: glustereventsd.service: Failed with result 'timeout'.
May 27 09:52:15 vm1 systemd[1]: Failed to start LSB: Gluster Events Server.
--

[Regression Potential]

The fix is dropping the sysv/upstart approach in favor of systemd approach.

Most modern debian/ubuntu package of glusterfs-server have glusterd & glustereventsd disabled voluntary, and require manual operation to either start it or enable it.

Since it's already in place is Cosmic and late as-is, it makes sense to me to stay consistent with what we have in later version.

All documentation I have seen so far, indicate that glustereventsd need to be enable using systemctl enable too, so that approach seems align with official documentation found online.

The only downside, that I can see, is that one shouldn't expect "glusterd" and "glustereventsd" to be up and running after the installation.

It's a behaviour change per see, but in fact, since the current package never been into a installable/useable state in Bionic, then it's a behaviour change that no one (or very little for those who might have workaround the install issue) will notice since again nobody was able to install the package in Bionic so far.

The other option will be to upgrade the package to a non EOL version, but same change behaviour will occur (same approach is used) but + code change. I don't know how this code change may or may not impact Bionic user, this will require more analysis.

The upgrade can also be part of a separate discussion, let's focus for now on fixing the installation IMHO.

Additionally, the package have never been able to be installed successfully in Bionic, so who knows what bugs this version might have, once installed. Of course, I did and will continue to do more testing once found in bionic-proposed, but if a bug arise, this will need to be fix as any other userspace bug. I don't see a big problem here.

[Other Info]

# General info about events api:
https://docs.gluster.org/en/latest/Administrator%20Guide/Events%20APIs/

Events API feature is integrated with GD2. Webhooks can be registered to listen for GlusterFS events.

# Bug itself:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=875651
https://launchpad.net/bugs/1712748

# EOL related:
https://www.gluster.org/release-schedule/
https://docs.gluster.org/en/latest/release-notes/4.0.0/
As 3.13 was a short term maintenance release, features which have been included in that release are available with 4.0.0 as well.

[Original Description]

glusterfs-server installation fails on Bionic

apt-get install glusterfs-server

...

Job for glustereventsd.service failed because a timeout was exceeded.
See "systemctl status glustereventsd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript glustereventsd, action "start" failed.
● glustereventsd.service - LSB: Gluster Events Server
   Loaded: loaded (/etc/init.d/glustereventsd; generated)
   Active: failed (Result: timeout) since Mon 2019-05-27 09:52:15 UTC; 15ms ago
     Docs: man:systemd-sysv-generator(8)
  Process: 17909 ExecStart=/etc/init.d/glustereventsd start (code=killed, signal=TERM)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/glustereventsd.service
           ├─17927 /usr/bin/python /usr/sbin/glustereventsd -p /var/run/glustereventsd.pid
           └─17931 /usr/bin/python /usr/sbin/glustereventsd -p /var/run/glustereventsd.pid

May 27 09:47:14 vm1 systemd[1]: Starting LSB: Gluster Events Server...
May 27 09:47:14 vm1 glustereventsd[17909]: * Starting glustereventsd service glustereventsd
dpkg: error processing package glusterfs-server (--configure):
 installed glusterfs-server package post-installation script subprocess returned error exit status 1
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for systemd (237-3ubuntu10.15) ...
Errors were encountered while processing:
 glusterfs-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

dpkg -l | grep gluster
ii glusterfs-client 3.13.2-1build1 amd64 clustered file-system (client package)
ii glusterfs-common 3.13.2-1build1 amd64 GlusterFS common libraries and translator modules
iF glusterfs-server 3.13.2-1build1 amd64 clustered file-system (server package)

systemctl status glustereventsd.service
● glustereventsd.service - LSB: Gluster Events Server
   Loaded: loaded (/etc/init.d/glustereventsd; generated)
   Active: failed (Result: timeout) since Mon 2019-05-27 09:52:15 UTC; 2min 9s ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/glustereventsd.service
           ├─17927 /usr/bin/python /usr/sbin/glustereventsd -p /var/run/glustereventsd.pid
           └─17931 /usr/bin/python /usr/sbin/glustereventsd -p /var/run/glustereventsd.pid

May 27 09:47:14 vm1 systemd[1]: Starting LSB: Gluster Events Server...
May 27 09:47:14 vm1 glustereventsd[17909]: * Starting glustereventsd service glustereventsd
May 27 09:52:15 vm1 systemd[1]: glustereventsd.service: Start operation timed out. Terminating.
May 27 09:52:15 vm1 systemd[1]: glustereventsd.service: Failed with result 'timeout'.
May 27 09:52:15 vm1 systemd[1]: Failed to start LSB: Gluster Events Server.

The issue is also reported here:
https://github.com/gluster/glusterfs-debian/issues/22

[Eric Desrochers (slashd)]

If I read this correctly[0], the glusterfs version found in Bionic is EOL at upstream point of view.

For now, I would say, let's first investigate the root cause of the actual problem as usual, regardless the fact that this is EOL upstream-wise.

... but while I don't know what possible impact a newer version could have on Bionic users, it's definitely worth also investigating if upgrading glusterfs for Bionic to use a non-EOL version is viable with good rationales.

# rmadison
 glusterfs-server | 3.13.2-1build1 | bionic/universe | amd64, arm64, armhf, i386, ppc64el, s390x
 glusterfs-server | 4.1.2-1 | cosmic/universe | amd64, arm64, armhf, i386, ppc64el, s390x
 glusterfs-server | 5.3-2 | disco/universe | amd64, arm64, armhf, i386, ppc64el, s390x

[0] - https://www.gluster.org/release-schedule/

[Eric Desrochers (slashd)]

Interesting fact, while testing the installation of glusterfs-server in Cosmic:

# COSMIC
Adding group `gluster' (GID 117) ...
Done.
Setting up glusterfs-client (4.1.2-1) ...
Setting up glusterfs-server (4.1.2-1) ...
glusterd.service is a disabled or a static unit, not starting it.
glustereventsd.service is a disabled or a static unit, not starting it.
Processing triggers for libc-bin (2.28-0ubuntu1) ...
Processing triggers for systemd (239-7ubuntu10.13) ...

# COSMIC: debian/rules
+override_dh_systemd_enable:
+ dh_systemd_enable --no-enable --name=glusterd
+ dh_systemd_enable --no-enable --name=glustereventsd

In Cosmic glustereventsd is disabled by default after installation, as oppose to Bionic.
Maybe there is a reasons for that behaviour change at installation. Might be worth to investigate further that aspect IMHO.

[Eric Desrochers (slashd)]

more details about "Events API":
https://docs.gluster.org/en/latest/Administrator%20Guide/Events%20APIs/

[Eric Desrochers (slashd)]

Looking later glusterfs-server package from Cosmic and late, the tendency is to have glustereventsd disabled, in order to stay consistent with the more modern package, this can be one avenue to fix the situation

Of course, I haven't look nor discuss about upgrading to a more recent (non-eol version).
But this eol/non-eol discussion can still happen separately later on even after we have fix that particular install breakage issue.

I built a test package with the disabling glustereventsd part, and installed it in a test container:

...
glusterd.service is a disabled or a static unit, not starting it.
glustereventsd.service is a disabled or a static unit, not starting it.
...

$ systemctl is-enabled glustereventsd.service
disabled

If I start it manually, everything works now, and the installation goes smooth.

$ systemctl start glustereventsd.service

$ systemctl status glustereventsd.service
● glustereventsd.service - Gluster Events Notifier
Loaded: loaded (/lib/systemd/system/glustereventsd.service; disabled; vendor preset: enabled)
Active: active (running) since Mon 2019-05-27 17:47:14 UTC; 1s ago
Main PID: 6175 (glustereventsd)
Tasks: 2 (limit: 4915)
CGroup: /system.slice/glustereventsd.service
├─6175 /usr/bin/python2 /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid
└─6176 /usr/bin/python2 /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid

The only downside, is that one shouldn't expect glusterfseventsd to be up and running after the installation, but it's better that current situation where nothing is installable at all.

[Łukasz Zemczak (sil2100)]

Hello Szilard, or anyone else affected,

Accepted glusterfs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glusterfs/3.13.2-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Eric Desrochers (slashd)]

[VERIFICATION BIONIC]

Installation works fine using the proposed package.
Both glusterd and glustereventsd are disabled by default and can easily be enabled (permanently) and/or simply started manually (temporarily), exactly like successor package starting with Cosmic and late.

# apt-get install glusterfs-server -y
...
Setting up python-chardet (3.0.4-1) ...
Setting up python-cryptography (2.1.4-1ubuntu1.3) ...
Setting up python-requests (2.18.4-2ubuntu0.1) ...
Setting up python-openssl (17.5.0-1ubuntu1) ...
Setting up glusterfs-common (3.13.2-1ubuntu1) ...
Adding group `gluster' (GID 116) ...
Done.
Setting up glusterfs-client (3.13.2-1ubuntu1) ...
Setting up glusterfs-server (3.13.2-1ubuntu1) ...
glusterd.service is a disabled or a static unit, not starting it.
glustereventsd.service is a disabled or a static unit, not starting it.
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for systemd (237-3ubuntu10.21) ...

# dpkg -l | grep -i gluster
ii glusterfs-client 3.13.2-1ubuntu1 amd64 clustered file-system (client package)
ii glusterfs-common 3.13.2-1ubuntu1 amd64 GlusterFS common libraries and translator modules
ii glusterfs-server 3.13.2-1ubuntu1 amd64 clustered file-system (server package)

# systemd unit "glustereventsd"

# systemctl status glustereventsd.service
● glustereventsd.service - Gluster Events Notifier
   Loaded: loaded (/lib/systemd/system/glustereventsd.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

# systemctl start glustereventsd.service

# systemctl status glustereventsd.service
● glustereventsd.service - Gluster Events Notifier
   Loaded: loaded (/lib/systemd/system/glustereventsd.service; disabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-06-03 20:01:01 UTC; 1s ago
 Main PID: 5950 (glustereventsd)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/glustereventsd.service
           ├─5950 /usr/bin/python2 /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid
           └─5951 /usr/bin/python2 /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid

Jun 03 20:01:01 glustersproposed systemd[1]: Started Gluster Events Notifier.

# systemd unit "glusterd"

# systemctl status glusterd.service
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/lib/systemd/system/glusterd.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

# systemctl starts glusterd.service

# systemctl status glusterd.service
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/lib/systemd/system/glusterd.service; disabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-06-03 20:00:31 UTC; 6s ago
  Process: 5858 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=e
 Main PID: 5859 (glusterd)
    Tasks: 8 (limit: 4915)
   CGroup: /system.slice/glusterd.service
           └─5859 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

Jun 03 20...

[Eric Desrochers (slashd)]

The following has been brought to my attention by a impacted Ubuntu user:

"Yes, we can install the updated package from proposed... thanks 😊"

[Launchpad Janitor (janitor)]

This bug was fixed in the package glusterfs - 3.13.2-1ubuntu1

---------------
glusterfs (3.13.2-1ubuntu1) bionic; urgency=medium

  * debian/rules: Drop init script in favor of systemd unit.
    This also prevent the package to break during the installation
    and end up in half-configured state. (LP: #1830615)

  * debian/glusterfs-server.maintscript:
    - Remove obsolete conffile

 -- Eric Desrochers <email address hidden> Fri, 31 May 2019 16:14:29 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for glusterfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.