slapd process failure is not detected by systemd

[Heitor Alves de Siqueira]

[Impact]
Systemd service reports slapd as active, even though it may have failed

[Description]
The slapd package for OpenLDAP is shipped with a SysV-style init script (/etc/init.d/slapd). Systemd automatically converts this to a systemd service by generating the unit file using the systemd-sysv-generator(8) utility. The generated unit file contains Type=forking and RemainAfterExit=yes directives.

If the slapd daemon process exits due to some failure (e.g., it receives a SIGTERM or SIGKILL), the failure is not detected properly by systemd. The service is still reported as active even though the child (daemon) process has exited with a signal.

We can easily fix this by including a proper systemd service file for slapd in the openldap package. Since the init.d script already does most of the necessary work (parsing configs, setting up PID files, etc.), we don't need anything complicated for the systemd unit file. Just making sure that RemainAfterExit is set to "no" makes the systemd service behave in the expected way.

[Test Case]
1) Deploy a disco container
$ lxc launch images:ubuntu/disco disco

2) Install slapd
ubuntu@disco:~$ sudo apt update && sudo apt install slapd -y

3) Verify that slapd is running with the auto-generated service
ubuntu@disco:~$ systemctl status slapd
● slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
   Loaded: loaded (/etc/init.d/slapd; generated)
   Active: active (running) since Fri 2019-03-22 11:51:22 UTC; 40min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1103 ExecStart=/etc/init.d/slapd start (code=exited, status=0/SUCCESS)
    Tasks: 3 (limit: 4915)
   Memory: 712.6M
   CGroup: /system.slice/slapd.service
           └─1109 /usr/sbin/slapd -h ldap:/// ldapi:/// -g openldap -u openldap -F /etc/ldap/slapd.d

4) SIGKILL the slapd process (PID is displayed in systemctl status output)
ubuntu@disco:~$ sudo kill -9 1109

5) Check if systemd service lists slapd as still active, even though it was terminated
ubuntu@disco:~$ systemctl status slapd
● slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
   Loaded: loaded (/etc/init.d/slapd; generated)
   Active: active (exited) since Fri 2019-03-22 11:51:22 UTC; 42min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1103 ExecStart=/etc/init.d/slapd start (code=exited, status=0/SUCCESS)

6) Check if systemd has loaded both /run/systemd/generator.late/slapd.service & /usr/lib/systemd/system/slapd.service.d/slapd-remain-after-exit.conf

$ systemctl cat slapd

[Regression Potential]
The regression potential for this fix should be very low, if we keep the new systemd unit file close to the one generated by systemd-sysv-generator(8). The only significant change would be the RemainAfterExit directive, and this should make the slapd service behave like a "normal" forking service. Nonetheless, we'll perform scripted test runs to make sure no regressions arise.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "debdiff for disco" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Ryan Tandy (rtandy)]

Hello Hector,

On Fri, Mar 22, 2019 at 12:36:57PM -0000, Heitor R. Alves de Siqueira wrote:
>The slapd package for OpenLDAP is shipped with a SysV-style init script
>(/etc/init.d/slapd). Systemd automatically converts this to a systemd
>service by generating the unit file using the systemd-sysv-generator(8)
>utility. The generated unit file contains Type=forking and
>RemainAfterExit=yes directives.
>
>If the slapd daemon process exits due to some failure (e.g., it receives
>a SIGTERM or SIGKILL), the failure is not detected properly by systemd.
>The service is still reported as active even though the child (daemon)
>process has exited with a signal.
>
>We can easily fix this by including a proper systemd service file for
>slapd in the openldap package.

Do you need a whole service file for this? I thought you could achieve
the same with a drop-in that just overrides the required keys:

/etc/systemd/systems/slapd.service.d/remain-after-exit.conf:

[Service]
Type=forking
RemainAfterExit=no
Restart=on-failure

(untested, based on bug 1488962)

[Dan Streetman (ddstreet)]

> Do you need a whole service file for this? I thought you could achieve
> the same with a drop-in that just overrides the required keys

why would we do a SRU to hack in an override when it's just as easy to do a SRU to actually create a correct service file so no override hack is needed?

[Heitor Alves de Siqueira (halves)]

The sysv init script does some funky parsing of configuration files that might give us some trouble in an independent service file. I'll check if we can't work around that and do away with the sysv init script altogether, otherwise we might as well go with the drop-in override since the new service wouldn't be much different from it.

[Heitor Alves de Siqueira (halves)]

I've been doing some research on how we could implement a proper systemd unit file for slapd, and came up with interesting results.

There are a number of options that we can pass to the slapd service that will give us lots of troubles on an "idiomatic" service file. To give some examples:
- The PID file can be directly obtained from /etc/default/slapd if set, or it
  has to be parsed from /etc/ldap/slapd.d or /etc/ldap/slapd.conf. The /etc/ldap
  files don't follow the /etc/default syntax, so we can't just pass them with
  "EnvironmentFile=" in the systemd service. We would have to parse those files
  as is currently done by the init script.
- slapd has the "-f" and "-F" flags that can be used to specify a config file or
  directory, respectively. It does not accept both at the same time, but
  SLAPD_CONF in /etc/default/slapd allows either a file or directory. The init
  script will figure out which of those is set, and pass the correct flag
  accordingly. I don't see a simple way of doing this in systemd without either
  polluting the SLAPD_CONF option with the necessary flag, or delegating it to a
  helper script which would check whether the option meant a file or a
  directory.
- The slapd.conf file is used by other related tools (see slapd.conf(5)), which
  prevents us from pulling some of the options to another centralized file
  without also updating these tools

I hoped that the upstream package would have some insight on the proper way of writing the service file, but unfortunately it seems that upstream Openldap devs are very much against the idea of shipping them (be they sysv-init or systemd files). There are some discussions on the openldap lists, one of which deals specifically with adding systemd support for slapd [0]. From that discussion, it's clear that upstream openldap is not ready to provide the necessary systemd files out-of-the-box, even when approached with relevant patches.

I also checked how CentOS implement their slapd.service file since they don't seem to run into these problems. They don't use an autogenerated unit file, and have a proper one instead [1]. From what I've checked, this is easier for them for two reasons:
1) The CentOS configs for slapd are much simpler than what we have in Debian/Ubuntu (their defaults don't include most of the options that we do)
2) They use helper scripts for the funky parsing we have in the sysv-init script ("ExecStartPre=/usr/libexec/openldap/check-config.sh", which also invokes some other helper scripts)

On Debian/Ubuntu, the autogenerated unit file for slapd uses the sysv-init script as a "shim" to manage the service, which makes it possible to keep the whole config-parsing insanity consistent between sysv-init and systemd.
It's not the most elegant solution, but for now I believe it makes sense to include an override file for the "RemainAfterExit=" option until we have a native systemd unit file for slapd (similar to what was done for apache2 in bug 1488962 mentioned by Ryan Tandy).

[0] https://www.openldap.org/its/index.cgi/Incoming?id=8707
[1] https://git.centos.org/blob/rpms!openldap.git/c7/SOURCES!slapd.service

[Heitor Alves de Siqueira (halves)]

@slashd I don't think we need anything special for systemd in the debian/{control,rules} files, since we're not shipping full systemd support with this. The slapd service is still handled by systemd-sysv-generator(8), so there's no service units that we need to include or activate in dh_systemd rules.

To make sure things are working fine I went ahead and tested a package upgrade with this debdiff:

ubuntu@disco:~$ dpkg -l | grep slapd
ii slapd 2.4.47+dfsg-3ubuntu1 amd64 OpenLDAP server (slapd)

ubuntu@disco:~$ sudo apt-get --only-upgrade install slapd

ubuntu@disco:~$ dpkg -l | grep slapd
ii slapd 2.4.47+dfsg-3ubuntu2 amd64 OpenLDAP server (slapd)

ubuntu@disco:~$ systemctl cat slapd
# /run/systemd/generator.late/slapd.service
# Automatically generated by systemd-sysv-generator

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/slapd
Description=LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
Before=multi-user.target
Before=multi-user.target
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/slapd start
ExecStop=/etc/init.d/slapd stop

# /usr/lib/systemd/system/slapd.service.d/slapd-remain-after-exit.conf
[Service]
Type=forking
RemainAfterExit=no

Thanks!

[Heitor Alves de Siqueira (halves)]

debdiff v2 with fixed author information

[Eric Desrochers (slashd)]

Sponsored in disco.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.47+dfsg-3ubuntu2

---------------
openldap (2.4.47+dfsg-3ubuntu2) disco; urgency=medium

  * Fix sysv-generator unit file by customizing parameters (LP: #1821343)
    - d/slapd-remain-after-exit.conf: Override RemainAfterExit to allow
      correct systemctl status for slapd daemon.
    - d/slapd.install: place override file in correct location.

 -- Heitor Alves de Siqueira <email address hidden> Mon, 08 Apr 2019 12:39:12 -0300

[Łukasz Zemczak (sil2100)]

Hello Heitor, or anyone else affected,

Accepted openldap into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.4.46+dfsg-5ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Adam Conrad (adconrad)]

Hello Heitor, or anyone else affected,

Accepted openldap into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.4.45+dfsg-1ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Adam Conrad (adconrad)]

Hello Heitor, or anyone else affected,

Accepted openldap into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.4.42+dfsg-2ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Heitor Alves de Siqueira (halves)]

Verified according to test case in description for cosmic:

root@cosmic:~# dpkg -l | grep slapd
ii slapd 2.4.46+dfsg-5ubuntu1.2 amd64 OpenLDAP server (slapd)

root@cosmic:~# systemctl status slapd
● slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
   Loaded: loaded (/etc/init.d/slapd; generated)
  Drop-In: /lib/systemd/system/slapd.service.d
           └─slapd-remain-after-exit.conf
   Active: active (running) since Mon 2019-04-22 20:15:01 UTC; 19s ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 3 (limit: 4915)
   Memory: 612.7M
   CGroup: /system.slice/slapd.service
           └─2061 /usr/sbin/slapd -h ldap:/// ldapi:/// -g openldap -u openldap -F /etc/ldap/slapd.d

root@cosmic:~# kill -9 2061

root@cosmic:~# systemctl status slapd
● slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
   Loaded: loaded (/etc/init.d/slapd; generated)
  Drop-In: /lib/systemd/system/slapd.service.d
           └─slapd-remain-after-exit.conf
   Active: inactive (dead) since Mon 2019-04-22 20:15:40 UTC; 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2145 ExecStop=/etc/init.d/slapd stop (code=exited, status=0/SUCCESS)

root@cosmic:~# systemctl cat slapd
# /run/systemd/generator.late/slapd.service
# Automatically generated by systemd-sysv-generator

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/slapd
Description=LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
Before=multi-user.target
Before=multi-user.target
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/slapd start
ExecStop=/etc/init.d/slapd stop

# /lib/systemd/system/slapd.service.d/slapd-remain-after-exit.conf
[Service]
Type=forking
RemainAfterExit=no

[Heitor Alves de Siqueira (halves)]

Verified according to test case in description for bionic:

root@bionic:~# dpkg -l | grep slapd
ii slapd 2.4.45+dfsg-1ubuntu1.2 amd64 OpenLDAP server (slapd)

root@bionic:~# systemctl status slapd
● slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
   Loaded: loaded (/etc/init.d/slapd; generated)
  Drop-In: /lib/systemd/system/slapd.service.d
           └─slapd-remain-after-exit.conf
   Active: active (running) since Mon 2019-04-22 20:15:10 UTC; 4min 23s ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/slapd.service
           └─907 /usr/sbin/slapd -h ldap:/// ldapi:/// -g openldap -u openldap -F /etc/ldap/slapd.d

root@bionic:~# kill -9 907

root@bionic:~# systemctl status slapd
● slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
   Loaded: loaded (/etc/init.d/slapd; generated)
  Drop-In: /lib/systemd/system/slapd.service.d
           └─slapd-remain-after-exit.conf
   Active: inactive (dead) since Mon 2019-04-22 20:19:40 UTC; 1s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1011 ExecStop=/etc/init.d/slapd stop (code=exited, status=0/SUCCESS)

root@bionic:~# systemctl cat slapd
# /run/systemd/generator.late/slapd.service
# Automatically generated by systemd-sysv-generator

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/slapd
Description=LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
Before=multi-user.target
Before=multi-user.target
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/slapd start
ExecStop=/etc/init.d/slapd stop

# /lib/systemd/system/slapd.service.d/slapd-remain-after-exit.conf
[Service]
Type=forking
RemainAfterExit=no

[Heitor Alves de Siqueira (halves)]

Verified according to test case in description for xenial:

root@xenial:~# dpkg -l | grep slapd
ii slapd 2.4.42+dfsg-2ubuntu3.5 amd64 OpenLDAP server (slapd)

root@xenial:~# systemctl status slapd
● slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
   Loaded: loaded (/etc/init.d/slapd; bad; vendor preset: enabled)
  Drop-In: /lib/systemd/system/slapd.service.d
           └─slapd-remain-after-exit.conf
   Active: active (running) since Mon 2019-04-22 20:23:10 UTC; 14s ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/slapd.service
           └─5920 /usr/sbin/slapd -h ldap:/// ldapi:/// -g openldap -u openldap -F /etc/ldap/slapd.d

root@xenial:~# kill -9 5920

root@xenial:~# systemctl status slapd
● slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
   Loaded: loaded (/etc/init.d/slapd; bad; vendor preset: enabled)
  Drop-In: /lib/systemd/system/slapd.service.d
           └─slapd-remain-after-exit.conf
   Active: inactive (dead) since Mon 2019-04-22 20:23:30 UTC; 1s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 5989 ExecStop=/etc/init.d/slapd stop (code=exited, status=0/SUCCESS)

root@xenial:~# systemctl cat slapd
# /run/systemd/generator.late/slapd.service
# Automatically generated by systemd-sysv-generator

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/slapd
Description=LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
Before=multi-user.target
Before=multi-user.target
Before=multi-user.target
Before=graphical.target
Before=shutdown.target
After=remote-fs.target
After=network-online.target
After=systemd-journald-dev-log.socket
Wants=network-online.target
Conflicts=shutdown.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/slapd start
ExecStop=/etc/init.d/slapd stop

# /lib/systemd/system/slapd.service.d/slapd-remain-after-exit.conf
[Service]
Type=forking
RemainAfterExit=no

[Heitor Alves de Siqueira (halves)]

There's a failing autopkgtest for gnupg2 that flagged this openldap update for cosmic, but openldap looks fine.

Checking autopkgtest logs for gnupg2 [0], the tests were failing before the openldap update, and it's not related to this fix. The culprit seems to be a dependency problem with wine/libwine [1].

[0] http://autopkgtest.ubuntu.com/packages/g/gnupg2/cosmic/amd64
[1] https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic/cosmic/amd64/g/gnupg2/20190416_135202_50915@/log.gz

[Dan Streetman (ddstreet)]

gnupg2 autopkgtest failure due to bug 1825186

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.46+dfsg-5ubuntu1.2

---------------
openldap (2.4.46+dfsg-5ubuntu1.2) cosmic; urgency=medium

  * Fix sysv-generator unit file by customizing parameters (LP: #1821343)
    - d/slapd-remain-after-exit.conf: Override RemainAfterExit to allow
      correct systemctl status for slapd daemon.
    - d/slapd.install: place override file in correct location.

 -- Heitor Alves de Siqueira <email address hidden> Wed, 10 Apr 2019 09:50:13 -0300

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for openldap has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.45+dfsg-1ubuntu1.2

---------------
openldap (2.4.45+dfsg-1ubuntu1.2) bionic; urgency=medium

  * Fix sysv-generator unit file by customizing parameters (LP: #1821343)
    - d/slapd-remain-after-exit.conf: Override RemainAfterExit to allow
      correct systemctl status for slapd daemon.
    - d/slapd.install: place override file in correct location.

 -- Heitor Alves de Siqueira <email address hidden> Wed, 10 Apr 2019 09:53:11 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.42+dfsg-2ubuntu3.5

---------------
openldap (2.4.42+dfsg-2ubuntu3.5) xenial; urgency=medium

  * Fix sysv-generator unit file by customizing parameters (LP: #1821343)
    - d/slapd-remain-after-exit.conf: Override RemainAfterExit to allow
      correct systemctl status for slapd daemon.
    - d/slapd.install: place override file in correct location.

 -- Heitor Alves de Siqueira <email address hidden> Wed, 10 Apr 2019 10:01:36 -0300