slapd crash with pwdAccountLockedTime and stacked overlays

[Ryan Tandy]

[Impact]
In the configuration and conditions described below, slapd can crash:

1. ppolicy overlay configured with pwdLockout: TRUE
2. smbk5pwd overlay stacked after ppolicy
3. an account locked out via pwdAccountLockedTime
4. a client binding to the locked-out account and also requesting the ppolicy control

[Test Case]

* get the files from the bug:
mkdir slapd-test-case; cd slapd-test-case
wget -ct0 https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1866303/+attachment/5334194/+files/slapd.conf https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1866303/+attachment/5334195/+files/data.ldif https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1866303/+attachment/5334196/+files/samba.schema https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1866303/+attachment/5334197/+files/script

* run the script:
sudo apt update && sudo sh ./script

* With the bug, the result is:
ldap_bind: Invalid credentials (49)
slapd dead

* If when confirming the bug you don't see "slapd dead" like above, check manually, as slapd might have been in the process of shutting down when the script checked its status: "sudo systemctl status slapd"

* With the fixed packages, you get a living slapd at the end (you can run the script again on the same system after updating the packages):

sudo sh ./script
...
slapd running
ldap_bind: Invalid credentials (49)
slapd running

[Regression Potential]
The fix is in the password policy overlay (not enabled by default), so any regressions would be around that area and could potentially impact authentication ("binding") to openldap.

[Other Info]
This was fixed in focal and "cooked" there for a long while, as suggested by the Debian maintainer. We haven't received further bug reports about this in focal+.

[Original Description]

Hello,

Please merge openldap 2.4.49+dfsg-2 from Debian unstable to fix an issue in the ppolicy overlay that can crash slapd. Please also consider SRUing the patch after it has had some testing time.

Upstream: https://openldap.org/its/?findid=9171
Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=953150

The ingredients for the crash are:

1: ppolicy overlay configured with pwdLockout: TRUE
2. smbk5pwd overlay stacked after ppolicy
3. an account locked out via pwdAccountLockedTime
4. a client binding to the locked-out account and also requesting the ppolicy control

The buggy code is not as specific as the above steps, so I suspect there are probably other configurations or steps that can trigger the same crash.

I will attach my test script and data for reproducing the crash.

Expected output (last lines):

[ ok ] Starting OpenLDAP: slapd.
slapd running
ldap_bind: Invalid credentials (49)
slapd running

Actual output (last lines):

[ ok ] Starting OpenLDAP: slapd.
slapd running
ldap_bind: Invalid credentials (49)
slapd dead

[Andreas Hasenack (ahasenack)]

Thanks a lot for this Ryan, and awesome testing script!

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.49+dfsg-2ubuntu1

---------------
openldap (2.4.49+dfsg-2ubuntu1) focal; urgency=medium

  * Merge with Debian unstable (LP: #1866303). Remaining changes:
    - Enable AppArmor support:
      - d/apparmor-profile: add AppArmor profile
      - d/rules: use dh_apparmor
      - d/control: Build-Depends on dh-apparmor
      - d/slapd.README.Debian: add note about AppArmor
    - Enable GSSAPI support:
      - d/patches/gssapi.diff, thanks to Jerry Carter (Likewise):
        - Add --with-gssapi support
        - Make guess_service_principal() more robust when determining
          principal
        [Dropped the ldap_gssapi_bind_s() hunk as that is already
      - d/configure.options: Configure with --with-gssapi
      - d/control: Added heimdal-dev as a build depend
      - d/rules:
        - Explicitly add -I/usr/include/heimdal to CFLAGS.
        - Explicitly add -I/usr/lib/<multiarch>/heimdal to LDFLAGS.
    - Enable ufw support:
      - d/control: suggest ufw.
      - d/rules: install ufw profile.
      - d/slapd.ufw.profile: add ufw profile.
    - Enable nss overlay:
      - d/rules:
        - add nssov to CONTRIB_MODULES
        - add sysconfdir to CONTRIB_MAKEVARS
      - d/slapd.install:
        - install nssov overlay
      - d/slapd.manpages:
        - install slapo-nssov(5) man page
    - d/{rules,slapd.py}: Add apport hook.
    - d/slapd.init.ldif: don't set olcRootDN since it's not defined in
      either the default DIT nor via an Authn mapping.
    - d/slapd.scripts-common:
      - add slapcat_opts to local variables.
      - Fix backup directory naming for multiple reconfiguration.
    - d/{slapd.default,slapd.README.Debian}: use the new configuration style.
    - d/rules: Enable -DLDAP_CONNECTIONLESS to build CLDAP (UDP) support
      in the openldap library, as required by Likewise-Open
    - Show distribution in version:
      - d/control: added lsb-release
      - d/patches/fix-ldap-distribution.patch: show distribution in version
    - d/libldap-2.4-2.symbols: Add symbols not present in Debian.
      - CLDAP (UDP) was added in 2.4.17-1ubuntu2
      - GSSAPI support was enabled in 2.4.18-0ubuntu2
    - d/p/contrib-makefiles: given the change in 2.4.47+dfsg-3 regarding
      Debian bug #919136, we also have to patch the nssov makefile
      accordingly and thus update this patch.

openldap (2.4.49+dfsg-2) unstable; urgency=medium

  * slapd.README.Debian: Document the initial setup performed by slapd's
    maintainer scripts in more detail. Thanks to Karl O. Pinc.
    (Closes: #952501)
  * Import upstream patch to fix slapd crashing in certain configurations when
    a client attempts a login to a locked account.
    (ITS#9171) (Closes: #953150)

 -- Andreas Hasenack <email address hidden> Fri, 06 Mar 2020 11:39:12 -0300

[Bryce Harrington (bryce)]

We're no longer looking at backporting fixes for disco.

This looks suitable for SRU so the other proposed series tasks are valid, and this is already in the server-next queue.

[Andreas Hasenack (ahasenack)]

This fix was added to focal, and we haven't received any crash reports about it as far as I know, so I'm proceeding with the SRU for the other ubuntu releases.

[Brian Murray (brian-murray)]

Hello Ryan, or anyone else affected,

Accepted openldap into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.4.48+dfsg-1ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Ryan, or anyone else affected,

Accepted openldap into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.4.45+dfsg-1ubuntu1.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Ryan, or anyone else affected,

Accepted openldap into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.4.42+dfsg-2ubuntu3.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openldap (2.4.48+dfsg-1ubuntu1.2) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

asterisk/1:16.2.1~dfsg-2build2 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#openldap

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Andreas Hasenack (ahasenack)]

Xenial verification

Reproducing the error:
root@xenial-openldap-saslauthd-1557157:~# ldapsearch -H ldapi:/// -LLL -b 'dc=example,dc=com' -s base -U root -Y PLAIN
SASL/PLAIN authentication started
Please enter your password:
ldap_sasl_interactive_bind_s: Other (e.g., implementation specific) error (80)
 additional info: SASL(-1): generic failure: Password verification failed

And dmesg:
[qua jul 8 11:50:42 2020] audit: type=1400 audit(1594219843.513:405): apparmor="DENIED" operation="connect" namespace="root//lxd-xenial-openldap-saslauthd-1557157_<var-snap-lxd-common-lxd>" profile="/usr/sbin/slapd" name="/run/saslauthd/mux" pid=83468 comm="slapd" requested_mask="wr" denied_mask="wr" fsuid=1000112 ouid=1000000

With the updated packages, ldapsearch works:
root@xenial-openldap-saslauthd-1557157:~# apt-cache policy slapd
slapd:
  Installed: 2.4.42+dfsg-2ubuntu3.9
  Candidate: 2.4.42+dfsg-2ubuntu3.9
  Version table:
 *** 2.4.42+dfsg-2ubuntu3.9 500
        500 http://br.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
...

root@xenial-openldap-saslauthd-1557157:~# ldapsearch -H ldapi:/// -LLL -b 'dc=example,dc=com' -s base -U root -Y PLAIN
SASL/PLAIN authentication started
Please enter your password:
SASL username: root
SASL SSF: 0
dn: dc=example,dc=com
objectClass: top
objectClass: dcObject
objectClass: organization
o: example
dc: example

And no dmesg apparmor error.

Xenial verification succeeded.

[Andreas Hasenack (ahasenack)]

I'm sorry, the above verification was for the other bug that this upload is fixing.

[Andreas Hasenack (ahasenack)]

Xenial verification (for real)

Reproducing the bug:
  Version table:
 *** 2.4.42+dfsg-2ubuntu3.8 500
        500 http://br.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://br.archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status

$ sudo sh ./script
...
Closing DB...
slapd running
ldap_bind: Invalid credentials (49)
slapd dead

With the packages from proposed, slapd remains running:
  Version table:
 *** 2.4.42+dfsg-2ubuntu3.9 500
        500 http://br.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

$ sudo sh ./script
...
Closing DB...
slapd running
ldap_bind: Invalid credentials (49)
slapd running

Xenial verification succeeded.

[Andreas Hasenack (ahasenack)]

Bionic verification

Reproducing the bug:
  Version table:
 *** 2.4.45+dfsg-1ubuntu1.5 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://br.archive.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        100 /var/lib/dpkg/status

$ sudo sh ./script
...
Closing DB...
slapd running
ldap_bind: Invalid credentials (49)
slapd dead

Updating to proposed:
  Version table:
 *** 2.4.45+dfsg-1ubuntu1.6 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

Now slapd remains running:
$ sudo sh ./script
...
Closing DB...
slapd running
ldap_bind: Invalid credentials (49)
slapd running

Bionic verification succeeded.

[Andreas Hasenack (ahasenack)]

Eoan verification

Reproducing the problem:
  Version table:
 *** 2.4.48+dfsg-1ubuntu1.1 500
        500 http://br.archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages
        500 http://br.archive.ubuntu.com/ubuntu eoan-security/main amd64 Packages
        100 /var/lib/dpkg/status

ubuntu@eoan-openldap-crash-1866303:~/slapd-test-case$ sudo sh ./script
...
Closing DB...
slapd running
ldap_bind: Invalid credentials (49)
slapd dead

With the proposed packages:
  Version table:
 *** 2.4.48+dfsg-1ubuntu1.2 500
        500 http://br.archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

slapd remains running:
ubuntu@eoan-openldap-crash-1866303:~/slapd-test-case$ sudo sh ./script
...
Closing DB...
slapd running
ldap_bind: Invalid credentials (49)
slapd running

Eoan verification succeeded.

[Andreas Hasenack (ahasenack)]

The asterisk DEP8 armhf test was retried and is now green.

[Andreas Hasenack (ahasenack)]

Kopanocore armhf is the only persistent red, but this test/package is known to be flaky on armhf.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.48+dfsg-1ubuntu1.2

---------------
openldap (2.4.48+dfsg-1ubuntu1.2) eoan; urgency=medium

  [ Andreas Hasenack ]
  * d/p/ITS-9171-Insert-callback-in-the-right-place.patch: Import upstream
    patch to fix slapd crashing in certain configurations when a client
    attempts a login to a locked account. (LP: #1866303)

  [ Sergio Durigan Junior ]
  * d/apparmor-profile: Update apparmor profile to grant access to
    the saslauthd socket, so that SASL authentication works. (LP: #1557157)

 -- Andreas Hasenack <email address hidden> Wed, 01 Jul 2020 16:43:06 -0300

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for openldap has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.45+dfsg-1ubuntu1.6

---------------
openldap (2.4.45+dfsg-1ubuntu1.6) bionic; urgency=medium

  [ Andreas Hasenack ]
  * d/p/ITS-9171-Insert-callback-in-the-right-place.patch: Import upstream
    patch to fix slapd crashing in certain configurations when a client
    attempts a login to a locked account. (LP: #1866303)

  [ Sergio Durigan Junior ]
  * d/apparmor-profile: Update apparmor profile to grant access to
    the saslauthd socket, so that SASL authentication works. (LP: #1557157)

 -- Andreas Hasenack <email address hidden> Wed, 01 Jul 2020 16:38:55 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.42+dfsg-2ubuntu3.9

---------------
openldap (2.4.42+dfsg-2ubuntu3.9) xenial; urgency=medium

  [ Andreas Hasenack ]
  * d/p/ITS-9171-Insert-callback-in-the-right-place.patch: Import upstream
    patch to fix slapd crashing in certain configurations when a client
    attempts a login to a locked account. (LP: #1866303)

  [ Sergio Durigan Junior]
  * d/apparmor-profile: Update apparmor profile to grant access to
    the saslauthd socket, so that SASL authentication works. (LP: #1557157)

 -- Andreas Hasenack <email address hidden> Wed, 01 Jul 2020 16:33:08 -0300