openldap upgrade fails. chwon of olcDbDirectory, /var/lib/ldap not empty and missing backup of suffix

[Peder Stray]

Upgrading from Ubuntu 12.04.5 LTS to 14.04.1 LTS (slapd-2.4.28-1.1ubuntu4.4 to slapd-2.4.31-1+nmu2ubuntu8), it fails horribly on my setup with one disk-backend and one ldap-backend (sync to another server).

First, dump_databases in /var/lib/dpkg/info/slapd.{config,preinst,postinst} fails due to get_directory returning "olcDbDirectory" instead of /var/lib/ldap. This is caused by a missing -h to grep, causing another : in the returned line, thus causing the '| cut -d: -f 2' to get the wrong part of the line. This causes both the backup of the suffix to fail, and later the chown after restore to fail.

    grep "olcDbDirectory:" `grep -l "olcSuffix: ...

should have been

    grep -h "olcDbDirectory:" `grep -l "olcSuffix: ...

later, get_suffix causes the same suffix to be reported twice, thus causing database load to fail on the second round of the while since /var/lib/ldap is already loaded with the correct files, but is expected to be empty. I fixed it with a '| sort -u' in get_suffix, but i guess it would be better to maybe redo load_databases' while loop to get both suffix and dbdir (since you can have the same suffix stored in more than one location, as I have, and thus first getting the suffix and then doing a grep for the olcDbDirectory in get_directory will cause the same storage to be reported for both instances of the suffix)

[Ryan Tandy (rtandy)]

Thanks for the report. At a glance, this appears like a more detailed duplicate of your comment on bug 322944; is that right, or is it a separate bug? Either way, if it's possible for you to provide a copy of your configuration that fails (with sensitive details/passwords removed), or steps to construct such a configuration, that would be really helpful. Thanks!

[Peder Stray (pstray)]

Yes, they are definitly related, but not caused by the exact same configuration. But still, the fixes i oulined may fix that bug too.

I can't provide our exact configuration, but I followed the guide at http://www.openldap.org/doc/admin24/replication.html#Syncrepl%20Proxy to set up replication to another ldap. This causes the suffix to appear in both of

    /etc/ldap/slap.d/cn=config/olcDatabase={1}hdb.ldif
    /etc/ldap/slap.d/cn=config/olcDatabase={2}ldap.ldif

Only the first of them has olcDbDirectory in it.

[Ryan Tandy (rtandy)]

Great. Thanks for that link, and also for the analysis and suggestions. You're right: there is an implicit assumption that suffixes are unique, and this setup is a valid one that breaks it.

With this change:

http://anonscm.debian.org/cgit/pkg-openldap/openldap.git/commit/debian/slapd.scripts-common?id=57b0cc5ae377b00219e73be87e0c3a151b5eda99

the dump part at least doesn't fail ("slapcat -b dc=example,dc=com" is executed twice, operating on the disk-backed db both times), but I wouldn't exactly call that correct.

Of course, running "slapadd -b dc=example,dc=com" twice is never going to work. (Well, maybe if the database is totally empty.) (Additionally, moving the old database out of the way doesn't work if we have {1}ldap and {2}hdb instead of the reverse.)

I already wanted to redo the loops like you suggested; this is just another reason to move forward on that. Probably it would be most robust to use slap{cat,add} -nN instead of -b $suffix; at least that would have the least chance for ambiguity.

[Nathan Stratton Treadway (nathanst)]

If you are working on cleaning up the slapd.postinst script, you may find some of these related discussions to be interesting and/or helpful...:

LP: #450645 "error during slapd configuration: chown: cannot access `olcDbDirectory\nolcDbDirectory'"
LP: #632051 "Improve slapd postinst error message in case database directory can't be determined for a given LDAP suffix"
LP: #571498 "slapd.postinst should put all backed-up items together in one place under /var/backups"
LP: #571481 "when slapd upgrade fails, later upgrade attempts overwrite saved backups of pre-upgrade configuration files"

[Ryan Tandy (rtandy)]

Thanks for the links. Several of those actually describe things I was already planning to work on. Good to have the bug #s for proper tracking and closing.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.40+dfsg-1ubuntu1

---------------
openldap (2.4.40+dfsg-1ubuntu1) wily; urgency=low

  * Merge from Debian testing (LP: #1395098, LP: #1316124). Remaining changes:
    - Enable AppArmor support:
      - d/apparmor-profile: add AppArmor profile
      - d/rules: use dh_apparmor
      - d/control: Build-Depends on dh-apparmor
      - d/slapd.README.Debian: add note about AppArmor
    - Enable GSSAPI support:
      - d/patches/gssapi.diff, thanks to Jerry Carter (Likewise):
        - Add --with-gssapi support
        - Make guess_service_principal() more robust when determining
          principal
      - d/configure.options: Configure with --with-gssapi
      - d/control: Added heimdal-dev as a build depend
    - Enable ufw support:
      - d/control: suggest ufw.
      - d/rules: install ufw profile.
      - d/slapd.ufw.profile: add ufw profile.
    - Enable nss overlay:
      - d/{patches/nssov-build,rules}: Apply, build and package the
        nss overlay.
    - d/{rules,slapd.py}: Add apport hook.
    - d/slapd.init.ldif: don't set olcRootDN since it's not defined in
      either the default DIT nor via an Authn mapping.
    - d/slapd.scripts-common:
      - add slapcat_opts to local variables.
      - Remove unused variable new_conf.
      - Fix backup directory naming for multiple reconfiguration.
    - d/{slapd.default,slapd.README.Debian}: use the new configuration style.
    - d/rules: Enable -DLDAP_CONNECTIONLESS to build CLDAP (UDP) support
      in the openldap library, as required by Likewise-Open
    - Show distribution in version:
      - d/control: added lsb-release
      - d/patches/fix-ldap-distribution.patch: show distribution in version
  * Drop patches included upstream:
    - d/patches/0001-ITS-7430-GnuTLS-Avoid-use-of-deprecated-function.patch
    - d/patches/bdb-deadlock.patch
    - d/patches/its-7354-fix-delta-sync-mmr.diff
  * Drop hardening-wrapper as Debian now sets PIE and bindnow flags.
  * debian/patches/nssov-build: Adjust for upstream changes.
  * debian/apparmor-profile:
    - Change 'r' to 'rw' for ldapi and nslcd sockets, required for apparmor
      kernel ABI v7 (utopic and later). (LP: #1392018)
    - Reduce permissions on /run/nslcd to just the nslcd socket.
  * Enable the mdb backend again on ppc64el, fixed upstream in ITS#7713.
    (LP: #1293250)

openldap (2.4.40+dfsg-1) unstable; urgency=medium

  * Remove inetorgperson.schema from the upstream source. Replace it with a
    copy stripped of RFC text. (Closes: #780283)
  * Adjust debian/watch for +dfsg versioning.
  * debian/patches/ITS7975-fix-mdb-onelevel-search.patch: Import upstream
    patch to fix scope=onelevel searches wrongly including the search base in
    results under the MDB backend. (ITS#7975) (Closes: #782212)

openldap (2.4.40-4) unstable; urgency=medium

  * debian/patches/ITS8027-deref-reject-empty-attr-list.patch: Import upstream
    patch to fix a crash when a search includes the Deref control with an
    empty attribute list. (ITS#8027) (CVE-2015-1545, Closes: #776988)
  * debian/patches/ITS8046-fix-vrFilter_free-crash.patch: Import upstream
    patch to fix a double free triggered by...