slapd 2.4.21-0ubuntu5 corrupts olcDatabase={-1}frontend.ldif with duplicate olcAccess lines (again)

[Stephen Warren]

Bug 526230 is back.

I had slapd 2.4.21-0ubuntu4 installed, then "apt-get dist-upgrade", which pulled in slapd 2.4.21-0ubuntu5. This modified /etc/ldap/slapd.d/cn=config/olcDatabase={-1}frontend.ldif by adding duplicate olcAccess lines without any {0} index prefix, causing slapd to fail to start. This caused:

==========
Setting up slapd (2.4.21-0ubuntu5) ...
  Backing up /etc/ldap/slapd.d/ in /var/backups/slapd-2.4.21-0ubuntu4... done.
Starting OpenLDAP: slapd - failed.
The operation failed but no output was produced. For hints on what went
wrong please refer to the system's logfiles (e.g. /var/log/syslog) or
try running the daemon in Debug mode like via "slapd -d 16383" (warning:
this will create copious output).

Below, you can find the command line options used by this script to
run slapd. Do not forget to specify those options if you
want to look to debugging output:
  slapd -h 'ldap:/// ldapi:///' -g openldap -u openldap -F /etc/ldap/slapd.d/
invoke-rc.d: initscript slapd, action "start" failed.
dpkg: error processing slapd (--configure):
 subprocess installed post-installation script returned error exit status 1
==========

and:

==========
Apr 27 21:15:16 esk slapd[8805]: @(#) $OpenLDAP: slapd 2.4.21 (Apr 26 2010 11:07:14) $#012#011buildd@rothera:/build/buildd/openldap-2.4.21/debian/build/servers/slapd
Apr 27 21:15:16 esk slapd[8805]: config error processing olcDatabase={-1}frontend,cn=config: ordered_value_sort failed on attr olcAccess#012
Apr 27 21:15:16 esk slapd[8805]: slapd stopped.
==========

due to content:

==========
dn: olcDatabase={-1}frontend
objectClass: olcDatabaseConfig
objectClass: olcFrontendConfig
olcDatabase: {-1}frontend
olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth manage by * break
olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth manage by * break
olcAddContentAcl: FALSE
olcLastMod: TRUE
olcMaxDerefDepth: 0
olcReadOnly: FALSE
olcSchemaDN: cn=Subschema
olcMonitoring: FALSE
olcAccess: to * by dn.exact=cn=localroot,cn=config manage by * break
structuralObjectClass: olcDatabaseConfig
entryUUID: 9d222b1e-24cc-102e-9a29-375c9ad51446
creatorsName: cn=config
createTimestamp: 20090824073643Z
entryCSN: 20090824073643.173347Z#000000#000#000000
modifiersName: cn=config
modifyTimestamp: 20090824073643Z
==========

Note: I tried "apt-get dist-upgrade" a few times to see if the problem would fix itself before investigating. I think each run added a new duplicate olcAccess entry without checking for pre-existing entries. After I deleted the first two olcAccess above, slapd would start again.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: slapd 2.4.21-0ubuntu5
ProcVersionSignature: Ubuntu 2.6.32-21.32-generic 2.6.32.11+drm33.2
Uname: Linux 2.6.32-21-generic i686
Architecture: i386
Date: Tue Apr 27 21:16:07 2010
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.utf8
 SHELL=/bin/bash
SourcePackage: openldap

Lucid Release Note:

== Openldap fails to start on upgrade ==

When upgrading some systems from Karmic openldap may fail to start by logging messages similar to "ordered_value_sort failed on attr olcAccess#012". To workaround the problem remove the line "olcAccess: to * by dn.exact=cn=localroot,cn=config manage by * break" from /etc/ldap/slapd.d/cn=config/olcDatabase={-1}frontend.ldif and /etc/ldap/slapd./cn=config/olcDatabase={0}config.ldif.

==========
SRU REPORT
==========

BUG IMPACT:
On systems upgraded from jaunty -> karmic -> lucid, the local root
user is mapped to cn=localroot,cn=config. The latter dn has then full
access to the cn=config tree. The olcAccess line added during the
karmic upgrade isn't prefixed with an index. Additional olcAccess
lines are added during the lucid upgrade which makes slapd fail to
start as all olcAccess lines need to be prefixed with an index.

BUG FIX:
The olcAccess line is updated to have an index during the upgrade.

TEST CASE:
1. Install slapd on a jaunty system.
2. Upgrade to karmic.
3. Upgrade to lucid:
   * without the fix: after upgrade slapd is not running.
   * with the fix: after upgrade slapd is running.

REGRESSION POTENTIAL:
Unknown.

[Nathan Stratton Treadway (nathanst)]

The history for bug 563829 includes some discussion of this situation with the olcDatabase={-1}frontend.ldif file.

[Nathan Stratton Treadway (nathanst)]

(Also, for what it's worth, the slapd.postinst script does include a package-version check which attempts to prevent the new line from being added more than once. However, since the slapd-failure prevents the package from reaching "configured" status, the script is still trying to upgrade from the older package version each time it's executed, and thus it would add a new copy of the line each time you ran "apt dist-upgrade".)

[Thierry Carrez (ttx)]

We'll need to release note that

[Mathias Gug (mathiaz)]

This bug should only affect systems that have been installed in Intrepid/Jaunty, upgraded to Karmic then Lucid.

Systems installed in Karmic and systems upgrading from Hardy shouldn't be affected.

[Nathan Stratton Treadway (nathanst)]

(I think systems installed in Hardy and then upgraded to pre-release Lucid versions before upgrading to 0ubuntu5 will also be affected.)

[Nathan Stratton Treadway (nathanst)]

(To be precise, if I have followed the changelog correctly, the problem will be triggered when the upgrade path looks like:
  slapd older than 2.4.17-1ubuntu3 -->
    slapd between 2.4.17-1ubuntu3 and 2.4.21-0ubuntu4 -->
       (maybe some upgrades within that range) -->
          slapd 2.4.21-0ubuntu5

The first of those upgrades would add the
   olcAccess: to * by dn.exact=cn=localroot,cn=config manage by * break
line, and then final one would add
  olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth manage by * break
, thus triggering the error.
)

[Nathan Stratton Treadway (nathanst)]

(To clarify my previous comment: note that while the symptoms are similar, this bug and bug 526230 actually have different underlying causes, and the thus details of the upgrade paths that trigger each one are different, too.)

[Mathias Gug (mathiaz)]

On Wed, Apr 28, 2010 at 02:06:00PM -0000, Nathan Stratton Treadway wrote:
> (I think systems installed in Hardy and then upgraded to pre-release
> Lucid versions before upgrading to 0ubuntu5 will also be affected.)
>

That is correct as well. The fix is always the same.

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

[Nathan Stratton Treadway (nathanst)]

As touched on in the discussion for bug #563829, the release notes should also mention that after upgrading to slapd 2.4.21-0ubuntu5, the user will need to manually clean up the slapd config files in order to complete the switch from the use of the "cn=localroot,cn=config" mapping to the direct use of the "gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" identifier in the security configuration. (This is true even when slapd does start up successfully after the upgrade.)

As far as I can tell from my own testing, this cleanup would involve removing any olcAccess lines referencing "cn=localroot,cn=config" from all /etc/ldap/slapd.d/cn=config/olcDatabase*.ldif files, and also removing the olcAuthzRegexp line mentioning that identifier from the /etc/ldap/slapd.d/cn=config.ldif file.

[Loïc Minier (lool)]

Could we fix it automatically by checking the contents of the config file?

Could we prepare this as a SRU sitting in -proposed at the time of release?

[Nathan Stratton Treadway (nathanst)]

A few other points that hopefully can be worked into the release notes:

* A symptom that indicates the need for this config-file cleanup is when commands that rely on EXTERNAL SASL authentication no longer work for the local root user (e.g. "ldapsearch -Y EXTERNAL -Hldapi:/// ....")

* One can avoid having dpkg abort the installation run by doing the cleanup before kicking off the upgrade to 2.4.21-0ubuntu5.

* If the cleanup isn't done beforehand, then (in addition to removing the "localroot" lines), the user will probably want to go ahead and delete any extra copies of the
  olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth manage by * break
line that get added to the olcDatabase{0}config.ldif and oldDatabase{-1}frontend.ldif files if the installation script is run multiple times. (This can happen automatically; e.g. aptitude will automatically retry the package install after the first dpkg failure.) The "intended" situation is to have exactly one copy of that line in each of the files.

[Stephen Warren (srwarren)]

Re: the mention of symptoms in comment #12 above: My symptom was that I could not log in at all, and in existing sessions, sudo wouldn't work etc. I store user information in LDAP, with just system users in /etc/passwd etc., so luckily I could still log in as root to fix this.

[Michel (michel-mno)]

I experienced the same problem on my laptop after upgrade from 9.10 to 10.4
and was able to bypass it with the description of Nathan in comment #10
Thank you.

[Thierry Carrez (ttx)]

Mathias: your proposed branch does not apply the cleanup if you upgrade from 0ubuntu5, is it on purpose ?

I propose that instead: lp:~ttx/ubuntu/lucid/openldap/cleanup-olcaccess

[Nathan Stratton Treadway (nathanst)]

Mathias, Thierry: neither of these scripts appear to clean up the
   olcAuthzRegexp: gidNumber=\[\[:digit:]]\+\\\+uidNumber=0,cn=peercred,cn=external,cn=auth cn=localroot,cn=config'
line that got added to the ${SLAPD_CONF}/cn=config.ldif file by earlier upgrades. I believe that as long as that mapping is there, the newly-added olcAccess lines referencing "dn.exact=gidNumber=0+uidNumber=0,..." will be ignored.

Does anyone know if "#" comments are officially supported in these slapd.d config files? (They worked in my manual tests, but I haven't had a chance to research whether one is really supposed to use them.) If they are supported, it might be better for the postinst edits just to comment out these lines, rather than than completely deleting them....

[Nathan Stratton Treadway (nathanst)]

On Thu, Apr 29, 2010 at 02:57:36 -0000, Stephen Warren wrote:
> Re: the mention of symptoms in comment #12 above: My symptom was that I
> could not log in at all, and in existing sessions, sudo wouldn't work
> etc. I store user information in LDAP, with just system users in
> /etc/passwd etc., so luckily I could still log in as root to fix this.

Ah, good point. I have been working with a test system not configured
for LDAP authentication, so I didn't check out that functionality.

When you say "still log in as root to fix this", did you have to make
additional edits after you got slapd running again (as you mentioned in
your original problem description)? That is, were you locked out just
because slapd wasn't running, and then back to normal again once you got
slapd restarted, or did you have to go back and fix the permission
settings before LDAP authentication started working again?

(If you did have to fix permissions, what exactly did you have to change
to get that part working?)

      Nathan

[Thierry Carrez (ttx)]

I tried 4->5->your proposed 5.1 and ended up with:

 olcDatabase: {0}config
 olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,...
 olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,...

[Thierry Carrez (ttx)]

As commented by Nathan on comment 10, should we also remove the olcAuthzRegexp line mentioning localroot from the /etc/ldap/slapd.d/cn=config.ldif file ?

[Stephen Warren (srwarren)]

> When you say "still log in as root to fix this", did you have to make
> additional edits after you got slapd running again (as you mentioned in
> your original problem description)? That is, were you locked out just
> because slapd wasn't running, and then back to normal again once you got
> slapd restarted, or did you have to go back and fix the permission
> settings before LDAP authentication started working again?
>
> (If you did have to fix permissions, what exactly did you have to change
> to get that part working?)

I *believe* all I did was:
* Install updates, things broke (e,.g. couldn't sudo), but I was still logged in
* Switched to a text VT, logged in as root
* Edited slapd config files
* Restarted slapd
* Went back to X VT, could immediately sudo

(I think)

[Nathan Stratton Treadway (nathanst)]

Thierry, any chance of of adding another release note covering the post-upgrade access permissions problems discussed here and in bug #571752?

Even though they won't cause the upgrade process to abort the way the ordered_value_sort error does, it still seems pretty significate that some LDAP client software will no longer function as expected after the upgrade....

[Mathias Gug (mathiaz)]

I'd advise against cleaning up cn=localroot,cn=config on upgrades as some systems may use cn=localroot,cn=config as part of their directory ACL infrastructure. The map to cn=localroot,cn=config may also have changed.

[Mathias Gug (mathiaz)]

Marking won't fix for maverick as this bugs is about covering upgrades from karmic to lucid.

[Martin Pitt (pitti)]

Accepted openldap into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Mathias Gug (mathiaz)]

I've run through the test case and confirm that the upgrade of slapd with -proposed enabled doesn't fail.

[Steve Beattie (sbeattie)]

Unfortunately, openldap 2.4.21-0ubuntu5.2 was published in lucid-security, and so this proposed update needs to be re-uploaded with the fixes from 2.4.21-0ubuntu5.2 applied.

[Martin Pitt (pitti)]

Meh, bad timing then. Mathias, can you please merge and reupload? Seems we need to rush that through if we want to land it on 10.04.1

[Martin Pitt (pitti)]

Accepted 2.4.21-0ubuntu5.3 into lucid-proposed, which merges the security update. Please re-test.

Waiving 7-day maturing period, since this was already tested before, and we just need to ensure that it built correctly, and that the upgrade still works.

[Mathias Gug (mathiaz)]

I've run through the test case and confirm that the upgrade of slapd with -proposed enabled doesn't fail.

[Steve Beattie (sbeattie)]

I've also confirmed that the version of openldap in lucid-proposed, 2.4.21-0ubuntu5.3, does not introduce any regressions in the openldap tests in lp:qa-regression-testing.

Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.21-0ubuntu5.3

---------------
openldap (2.4.21-0ubuntu5.3) lucid-proposed; urgency=low

  * debian/slapd.postinst: Properly index cn=localroot,cn=config olcAccess
    line on upgrade from karmic.
    (LP: #571057)
 -- Mathias Gug <email address hidden> Tue, 10 Aug 2010 12:36:27 -0400

[David McNeill (davemc)]

Confirming occurrence
    New install intrepid > upgrade jaunty > upgrade karmic > upgrade lucid

Confirming fix above at #12.3 corrects the issue.

Confirming the issue causes problems with lam, phpldapadmin, samba-ldap