slapd: slapcat output truncated every now and then

Bug #1185908 reported by Ryan Tandy
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openldap (Debian)
Fix Released
Unknown
openldap (Ubuntu)
Undecided
Unassigned
Precise
Medium
Unassigned
Quantal
Medium
Unassigned
Raring
Medium
Unassigned

Bug Description

SRU justification:

[Impact]

The slapd tools (slapcat, slapadd, et al) don't retry after failing to acquire a BDB read lock, and on a busy LDAP server can sometimes return incomplete data. This could result in data loss, for example when slapcat is used to take a hot backup.

Debian bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=673038
OpenLDAP thread: http://www.openldap.org/lists/openldap-technical/201301/msg00195.html
OpenLDAP ITS:
 - http://www.openldap.org/its/index.cgi?findid=6365
 - http://www.openldap.org/its/index.cgi?findid=7503

The impact is limited to slapd servers with a write load high enough to generate lock contention. It's been present at least since 2.4.17 and maybe longer, and at least some people got used to working around it, e.g. [1], but not everyone is aware that the problem exists.

The fix is minimal and has been tested in OpenLDAP upstream and Debian wheezy.

[1] https://github.com/elmar/ldap-git-backup/blob/master/README.mdown#safe-ldif

[Test Case]

# apt-get install ldap-utils slapd
<configure admin password>
# ldapadd -D cn=admin,dc=nodomain -w adminpw <<end
dn: cn=test,dc=nodomain
objectClass: organizationalRole
objectClass: simpleSecurityObject
cn: test
userPassword: test

end
# while true; do slapcat | wc -l; done

and in another terminal...

$ while true; do ldappasswd -H ldap:// -D cn=admin,dc=nodomain -w adminpw cn=test,dc=nodomain; done

In the first terminal, note that the output from wc is usually 41 but sometimes smaller. It should be the same line count every time.

[Regression Potential]

The regression risk should be small. The change is minimal, was authored by upstream, and has been accepted and released in Debian wheezy. I admit to not being familiar enough with the code to comment in detail on what regressions might be possible. If the fix were faulty wrt locking, I would hope for it to turn up during verification since the test case involves inducing a heavy write load on the server.

original description:

Debian #673038 was fixed in wheezy but the fix has never been merged to Ubuntu. I verified the existence of this bug in precise, quantal, raring, and saucy using more or less the procedure from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=673038#111:

# apt-get install ldap-utils slapd
<configure admin password>
# ldapadd -D cn=admin,dc=nodomain -w adminpw <<end
dn: cn=test,dc=nodomain
objectClass: organizationalRole
objectClass: simpleSecurityObject
cn: test
userPassword: test

end
# while true; do slapcat | wc -l; done

and in another terminal...

$ while true; do ldappasswd -H ldap:// -D cn=admin,dc=nodomain -w adminpw cn=test,dc=nodomain; done

In the first terminal, note that the output from wc is usually 41 but sometimes smaller. It should be the same line count every time.

I'm building and testing patched packages now and will post debdiffs shortly.

Changed in openldap (Debian):
status: Unknown → Fix Released
Ryan Tandy (rtandy)
description: updated
Revision history for this message
Ryan Tandy (rtandy) wrote :

Tried to propose a branch. New to this UDD thing, please give me hints if I've done some part incorrectly. Tested the fix on saucy amd64.

Revision history for this message
James Page (james-page) wrote :

Hi Ryan

Thanks for the merge proposal; this bug was actually fixed in the latest merge from Debian unstable in Ubuntu Saucy:

openldap (2.4.31-1+nmu1) unstable; urgency=medium

  * Non-maintainer upload.
  * Avoid deadlocks in back-bdb that truncate slapcat output (closes: #673038).

 -- Michael Gilbert <email address hidden> Tue, 16 Apr 2013 03:35:31 +0000

Marking 'Fix Released'.

Changed in openldap (Ubuntu):
status: New → Fix Released
Revision history for this message
James Page (james-page) wrote :

Hi Ryan

As this impacts older releases as well, would you like to prepare SRU's for precise, quantal and raring?

You can find details on how todo this at https://wiki.ubuntu.com/StableReleaseUpdates

Cheers

James

Changed in openldap (Ubuntu Precise):
importance: Undecided → Medium
Changed in openldap (Ubuntu Quantal):
importance: Undecided → Medium
Changed in openldap (Ubuntu Raring):
importance: Undecided → Medium
Changed in openldap (Ubuntu Precise):
status: New → Triaged
Changed in openldap (Ubuntu Quantal):
status: New → Triaged
Changed in openldap (Ubuntu Raring):
status: New → Triaged
Revision history for this message
Ryan Tandy (rtandy) wrote :

Thanks James. Added SRU info in the description, will propose SRU merges as I have time to prepare and test them...

description: updated
Revision history for this message
Ryan Tandy (rtandy) wrote :
Revision history for this message
Ryan Tandy (rtandy) wrote :
Revision history for this message
Ryan Tandy (rtandy) wrote :
Revision history for this message
Ryan Tandy (rtandy) wrote :

Note to sponsors/SRU team: the impact of this bug is limited to slapd only and I really don't expect people to be running LDAP servers on quantal or raring: therefore I think it's most important to fix this in precise, less so for those others.

tags: added: patch precise quantal raring
Revision history for this message
Sebastien Bacher (seb128) wrote :

@Ryan: thanks, I've sponsored the raring and precise ones (current stable and LTS), the SRU and verification teams are quite busy so we try to avoid the non-current-non-LTS SRUS, I'm going to skip the quantal one

Changed in openldap (Ubuntu Precise):
status: Triaged → In Progress
Changed in openldap (Ubuntu Quantal):
status: Triaged → Won't Fix
Changed in openldap (Ubuntu Raring):
status: Triaged → In Progress
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Ryan, or anyone else affected,

Accepted openldap into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/openldap/2.4.31-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in openldap (Ubuntu Raring):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Ryan, or anyone else affected,

Accepted openldap into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/openldap/2.4.28-1.1ubuntu4.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in openldap (Ubuntu Precise):
status: In Progress → Fix Committed
Revision history for this message
Ryan Tandy (rtandy) wrote :

Hi,

(I'm allowed to verify my own bugs, right?)

Verified on precise and raring, i386 and amd64.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openldap - 2.4.31-1ubuntu2.1

---------------
openldap (2.4.31-1ubuntu2.1) raring-proposed; urgency=low

  * Avoid deadlocks in back-bdb that truncate slapcat output (LP: #1185908):
    - d/patches/bdb-deadlock.patch: Patch copied from Debian #673038
 -- Ryan Tandy <email address hidden> Tue, 04 Jun 2013 09:18:48 -0700

Changed in openldap (Ubuntu Raring):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openldap - 2.4.28-1.1ubuntu4.3

---------------
openldap (2.4.28-1.1ubuntu4.3) precise-proposed; urgency=low

  * Avoid deadlocks in back-bdb that truncate slapcat output (LP: #1185908):
    - d/patches/bdb-deadlock.patch: Patch copied from Debian #673038
 -- Ryan Tandy <email address hidden> Tue, 04 Jun 2013 09:00:09 -0700

Changed in openldap (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.