nslcd segfaults with nslcd[xxx]: segfault at 0 ip xxx error 4 in libc-2.15.so

[Yolanda Robla]

Under precise, we are seeing that segfault nearly daily.
As a consequence, ssh logins using ldap auth are not possible, and nslcd needs to be restarted in order to make it functional again.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: nslcd 0.8.4ubuntu0.4
ProcVersionSignature: Ubuntu 3.13.0-53.89~precise1-generic 3.13.11-ckt19
Uname: Linux 3.13.0-53-generic x86_64
ApportVersion: 2.0.1-0ubuntu17.9
Architecture: amd64
Date: Fri Jun 12 10:03:42 2015
InstallationMedia: Ubuntu-Server 12.04.3 LTS "Precise Pangolin" - Release amd64 (20130820.2)
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: nss-pam-ldapd
UpgradeStatus: No upgrade log present (probably fresh install)

[Yolanda Robla (yolanda.robla)]

Backtrace when segfault

[Arthur de Jong (adejong)]

Thanks for your bug report. Can you provide your nslcd.conf?

Also, can you run an nslcd that contains the debug symbols? The easiest way to do that is probably to compile nslcd from source. Also installing libkrb5-dbg, libldap-2.4-2-dbg, libc6-dbg, libcomerr2-dbg, cyrus-sasl2-dbg, libgnutls26-dbg and libgcrypt11-dbg packages will probably result in a more useful backtrace.

[Yolanda Robla (yolanda.robla)]

I cannot compìle from source because problem is happening on a production machine, but i send you a new trace with new debug symbols enabled.

[Yolanda Robla (yolanda.robla)]

nslcd.conf file:

# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The location at which the LDAP server(s) should be reachable.
uri ldaps://ldap.domain.com/

# The search base that will be used for all queries.
base o=domain.com

# The LDAP protocol version to use.
#ldap_version 3

# The DN to bind with for normal lookups.
#binddn cn=annonymous,dc=example,dc=net
#bindpw secret

# The DN used for password modifications by root.
#rootpwmoddn cn=admin,dc=example,dc=com
idle_timelimit 100

# SSL options
ssl yes

# needed for hp ldap to connect
tls_reqcert allow

# The search scope.
scope sub

map passwd uid hpUnixUserName
#map group uniqueMember member # Not needed after 0.8.4

nss_initgroups_ignoreusers backup,bin,daemon,games,gnats,irc,libuuid,list,lp,mail,man,news,proxy,root,sshd,sync,sys,syslog,uucp,www-data,sitescop,postfix,ntp

[Yolanda Robla (yolanda.robla)]

Also, running nslcd in debug mode, segfault happens after a massive ldap search:

nslcd: [78ebeb] <group(all)> DEBUG: myldap_search(<email address hidden>,ou=People,o=domain.com", filter="(objectClass=posixAccount)")
nslcd: [78ebeb] <group(all)> DEBUG: ldap_result(): end of results

[1]+ Segmentation fault (core dumped) /usr/sbin/nslcd -d

[Arthur de Jong (adejong)]

The trace doesn't help me much in tracking it down I'm afraid. I've checked all the uses of strcmp() in nslcd that could be reachable and have not found a likely candidate. You could try building a package on another machine that includes the debug symbols (see https://jameswestby.net/tips/tips/compiling-debian-package-for-debug.html) or try running nslcd under valgrind. Running under valgrind does have a performance impact though.

Is the crash always after such a search? Is there any pattern in the crashes?

[Yolanda Robla (yolanda.robla)]

I cannot send any more trace at the moment, will work on that.
nslcd crashes after a massive search of users, these myldap_search are regularly executed for an important amount of users, and if segfaults always on that bit.
I enabled pageSize=1000 on nslcd.conf but i got same result.

[Yolanda Robla (yolanda.robla)]

More output from the debug log

nslcd: [c6c33a] <group(all)> DEBUG: myldap_search(base="uid=xxxx,ou=People,o=xxx", filter="(objectClass=posixAccount)")
nslcd: [55585c] <group(all)> DEBUG: ldap_result(): end of results
nslcd: [55585c] <group(all)> DEBUG: myldap_search(base="uid=xxx,ou=People,o=xxx", filter="(objectClass=posixAccount)")
nslcd: [c6c33a] <group(all)> DEBUG: myldap_search(base="uid=xxx,ou=People,o=xxx", filter="(objectClass=posixAccount)")
nslcd: [55585c] <group(all)> DEBUG: ldap_result(): end of results
nslcd: [55585c] <group(all)> error writing to client: Broken pipe

I see that broken pipe repeating on error log

[Rohanil Raje (rohanil)]

This issue is happening in few of the my servers as well. I found workaround on RHEL servers https://bugzilla.redhat.com/show_bug.cgi?id=1312297
But I want to know the root cause of this segmentation fault. Anyone have idea about it?