nscd leaking memory using libnss-ldap

Bug #292971 reported by Jouko Markkanen
Affects Status Importance Assigned to Milestone
eglibc (Ubuntu)
Mathias Gug
Mathias Gug

Bug Description

We've got our ubuntu servers (8.04.1 LTS) authenticating users against an active directory libpam-krb5 and user account info is gotten via libnss-ldap and nscd. However, the nscd process keeps growing in memory image size until it fills the system memory completely. Memory leak speed seems to correlate to the number of queries against nss.

The AD is Windows 2003 R2, using the R2 provided Identity management for UNIX (ex-SFU) to provide LDAP attributes. There are about 50 user objects and a dozen group objects matching the search filters specified in /etc/ldap.conf, plus the stuff that goes with AD by default.

On our mail server, which uses nss queries the most and suffers worst from this problem, the memory usage gets up to 95MB (RES) and 203MB (VIRT) in 12 hours as observed by top, and exceeds one gigabyte in 3-5 days.

I have tried turning paranoia mode on to restart the service periodically, but for some reason it does not restart, just quits the whole process (I don't know whether the paranoia mode is actually supposed to workin Ubuntu, as it is not mentioned in documentation provided with Ubuntu release, but it's parameters are in default config file, so I decided to try).

Just tried to install a fresh Ubuntu 8.04.1 server where I installed only libnss-ldap, nscd and their requirements, configured it for AD connection and left idle for the night. nscd memory usage was up to 100MB in the morning when it initially (few minutes after restart) is about 3.5MB.

*** Release and package info ***
Description: Ubuntu 8.04.1
Release: 8.04

  Installed: 2.7-10ubuntu4
  Candidate: 2.7-10ubuntu4
  Version table:
 *** 2.7-10ubuntu4 0
        500 http://fi.archive.ubuntu.com hardy-updates/universe Packages
        100 /var/lib/dpkg/status
     2.7-10ubuntu3 0
        500 http://fi.archive.ubuntu.com hardy/universe Packages

*** Sanitized /etc/ldap.conf ***
base dc=our,dc=ad,dc=domain
uri ldaps://dc1.our.ad.domain ldaps://dc2.our.ad.domain
ldap_version 3
rootbinddn <email address hidden>
scope sub
pam_password crypt
nss_base_passwd dc=our,dc=ad,dc=domain?sub?&(uidNumber=*)
nss_base_group dc=our,dc=ad,dc=domain?sub?&(gidNumber=*)
nss_map_objectclass posixAccount user
nss_map_objectclass shadowAccount user
nss_map_attribute uid sAMAccountName
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_attribute shadowLastChange pwdLastSet
nss_map_objectclass posixGroup group
nss_map_attribute uniqueMember member
pam_login_attribute sAMAccountName
pam_filter objectclass=User
nss_map_attribute userPassword unixUserPassword
ssl on
tls_checkpeer no
sasl_secprops maxssf=0
nss_initgroups_ignoreusers backup,bin,daemon,dhcp,dovecot,ftp,games,gnats,irc,klog,libuuid,list,lp,mail,man,news,ntp,postfix,proftpd,proxy,root,snmp,sshd,sync,sys,syslog,uucp,www-data

*** Password for <email address hidden> in /etc/ldap.secret ***

*** Using default package provided /etc/nscd.conf ***

Related branches

Revision history for this message
Doug Engert (deengert) wrote :

A memory leak in libnss-ldap over time can cause the nscd
process to grow extremely large. For example one nscd
process that had been running for three months was using
6GB of swap!

The problem is in the original Padl nss-ldap in at least versions
258, 261 and 265. Ubuntu Hardy uses 258, Karmic uses 261, and
the Padl current release is 265.

The ldap-nss.c do_init() may be called more then once,
to initialize an ldap session and save the session in
in __session.ls_conn and set the __session.ls_stat = LD_INITIALIZED
But it does not check the state to see if has be initialized,
and at line 1239: __session.ls_conn = NULL;

The attached patch to to the libnss-ldap_261-2.1Ubuntu4 fixes
the problem, by testing __session.ls_stat == LD_INITIALIZED

The ldap-nss.c also has a patch to call do_close twice that
I had previously turned it to Padl and is now in 265.

For testing purposes, the patch also adds atexit(do_atexit);
and a do_atexit routine to call do_close. This will then cause
the last session to be be freed. This make it much easier to
use valgrind to check for memory leaks. (in the nscd.c in lib6c
the _exit was change to exit so the atexit would be called.)

Debug versions of nscd and libnss-ldap where created, and
and used with valgrind to track down the memory leaks.
Attached is a script used with valgrind. The LD_PRELOAD was
needed so dynamic libs would not be unloaded, and valgrind
could find the symbol tables.

Revision history for this message
Doug Engert (deengert) wrote :
Revision history for this message
Doug Engert (deengert) wrote :
Doug Engert (deengert)
affects: glibc (Ubuntu) → libnss-ldap (Ubuntu)
Revision history for this message
Doug Engert (deengert) wrote :

Also reported to original developers as bug #418


Kees Cook (kees)
Changed in libnss-ldap (Ubuntu Lucid):
milestone: none → ubuntu-10.04-beta-2
status: New → Triaged
importance: Undecided → High
Changed in libnss-ldap (Ubuntu Hardy):
status: New → Triaged
importance: Undecided → High
Changed in libnss-ldap (Ubuntu Intrepid):
status: New → Triaged
importance: Undecided → High
Changed in libnss-ldap (Ubuntu Jaunty):
status: New → Triaged
importance: Undecided → High
Changed in libnss-ldap (Ubuntu Karmic):
status: New → Triaged
importance: Undecided → High
Thierry Carrez (ttx)
Changed in libnss-ldap (Ubuntu Lucid):
assignee: nobody → Mathias Gug (mathiaz)
Mathias Gug (mathiaz)
Changed in libnss-ldap (Ubuntu Lucid):
status: Triaged → In Progress
Mathias Gug (mathiaz)
Changed in libnss-ldap (Ubuntu Lucid):
status: In Progress → Fix Committed
Revision history for this message
Mathias Gug (mathiaz) wrote :

To setup a test environment:

 1. Install the following packages:

     sudo apt-get install slapd ldap-utils ldapscripts libnss-ldap

     NB: Don't enter a root password.

 2. Check out the openldap-dit project to quickly setup a default DIT:

     bzr co lp:~mathiaz/openldap-dit/add-ldapscripts-files
     cd add-ldapscripts-files/

     sudo ln -sf ~/add-ldapscripts-files /usr/share/slapd/openldap-dit

 3. Install the default DIT:

    sudo ./openldap-dit-setup.sh -y -d example.org -p mypwd

 4. Install the ldapscripts configuration files:

    sudo cp ldapscripts/* /etc/ldapscripts/

 5. Add a new group:

   sudo ldapaddgroup testgroup1

 6. Edit nsswitch.conf to use ldap as the first provider for the group service.

 7. Check that the testgroup1 is found:

    getent group testgroup1

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libnss-ldap - 264-2ubuntu2

libnss-ldap (264-2ubuntu2) lucid; urgency=low

  * Fix ldap sessions memory leak (LP: #292971).
 -- Mathias Gug <email address hidden> Fri, 02 Apr 2010 17:39:51 -0400

Changed in libnss-ldap (Ubuntu Lucid):
status: Fix Committed → Fix Released
Revision history for this message
Chuck Short (zulcss) wrote :

Closing this SRU based on the fact the Intrepid has reached EOL.

Changed in libnss-ldap (Ubuntu Intrepid):
status: Triaged → Won't Fix
affects: libnss-ldap (Ubuntu Lucid) → eglibc (Ubuntu Lucid)
Rolf Leggewie (r0lf)
Changed in eglibc (Ubuntu Jaunty):
status: Triaged → Won't Fix
Changed in eglibc (Ubuntu Karmic):
status: Triaged → Won't Fix
Revision history for this message
Rolf Leggewie (r0lf) wrote :

Hardy has seen the end of its life and is no longer receiving any updates. Marking the Hardy task for this ticket as "Won't Fix".

Changed in eglibc (Ubuntu Hardy):
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.