sssd fails to find memberof.so

Bug #746981 reported by Matt Mossholder on 2011-04-01
90
This bug affects 13 people
Affects Status Importance Assigned to Milestone
sssd (Ubuntu)
High
Timo Aaltonen
Natty
High
Unassigned
Oneiric
High
Timo Aaltonen

Bug Description

Binary package hint: sssd

1) Natty Beta 1
2) sssd 1.2.1-4.1ubuntu3
3) sssd works with group memberships in LDAP (worked in maverick, as long as you worked around the VERY similar bug #658909)
4) sssd exits, complaining that memberof.so cannot be located.

This is NOT the same as bug #658909. The symptoms are the same, but the solution in that bug does not work. strace shows that sssd is stat'ing the /usr/lib/ldb/modules/ldb directory, which contains memberof.so , but it still exits, complaining it cannot be found. Setting LDB_MODULES_PATH does not seem to resolve the issue.

Fri Apr 1 00:45:59 2011) [sssd] [ldb] (0): WARNING: Module [memberof] not found - do you need to set LDB_MODULES_PATH?
(Fri Apr 1 00:45:59 2011) [sssd] [ldb] (0): Unable to load modules for /var/lib/sss/db/cache_MOSSHOLDER.COM.ldb: (null)

Steve Langasek (vorlon) wrote :

Confirmed. The issue here is that the ldb module interface has changed between libldb0 and libldb1, and the memberof.so module built by sssd is still using the old API - so libldb can't do anything with it.

There are new upstream versions out, that probably add support for the new libldb.

Fabrice, you did the upload of sssd for the libldb transition - are you interested in following through on this for natty?

Changed in sssd (Ubuntu):
assignee: nobody → Fabrice Coutadeur (fabricesp)
importance: Undecided → High
status: New → Triaged

The commits:

23e8d84320ae8b76d244764c02e44036e96cd4df
21f28bdbab10881b9fb0b890dfa15af429326606
29dfae2a89551026f861f1f857187c22e30730c9
bd880fde928e0cb0eee5d59e2fd5f26d75698b5c

Are the necessary upstream changes to fix this issue. They don't apply cleanly on the 1.2.x branch since the primary Makefile changed locations, but the merges should be fairly easy to manage.

Also, 0b52717b76bf306afd30bbeb6d6c619365cfb548 should give you a fairly good idea of how to auto-detect the memberof location in your build scripts.

One unfortunate side-effect of some poor decisions made by libldb upstream: SSSD will now need a trivial rebuild against any new version of libldb that you want to ship (if any part of the version number changes).

Fabrice Coutadeur (fabricesp) wrote :

Thanks for the pointers: will check those commits now

Changed in sssd (Ubuntu):
status: Triaged → In Progress
Fabrice Coutadeur (fabricesp) wrote :

Hi,

I've uploaded the patched package to my ppa (https://launchpad.net/~fabricesp/+archive/ppa/+packages).
Could you please check that the fix works correctly?

Thanks,
Fabrice

Fabrice,
 Thanks for the help! It looks like some of the files are missing from the
packages... in particular the libsss_*.so files that provide the backends
(like libsss_ldap.so and libsss_krb5.so).

Here is a sample log entry:

(Fri Apr 8 16:31:54 2011) [sssd] [monitor_service_init] (3): Initializing
D-BUS Service
Unable to load ldap module with path
(/usr/lib/x86_64-linux-gnu/sssd/libsss_ldap.so), error: /usr/lib/x86
_64-linux-gnu/sssd/libsss_ldap.so: cannot open shared object file: No such
file or directory

    Thanks again,

     --Matt

On Fri, Apr 8, 2011 at 4:06 PM, Fabrice Coutadeur <email address hidden>wrote:

> Hi,
>
> I've uploaded the patched package to my ppa (
> https://launchpad.net/~fabricesp/+archive/ppa/+packages<https://launchpad.net/%7Efabricesp/+archive/ppa/+packages>
> ).
> Could you please check that the fix works correctly?
>
> Thanks,
> Fabrice
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/746981
>
> Title:
> sssd fails to find memberof.so
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/746981/+subscribe
>

Fabrice Coutadeur (fabricesp) wrote :

Good catch! It seems that it was missing since the beginning of the Natty version.

I've updated the package in my ppa: could you please try again?

thanks,
Fabrice

Getting closer! I now end up with a /usr/lib/x86_64-linux-gnu/sssd/
directory, full of links like libsss_{ipa,krb5,ldap,proxy,simple}.so, but
the files they actually point to are missing.

    Thanks!
     --Matt

On Sat, Apr 9, 2011 at 2:20 AM, Fabrice Coutadeur <email address hidden>wrote:

> Good catch! It seems that it was missing since the beginning of the
> Natty version.
>
> I've updated the package in my ppa: could you please try again?
>
> thanks,
> Fabrice
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/746981
>
> Title:
> sssd fails to find memberof.so
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/746981/+subscribe
>

Fabrice Coutadeur (fabricesp) wrote :

New package uploaded, with fixed links. This should work now.

Thanks for your patience!

Fabrice

Fabrice Coutadeur (fabricesp) wrote :

(uploaded in my ppa)

Download full text (14.4 KiB)

More progress! We seem to be past the stuff that has to do with missing
files now. However, it looks like the patches didn't quite do the trick.
Something is causing the LDAP entries not to get returned. In fact, it
doesn't even seem to try and process anything past the first user found.
Here are the log entries (sssd -d 10 -f):

(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]] [fo_set_port_status]
(4): Marking port 389 of server 'auth.mossholder.com' as 'working'
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[set_server_common_status] (4): Marking server 'auth.mossholder.com' as
'working'
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (6): calling ldap_search_ext with
[(&(uid=*)(objectclass=posixAccount))][dc=mossho
lder,dc=com].
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [objectClass]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [uid]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [userPassword]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [uidNumber]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [gidNumber]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [gecos]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [homeDirectory]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [loginShell]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [krbPrincipalName]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [cn]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [modifyTimestamp]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [shadowLastChange]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [shadowMin]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [shadowMax]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [shadowWarning]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [shadowInactive]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [shadowExpire]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [shadowFlag]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [krbLastPwdChange]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [krbPasswordExpiration]
(Sat Apr 9 17:27:42 2011) [sssd[be[MOSSHOLDER.COM]]]
[sdap_get_generic_send] (7): Requesting attrs: [pwdAttr...

That's an unrelated issue. You wouldn't see this with 'enumerate = false', for one thing.

What it's telling you is that 300s isn't long enough to fully process all users, groups and group memberships. Chances are that you have a very large or complicated group setup.

Try setting 'ldap_enumeration_refresh = 600'.

Stephen,

     I'm pretty sure 300 seconds is more than enough, because A) it worked
previously, and B) this is a home setup.. I have less that 30 total groups
and users in LDAP.

     I did try changing the setting to 600 seconds, and the results are the
same. I also tried setting ldap_search_timeout = 60, with no effect.

     I probably should have mentioned previously, that with sssd running
from the latest debs, any process that attempts to lookup a user or group
via NSS or PAM hangs. Even "getent passwd"...

     --Matt

On Mon, Apr 11, 2011 at 8:46 AM, Stephen Gallagher <
<email address hidden>> wrote:

> That's an unrelated issue. You wouldn't see this with 'enumerate =
> false', for one thing.
>
> What it's telling you is that 300s isn't long enough to fully process
> all users, groups and group memberships. Chances are that you have a
> very large or complicated group setup.
>
> Try setting 'ldap_enumeration_refresh = 600'.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/746981
>
> Title:
> sssd fails to find memberof.so
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/746981/+subscribe
>

joyride (joyride) wrote :

I also tried you ppa packages, and ssh hangs when trying to login and I do not really get any good debug output from sssd or ssh.

Is there anything else I can do to help move this to towards resolution?

     Thanks,

              --Matt

Kiall Mac Innes (kiall) wrote :

Just adding another confirmation of the issue with `getent passwd` etc hanging after installing from the PPA mentioned above.

Franz (franz.pammer) wrote :

Hi, I have the same problem wtih the packages from PPA for fabrice_sp (1.2.1-4.1ubuntu4~ppa3)

  id <LDAP username>
  getent passwd

dosen't return

i have also testet the packages from natty, and i have the same errors as comment 0

thx
Franz

urusha (urusha) wrote :

Confirming too:
1. natty version doesn't start at all with the same error.
2. nss queries hang with version from ppa. And we don't use enumeration in our environment.

Sorry folks, this bug completely fell off my radar (mostly because no one reported it upstream).

I should point out that SSSD 1.2.x is EOL in favor of SSSD 1.5.x LTM (long-term maintenance).

It would be best to get your package maintainer to update to a supported package. Otherwise, I can't guarantee that you will see any fixes from upstream that will help.

At a minimum, Ubuntu should update to SSSD 1.2.4, as it includes many bugfixes and one security fix compared to 1.2.1.

Fabrice Coutadeur (fabricesp) wrote :

Hi,

I've packaged 1.5.7 in my ppa (https://launchpad.net/~fabricesp/+archive/experimental): could you please check if that issue is gone with that version, and that the package works correctly?

If so, I'll push it to the archive, try to get get it backported and try to get it updated in Debian, as this is where the package comes from.

Thanks,
Fabrice

Tried the updated packages and got the same errors as comment #5, missing libsss_ldap.so

urusha (urusha) wrote :

1.5.7. The same:
[sssd[be[TELROS.LAN]]] [load_backend_module] (0): Unable to load ldap module with path (/usr/lib/i386-linux-gnu/sssd/libsss_ldap.so), error: /usr/lib/i386-linux-gnu/sssd/libsss_ldap.so: cannot open shared object file: No such
file or directory

Fabrice Coutadeur (fabricesp) wrote :

Hi,

Fixed package has been uploaded to the previous ppa. Thanks for testing it!

Fabrice

Yep, that's fixed it for me.

Kiall Mac Innes (kiall) wrote :

1.5.7-0ubuntu1~natty2 from the PPA fixed it here aswell - thanks! Lets hope you can get this into natty :)

urusha (urusha) wrote :

Now works fine. Thanks.
Is it possible to build it for lucid too?

Michael Jeanson (mjeanson) wrote :

As far as I know, a version upgrade is out of the question for an SRU. The only way to get this fixed in Natty is to have a working patch against 1.2.1, correct me if I'm wrong.

The upgrade to the 1.5.x series should definitely be done for Oneiric though.

In response to comment #26 by Michael, keeping sssd 1.5.x out of Natty seems rather silly, as sssd 1.2.x never, ever worked with Natty in the first place. sssd 1.2.x might have compiled, and been available for install, but it was never functional. The restrictions on version upgrades in SRUs are there to ensure compatibility within a release. Since sssd have been broken the entire time, I would think the correct thing to do in this case would be to deploy the newer version, rather than hoping someone is willing to invest their time and energy in trying to get 1.2.x to work.

Just my $0.02.

Michael Jeanson (mjeanson) wrote :

I fully agree with you, it's just that policies are usually hard to circumvent. The first step in any case is to get 1.5.x in oneiric so we have something to SRU in natty. I have started work on that using the great packaging of Fabrice Coutadeur, ding-libs is in the upload queue and sssd will follow when it's goes trough.

Fabrice Coutadeur (fabricesp) wrote :

Hi Michael,

Where did you uploaded ding-libs? In Ubuntu, for new packages, you have to follow the New package process. Anyway,as a MOTU, I have upload rights (I just don't have the time right now to work on it properly), so I was planning to upload it by myself.

The package should be upgraded in Debian (this is where the original package for 1.2 comes from), so if you could upload it there, it would be great.

Thanks,
Fabrice

Michael Jeanson (mjeanson) wrote :

Hi Fabrice,

I followed the new package process, you can consult the needs-packaging bug : LP#790362 It was sponsored and is now waiting in new. I'll contact the debian maintainer to see what is plans are, I agree it would make more sense to merge from debian but I wanted to have something in the archive early so it could get proper testing this cycle. I'll contact some people at Canonical corporate services, they were very interested in sssd at last UDS and could probably do some testing.

Michael

Using the updated Package from Post #19 Above,

I am getting a segfault on libsss_ldap.so

In /var/log/syslog:
Jun 9 18:16:34 pre-deploy kernel: [ 1324.100853] sssd_be[4488]: segfault at 0 ip b6c8313e sp bfe02320 error 4 in libsss_ldap.so.1.0.0[b6c69000+67000]
Jun 9 18:16:35 pre-deploy sssd[be[CODERYTE]]: Starting up

After setting up sssd.conf, and starting it. I can do both 'id username' and 'getent passwd' successfully. But a login attempt with correct password timesout and this is the corresponding error. Login with incorrect password bumps out immediately (as expected, with no error)

I thought I would post here first, as this was in trying to fix the original issue of the bug. If there is any more useful info I can provide, let me know. Create a new bug if necessary as well.

Jakub Hrozek (jakub-hrozek) wrote :

Richard,

are you running 1.5.7 from the PPA? I skimmed through the upstream bugs fixed since 1.5.7 and I could not see any crasher bugs. It would be very helpful if you could grab a core file and attach it either here or even to upstream bug tracker (https://fedorahosted.org/sssd/).

I am indeed using running 1.5.7 from the PPA.

I do not know what 'grab a core file' would entail sorry. I do not normally debug and report at this level :/

If you could PM me some instructions to get the needed info, I would be more than happy to assist and get this crash reported the proper way :)

Jakub Hrozek (jakub-hrozek) wrote :

Sorry Richard, I should have explained myself in greater detail. A core file is a file that contains a memory dump of a program at a time it crashed. It is very useful for debugging the crash.

On some systems (production systems in general), generating core files is disabled, as they can potentially take a lot of disk space. You can check whether your system would generate core files by running "ulimit -c" in a terminal. If the output says "0", generating core files is disabled. To enable core files, run "ulimit -c unlimited".

Then, from the same terminal, restart the sssd service and run the case that was crashing sssd for you. You should see a file named "core.XXXXX" where XXXXX is sssd process ID in the root directory "/" as this is where sssd sets its working directory to.

If the core file does not get created after the crash, the file /proc/sys/kernel/core_pattern tell where the core files gets created by default.

Richard, please open a new bug, as this crash is not related to the issue described here.

Please include:
sssd.conf (sanitized as needed)
Your PAM stack (not sure what it is on Ubuntu. On Fedora it would be /etc/pam.d/system-auth)
sssd_<domainname>.log (set debug_level = 9 in the [domain/<domainname>] section of sssd.conf and sanitize as needed)

Also include a description of the exact steps to reproduce. Does it occur any time you attempt to log in?

Stephen,

https://fedorahosted.org/sssd/ticket/896

I hope that is correct for you :) If you need more please let me know. (This is a great learning experience for me)

Stéphane Graber (stgraber) wrote :

I just uploaded 1.5.8 to Oneiric.
Adding a Natty and an Oneiric task and marking the Oneiric one as Fix released.

Changed in sssd (Ubuntu Natty):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Fabrice Coutadeur (fabricesp)
Changed in sssd (Ubuntu Oneiric):
assignee: Fabrice Coutadeur (fabricesp) → nobody
status: In Progress → Fix Released
Franz (franz.pammer) wrote :

Hi,

I have a fresh install of Oneiric, with the sssd version 1.5.8-0ubuntu3
and I have the following errors with default config of sssd.conf

(Thu Sep 1 10:42:04 2011) [sssd[pam]] [ldb] (0): Unable to load modules for /var/lib/sss/db/sssd.ldb: (null)
(Thu Sep 1 10:42:04 2011) [sssd[pam]] [sss_process_init] (0): fatal error initializing resp_ctx
(Thu Sep 1 10:42:05 2011) [sssd[nss]] [ldb] (0): WARNING: Module [memberof] not found - do you need to set LDB_MODULES_PATH?
(Thu Sep 1 10:42:05 2011) [sssd[nss]] [ldb] (0): Unable to load modules for /var/lib/sss/db/sssd.ldb: (null)
(Thu Sep 1 10:42:05 2011) [sssd[nss]] [sss_process_init] (0): fatal error initializing resp_ctx
(Thu Sep 1 10:42:06 2011) [sssd[pam]] [ldb] (0): WARNING: Module [memberof] not found - do you need to set LDB_MODULES_PATH?
(Thu Sep 1 10:42:06 2011) [sssd[pam]] [ldb] (0): Unable to load modules for /var/lib/sss/db/sssd.ldb: (null)
(Thu Sep 1 10:42:06 2011) [sssd[pam]] [sss_process_init] (0): fatal error initializing resp_ctx
(Thu Sep 1 10:42:07 2011) [sssd[nss]] [ldb] (0): WARNING: Module [memberof] not found - do you need to set LDB_MODULES_PATH?
(Thu Sep 1 10:42:07 2011) [sssd[nss]] [ldb] (0): Unable to load modules for /var/lib/sss/db/sssd.ldb: (null)
(Thu Sep 1 10:42:07 2011) [sssd[nss]] [sss_process_init] (0): fatal error initializing resp_ctx

Franz (franz.pammer) wrote :

Hi,

I think sssd is with the wrong libldb version linked

libldb1 = 1:1.1.2~git20110807-1

ln -s /usr/lib/ldb/modules/ldb/memberof.so /usr/lib/i386-linux-gnu/ldb/modules/ldb/

sssd -i -d 10
(Fri Sep 2 16:49:24 2011) [sssd] [check_file] (1): lstat for [/var/run/nscd/socket] failed: [2][No such file or directory].
ldb: module version mismatch in src/ldb_modules/memberof.c : ldb_version=1.1.2 module_version=1.1.1
ldb: failed to initialise module /usr/lib/i386-linux-gnu/ldb/modules/ldb/memberof.so : Unavailable
(Fri Sep 2 16:49:24 2011) [sssd] [load_configuration] (0): The confdb initialization failed
(Fri Sep 2 16:49:24 2011) [sssd] [main] (1): Error loading configuration database: [5]: Input/output error

Timo Aaltonen (tjaalton) wrote :

apparently broken again after a libldb update.

Changed in sssd (Ubuntu Oneiric):
assignee: nobody → Timo Aaltonen (tjaalton)
status: Fix Released → In Progress
Bruno Léon (bruno-leon) wrote :

Might be usefull to know that Timo has a working package in his ppa.

tags: added: rls-mgr-o-tracking
Timo Aaltonen (tjaalton) wrote :

indeed, there is bug 860297 for a FFE to update to 1.5.13 as well as fixing this and a bunch of other grave bugs.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sssd - 1.5.13-0ubuntu1

---------------
sssd (1.5.13-0ubuntu1) oneiric; urgency=low

  * FFE: New upstream release. (LP: #860297)
    - control: Add libunistring-dev to build-depends.
    - sssd.install: Include libipa_hbac.so*.
  * Rebuild against current libldb1, and use the multiarch path
    for libldb modules. (LP: #746981)
  * sssd.default:
    - Move the option to run as daemon here.
    - Add option that makes the daemon to use logfiles. (LP: #859602)
  * sssd.upstart:
    - Don't start before net-device-up. (LP: #812943)
    - Source /etc/default/sssd. (LP: #812943)
  * rules: Install the Python API files to /usr/share/sssd, as discussed
    with upstream. (LP: #859611)
  * fix-python-api-path.dpatch: Use the new location for the API files.
    (LP: #859611)
  * libpam-sss.pam-auth-update:
    - Add 'forward_pass' to auth stack to fix ecryptfs mounts. (LP: #826643)
    - Add pam_localuser.so to account stack to allow local users to log in.
      (LP: #860488)
  * control: sssd now Recommends libpam-sss and libnss-sss, since sssd is
    mostly useless without them. (LP: #767337)
 -- Timo Aaltonen <email address hidden> Tue, 27 Sep 2011 06:03:41 +0300

Changed in sssd (Ubuntu Oneiric):
status: In Progress → Fix Released
Timo Aaltonen (tjaalton) wrote :

11.04 is EOL

Changed in sssd (Ubuntu Natty):
assignee: Fabrice Coutadeur (fabricesp) → nobody
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.