Autofs 5.1.1-1ubuntu2 crashes with segfault on startup

[Adam Pocock]

I'm running Wily Beta 2, updated to the latest packages in the wily repository, with autofs-ldap using mounts stored in LDAP. On startup systemd immediately puts autofs into a failed state as it segfaults when reading the auto.master.

Autofs & autofs-ldap version is 5.1.1-1ubuntu2.

[Impact]

This bug prevents the autofs-ldap module from loading, and causes it to segfault on startup. The proposed fix allows the module to load normally. The fix initialises configuration cache before it is first used (it is the access to the uninitialised cache which causes the segfault).

[Test Case]

Install the autofs-ldap package and have it read an automount map from LDAP. The necessary auto.master & ldap.conf configuration is dependent on the LDAP server that provides the automount maps.

[Regression Potential]

Given the shipped package does not load at all, it can't be a regression on the shipped binary.

[Other Info]

The log in journalctl looks like this:

Oct 05 15:47:45 <hostname> autofs[2232]: * Starting automount...
Oct 05 15:47:45 <hostname> automount[2240]: Starting automounter version 5.1.1, master map /etc/auto.master
Oct 05 15:47:45 <hostname> automount[2240]: using kernel protocol version 5.02
Oct 05 15:47:45 <hostname> automount[2240]: lookup_nss_read_master: reading master file /etc/auto.master
Oct 05 15:47:45 <hostname> automount[2240]: parse_init: parse(sun): init gathered global options: (null)
Oct 05 15:47:45 <hostname> automount[2240]: spawn_mount: mtab link detected, passing -n to mount
Oct 05 15:47:45 <hostname> automount[2240]: spawn_umount: mtab link detected, passing -n to mount
Oct 05 15:47:45 <hostname> automount[2240]: lookup_read_master: lookup(file): read entry /net
Oct 05 15:47:45 <hostname> automount[2240]: lookup_read_master: lookup(file): read entry +dir:/etc/auto.master.d
Oct 05 15:47:45 <hostname> automount[2240]: lookup_nss_read_master: reading master dir /etc/auto.master.d
Oct 05 15:47:45 <hostname> automount[2240]: lookup(dir): dir map /etc/auto.master.d missing or not readable
Oct 05 15:47:45 <hostname> automount[2240]: lookup(file): failed to read included master map dir:/etc/auto.master.d
Oct 05 15:47:45 <hostname> automount[2240]: lookup_read_master: lookup(file): read entry +auto.master
Oct 05 15:47:45 <hostname> automount[2240]: lookup_nss_read_master: reading master files auto.master
Oct 05 15:47:45 <hostname> automount[2240]: parse_init: parse(sun): init gathered global options: (null)
Oct 05 15:47:45 <hostname> automount[2240]: lookup_nss_read_master: reading master ldap auto.master
Oct 05 15:47:45 <hostname> kernel: automount[2240]: segfault at 0 ip 00007fb87ade5bcb sp 00007ffc30b34550 error 4 in lookup_ldap.so[7fb87adca000+38000]
Oct 05 15:47:47 <hostname> autofs[2232]: ...fail!
Oct 05 15:47:47 <hostname> systemd[1]: autofs.service: Control process exited, code=exited status=1

This occurs whenever the service is started with the line in auto.master which loads the ldap mounts. The line which causes the issue is "+ldap:automountMapName=auto.master.linux,ou=automount,ou=<location>,ou=<group>,o=<group>,dc=<base>,dc=com", and this configuration worked on 14.04, 14.10 and 15.04.

[Sebastien Bacher (seb128)]

 Thank you for taking the time to report this bug and helping to make Ubuntu better. Please try to obtain a backtrace following the instructions at http://wiki.ubuntu.com/DebuggingProgramCrash and upload the backtrace (as an attachment) to the bug report. This will greatly help us in tracking down your problem.

[Adam Pocock (craigacp)]

This is the backtrace I got out of apport-cli, but I couldn't figure out where the debug symbols for autofs are. I followed the instructions to add extra debug symbols (https://wiki.ubuntu.com/DebuggingProgramCrash).

[Brian Murray (brian-murray)]

I've gone ahead and manually retraced the crash file.

StacktraceAddressSignature: /usr/sbin/automount:11:/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so+1bbcb:/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so+1bc48:/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so+1c50c:/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so+1df87:/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so+e56a:/usr/sbin/automount+15db6:/usr/sbin/automount+166e7:/usr/sbin/automount+172b1:/usr/lib/x86_64-linux-gnu/autofs/lookup_file.so+a253:/usr/sbin/automount+16701:/usr/sbin/automount+17323:/usr/sbin/automount+29378:/usr/sbin/automount+b42e:/usr/sbin/automount+256ac:/lib/x86_64-linux-gnu/libpthread-2.21.so+76aa
StacktraceSource:
 #0 0x00007fedfa228bcb in ?? () from /mnt/sec-machines/apport-retrace/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so
 #1 0x00007fedfa228c48 in ?? () from /mnt/sec-machines/apport-retrace/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so
 #2 0x00007fedfa22950c in ?? () from /mnt/sec-machines/apport-retrace/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so
 #3 0x00007fedfa22af87 in defaults_get_ldap_timeout () from /mnt/sec-machines/apport-retrace/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so
 #4 0x00007fedfa21b56a in lookup_init () from /mnt/sec-machines/apport-retrace/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so
 #5 0x00005626d6cdfdb6 in open_lookup ()
 #6 0x00005626d6ce06e7 in ?? ()
 #7 0x00005626d6ce12b1 in lookup_nss_read_master ()
 #8 0x00007fedfaabb253 in lookup_read_master () from /mnt/sec-machines/apport-retrace/usr/lib/x86_64-linux-gnu/autofs/lookup_file.so
 #9 0x00005626d6ce0701 in ?? ()
 #10 0x00005626d6ce1323 in lookup_nss_read_master ()
 #11 0x00005626d6cf3378 in master_read_master ()
 #12 0x00005626d6cd542e in main ()
 #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
   [Error: pthread_cond_wait.S was not found in source tree]
 #1 0x00005626d6cef6ac in ?? ()
 #2 0x00007fedfdf756aa in start_thread (arg=0x7fedfe3ac700) at pthread_create.c:333
   [Error: pthread_create.c was not found in source tree]
 #3 0x00007fedfd6eeeed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
   [Error: clone.S was not found in source tree]
 #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
   [Error: pthread_cond_wait.S was not found in source tree]
 #1 0x00005626d6ce4773 in ?? ()
 #2 0x00007fedfdf756aa in start_thread (arg=0x7fedfe38a700) at pthread_create.c:333
   [Error: pthread_create.c was not found in source tree]
 #3 0x00007fedfd6eeeed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
   [Error: clone.S was not found in source tree]
 #0 0x00007fedfa228bcb in ?? () from /mnt/sec-machines/apport-retrace/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so
 #1 0x00007fedfa228c48 in ?? () from /mnt/sec-machines/apport-retrace/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so
 #2 0x00007fedfa22950c in ?? () from /mnt/sec-machines/apport-retrace/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so
 #3 0x00007fedfa22af87 in defaults_get_ldap_timeout () from /mnt/sec-machines/apport-retrace/usr/lib/x86_64-linux-gnu/autofs/lookup_ldap.so
 #4 0x000...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mike Dunphy (mike-dunphy)]

Unfortunately I just rolled 20 servers to 15.10 and this bug bit me big time. Is there a workaround or estimate when
complete or do I need to roll back to 15.04. Is it possible to just install the autofs from the 15.04 distro ?

[Adam Pocock (craigacp)]

I've been running the autofs-ldap & autofs binaries from 14.04 in 15.10, they work fine.

I had different issues with the 14.10 and 15.04 binaries (on their respective releases), which never worked with LDAP, but appeared to hard lock rather than segfault so it was harder to track down.

[Malcolm Scott (malcscott)]

This is a regression due to recent (upstream?) refactoring of the lookup_ldap module's lookup_init function (modules/lookup_ldap.c).

Nothing has initialised the config cache in lib/defaults.c before the call to defaults_get_ldap_timeout on line 1668.

Previously, this module's first call to defaults.c was defaults_get_searchdns(); that calls defaults_read_config to initialise the config cache, whereas defaults_get_ldap_timeout doesn't do this.

Patch attached.

Possibly, for completeness, more functions in defaults.c (e.g. defaults_get_ldap_network_timeout) should check that the config cache is initialised before reading from it (at the risk of opening the config file repeatedly).

[Malcolm Scott (malcscott)]

And here's the patch in debdiff form, for convenience.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "autofs_lookup_ldap_crash.diff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Dhawal Yogesh Bhanushali (imkin)]

I have tried the patch attached and found that the automount does not crash anymore with the error.

However I could not find the automount actually mounting anything.

[Dhawal Yogesh Bhanushali (imkin)]

Appologies for the above update. I had made building the package. The issue gets resolved with the patch.

Thanks for patch. Hopefully this would be pushed soon.

[Sören (soren-brinkmann)]

I rebuilt autofs with the patch from comment 8. That seems to work well for me.

[Chris Privitere (cprivite)]

Another workaround for folks still waiting for the real patch to be released...bind to LDAP through SSSD instead of using ldap-client. Then purge autofs-ldap off your system and install libpam-sss, then autofs will use sssd to get the ldap netgroups and you won't run into this segfault.

[Michael Terry (mterry)]

Thanks so much for the patch! I've uploaded it to xenial (with a slight modification -- moved the defaults_read_config out to lookup_init, to make the patch slightly more generic/future-proof).

This should probably be SRU'd to wily too, but lets see how it works in xenial first.

[Launchpad Janitor (janitor)]

This bug was fixed in the package autofs - 5.1.1-1ubuntu3

---------------
autofs (5.1.1-1ubuntu3) xenial; urgency=medium

  [ Malcolm Scott ]
  * d/p/fix-lookup-ldap-crash.patch:
    - Fix crash on startup due to uninitialized config. LP: #1503034

  [ Michael Terry ]
  * d/p/fix-lookup-ldap-crash.patch:
    - Adjust above patch to be slightly more generic

 -- Michael Terry <email address hidden> Thu, 19 Nov 2015 17:31:09 -0500

[C. Alex. North-Keys (erlkonig-talisman)]

Just upgraded to Wily here as well and I'm running into the failing autofs too. Moving from ldap client to sssd is more dramatic than desired here - how soon will the 1ubuntu3 be available in the Ubuntu Wily repos? As of the time of this post, only 5.1.1-1ubuntu2 is showing in the Wily repo.

[Mathew Hodson (mathew-hodson)]

In order for this to be SRUed to Wily, it needs to be made compliant with https://wiki.ubuntu.com/StableReleaseUpdates (Please add a rational, test case, and regression potential to the description of this bug.)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Adam Pocock (craigacp)]

I edited the description in line to make it mostly in line with the SRU wiki. It's hard to provide a test case without having a full ldap client and server configuration, plus an NFS system. I don't have access to the server side configs so I can't provide those.

What extra information is necessary to get this into the Wily repository?

[Brian Murray (brian-murray)]

I've uploaded this to the wily -proposed queue for review by the SRU team.

[Brian Murray (brian-murray)]

Hello Adam, or anyone else affected,

Accepted autofs into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autofs/5.1.1-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mike Dunphy (mike-dunphy)]

Thanks so much Brian, I tested it out of wily-proposed and it works great for me.
Regards
-mjd

[Paul Gaborit (paul-gaborit)]

 I tested the new wily-proposed version (5.1.1-1ubuntu2.1 ) and it works great for me.

Thank you !

[Adam Pocock (craigacp)]

The wily-proposed version (5.1.1-1ubuntu2.1) fixed it for me too.

Thanks Brian, it's much appreciated.

Adam

[Launchpad Janitor (janitor)]

This bug was fixed in the package autofs - 5.1.1-1ubuntu2.1

---------------
autofs (5.1.1-1ubuntu2.1) wily-proposed; urgency=medium

  [ Malcolm Scott ]
  * d/p/fix-lookup-ldap-crash.patch:
    - Fix crash on startup due to uninitialized config. LP: #1503034

  [ Michael Terry ]
  * d/p/fix-lookup-ldap-crash.patch:
    - Adjust above patch to be slightly more generic

 -- Brian Murray <email address hidden> Wed, 27 Jan 2016 11:07:01 -0800

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for autofs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Robie Basak (racb)]

Did this patch get sent upstream? We're still having to maintain the Ubuntu delta here.