winbind crashes on authentication (winbind_pam_auth)

[John Shearar]

Binary package hint: samba

This box was configured as a domain member on a window2k8 Active Directory domain. It was recently upgraded from Intrepid 8.10 , running samba 2:3.2.3-1ubuntu3.6, to Jaunty 9.04, running 2:3.3.2-1ubuntu3.2, and now the winbind daemon frequently crashes on authentication (but occasionally it will work for several auths with no problem).

# lsb_release -rd
Description: Ubuntu 9.04
Release: 9.04

# apt-cache policy winbind
winbind:
  Installed: 2:3.3.2-1ubuntu3.2
  Candidate: 2:3.3.2-1ubuntu3.2
  Version table:
 *** 2:3.3.2-1ubuntu3.2 0
        500 http://za.archive.ubuntu.com jaunty-updates/main Packages
        500 http://za.archive.ubuntu.com jaunty-security/main Packages
        100 /var/lib/dpkg/status
     2:3.3.2-1ubuntu3 0
        500 http://za.archive.ubuntu.com jaunty/main Packages

To reproduce the problem:

# wbinfo -t
checking the trust secret via RPC calls succeeded
# wbinfo -a john
Enter john's password:
plaintext password authentication succeeded
Enter john's password:^C
Interupted by signal.

/var/log/auth.log shows the authentication as successful...
Jan 5 09:56:24 havelock sshd[8729]: pam_winbind(sshd:auth): user 'john' granted access

... but /var/log/samba/log.winbindd: (debug 4) indicates the crash

[2010/01/05 09:56:24, 3] winbindd/winbindd_pam.c:winbindd_pam_auth(827)
  [ 8729]: pam auth john
*** glibc detected *** /usr/sbin/winbindd: double free or corruption (!prev): 0x00007faef5b91d90 ***
======= Backtrace: =========
/lib/libc.so.6[0x7faef1465cb8]
/lib/libc.so.6(cfree+0x76)[0x7faef1468276]
/usr/lib/libtalloc.so.1[0x7faef196f888]
/usr/lib/libtalloc.so.1(talloc_free+0xd8)[0x7faef1971b38]
/usr/sbin/winbindd[0x7faef37dacd7]
/usr/sbin/winbindd[0x7faef37db443]
/usr/sbin/winbindd(main+0xd6a)[0x7faef37dc299]
/lib/libc.so.6(__libc_start_main+0xe6)[0x7faef140c5a6]
/usr/sbin/winbindd[0x7faef37da249]

I have duplicated the box as a vm, and upgraded that to Karmic, which has the same symptom, although with a slightly different error. I -think- the problems are related, so I'll include it :

Description: Ubuntu 9.10
Release: 9.10

winbind:
  Installed: 2:3.4.0-3ubuntu5.1
  Candidate: 2:3.4.0-3ubuntu5.1
  Version table:
 *** 2:3.4.0-3ubuntu5.1 0
        500 http://za.archive.ubuntu.com karmic-updates/main Packages
        100 /var/lib/dpkg/status
     2:3.4.0-3ubuntu5 0
        500 http://za.archive.ubuntu.com karmic/main Packages

On any winbind authentication: ssh, console login, wbinfo -a

/var/log/auth.log: pam_winbind.so debug

Jan 5 10:40:23 testhavelock sshd[3306]: pam_winbind(sshd:auth): Verify user 'john'
Jan 5 10:40:23 testhavelock sshd[3306]: pam_winbind(sshd:auth): request wbcLogonUser succeeded
Jan 5 10:40:23 testhavelock sshd[3306]: pam_winbind(sshd:auth): user 'john' granted access
Jan 5 10:40:23 testhavelock sshd[3306]: pam_winbind(sshd:auth): Returned user was 'john'
Jan 5 10:40:23 testhavelock sshd[3306]: pam_winbind(sshd:auth): [pamh: 0x7f59513206c0] LEAVE: pam_sm_authenticate returning 0 (PAM_SUCCESS)
Jan 5 10:40:23 testhavelock sshd[3306]: pam_winbind(sshd:account): [pamh: 0x7f59513206c0] ENTER: pam_sm_acct_mgmt (flags: 0x0000)

/var/log/samba/log.winbindd: (debug 10)

[2010/01/05 10:40:23, 3] winbindd/winbindd_pam.c:827(winbindd_pam_auth)
  [ 3306]: pam auth john
[2010/01/05 10:40:23, 10] winbindd/winbindd_cache.c:492(refresh_sequence_number)
  refresh_sequence_number: FFAD time ok
[2010/01/05 10:40:23, 10] winbindd/winbindd_cache.c:537(refresh_sequence_number)
  refresh_sequence_number: FFAD seq number is now 207165758
[2010/01/05 10:40:23, 5] winbindd/winbindd_cache.c:1161(resolve_alias_to_username)
  resolve_alias_to_username: backend query returned NT_STATUS_NOT_IMPLEMENTED
[2010/01/05 10:40:23, 10] winbindd/winbindd_dual.c:125(async_request)
  Sending request to child pid 3296 (domain=FFAD)
[2010/01/05 10:40:23, 10] winbindd/winbindd_cache.c:2667(cache_retrieve_response)
  Retrieving response for pid 3296
[2010/01/05 10:40:23, 10] winbindd/winbindd_cache.c:2689(cache_retrieve_response)
  Retrieving extra data length=222
[2010/01/05 10:40:23, 6] winbindd/winbindd.c:834(new_connection)
  accepted socket 21
*** glibc detected *** /usr/sbin/winbindd: corrupted double-linked list: 0x00007fe687b142b0 ***
======= Backtrace: =========
/lib/libc.so.6[0x7fe68401fdd6]
/lib/libc.so.6[0x7fe6840201f2]
/lib/libc.so.6[0x7fe6840224c9]
/lib/libc.so.6(__libc_malloc+0x6e)[0x7fe6840247ee]
/usr/lib/libtalloc.so.1(_talloc_zero+0x16d)[0x7fe68474166d]
/usr/sbin/winbindd[0x7fe6863b8b8b]
/usr/sbin/winbindd(run_events+0x139)[0x7fe68647bd99]
/usr/sbin/winbindd(main+0xb5b)[0x7fe6863ba04b]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7fe683fc8abd]
/usr/sbin/winbindd[0x7fe6863b7a69]

More detailed logs and GDB backtraces are attached.

[Thierry Carrez (ttx)]

Couldn't find anything relevant on upstream bugtracker. Let me know if you could also test a lucid upgrade in a vm to check if 3.4.3 behaves any better in your configuration.

[John Shearar (john-shearar)]

The problem doesn't seem to occur in samba 3.4.3-2ubuntu1 in my test environment.

samba:
  Installed: 2:3.4.3-2ubuntu1
  Candidate: 2:3.4.3-2ubuntu1
  Version table:
 *** 2:3.4.3-2ubuntu1 0
        500 http://za.archive.ubuntu.com lucid/main Packages
        100 /var/lib/dpkg/status

Thank you for your time!

[John Shearar (john-shearar)]

It is not feasible to upgrade my production boxes to a development release, and I would prefer not to mix'n'match sources from different releases (there-in lies madness). Since lucid is still a way off, would it be possible for this version of samba to be backported to karmic?

[Thierry Carrez (ttx)]

@John: Sure. The bug has been nominated for karmic and jaunty, now it's a matter of identifying the necessary patch, since I didn't find anything close in samba's bugzilla.

[John Shearar (john-shearar)]

@Thierry: I don't know if it's helpful, but I took a look at the release
notes for the later versions of samba and found three fixes that roughly
relate to the same module, at least:

(3.4.3)
* BUG 6793 <https://bugzilla.samba.org/show_bug.cgi?id=6793> : Fix segfault
in winbindd_pam_auth.
* BUG 6811 <https://bugzilla.samba.org/show_bug.cgi?id=6811> : Fix reference
to freed memory in pam_winbind.
* BUG 6840 <https://bugzilla.samba.org/show_bug.cgi?id=6840> : Fix crash in
pam_winbind.

Perhaps you've already disqualified them; I don't know enough about the
codebase to confirm one way or the other.

Regards,
John

On Fri, Jan 15, 2010 at 10:10 AM, Thierry Carrez
<email address hidden>wrote:

> @John: Sure. The bug has been nominated for karmic and jaunty, now it's
> a matter of identifying the necessary patch, since I didn't find
> anything close in samba's bugzilla.
>
> --
> winbind crashes on authentication (winbind_pam_auth)
> https://bugs.launchpad.net/bugs/503402
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in “samba” package in Ubuntu: Fix Released
> Status in “samba” source package in Jaunty: New
> Status in “samba” source package in Karmic: New
>
> Bug description:
> Binary package hint: samba
>
> This box was configured as a domain member on a window2k8 Active Directory
> domain. It was recently upgraded from Intrepid 8.10 , running samba
> 2:3.2.3-1ubuntu3.6, to Jaunty 9.04, running 2:3.3.2-1ubuntu3.2, and now the
> winbind daemon frequently crashes on authentication (but occasionally it
> will work for several auths with no problem).
>
> # lsb_release -rd
> Description: Ubuntu 9.04
> Release: 9.04
>
> # apt-cache policy winbind
> winbind:
> Installed: 2:3.3.2-1ubuntu3.2
> Candidate: 2:3.3.2-1ubuntu3.2
> Version table:
> *** 2:3.3.2-1ubuntu3.2 0
> 500 http://za.archive.ubuntu.com jaunty-updates/main Packages
> 500 http://za.archive.ubuntu.com jaunty-security/main Packages
> 100 /var/lib/dpkg/status
> 2:3.3.2-1ubuntu3 0
> 500 http://za.archive.ubuntu.com jaunty/main Packages
>
> To reproduce the problem:
>
> # wbinfo -t
> checking the trust secret via RPC calls succeeded
> # wbinfo -a john
> Enter john's password:
> plaintext password authentication succeeded
> Enter john's password:^C
> Interupted by signal.
>
> /var/log/auth.log shows the authentication as successful...
> Jan 5 09:56:24 havelock sshd[8729]: pam_winbind(sshd:auth): user 'john'
> granted access
>
> ... but /var/log/samba/log.winbindd: (debug 4) indicates the crash
>
> [2010/01/05 09:56:24, 3] winbindd/winbindd_pam.c:winbindd_pam_auth(827)
> [ 8729]: pam auth john
> *** glibc detected *** /usr/sbin/winbindd: double free or corruption
> (!prev): 0x00007faef5b91d90 ***
> ======= Backtrace: =========
> /lib/libc.so.6[0x7faef1465cb8]
> /lib/libc.so.6(cfree+0x76)[0x7faef1468276]
> /usr/lib/libtalloc.so.1[0x7faef196f888]
> /usr/lib/libtalloc.so.1(talloc_free+0xd8)[0x7faef1971b38]
> /usr/sbin/winbindd[0x7faef37dacd7]
> /usr/sbin/winbindd[0x7faef37db443]
> /usr/sbin/winbindd(main+0xd6a)[0x7faef37dc299]
> /lib/libc.so.6(__libc_start_main+0xe6)[0x7faef140c5a6]
> /u...

[Thierry Carrez (ttx)]

Thanks for the pointers, it's very useful. I'd try the first one first, it seems closer to what you experience. I'll see if I can spend some time getting a patch in a PPA for you to test the fix.

[Chuck Short (zulcss)]

We havent tracked down the fix for this yet.

Regards
chuck

[Ray Van Dolson (rvandolson)]

Just to clarify, you've tried the 3.3.x patches in Samba bug 6793[1] and they do not resolve the issue? Also am trying to track this problem down.

[1] https://bugzilla.samba.org/show_bug.cgi?id=6793

[Ray Van Dolson (rvandolson)]

I was attempting to track this down for RHEL5 (see here[1]). When I apply the patch generated by this commit[2], I can no longer reproduce the winbind segfault.

This is against RHEL5's 3.3.8 and, I would assume, would be appropriate for Ubuntu's 3.3.x version as well.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=565915
[2] http://gitweb.samba.org/?p=samba.git;a=commit;h=62a1d9101cf0c2d45f81ba703cfdef5f42006b3f

[Thierry Carrez (ttx)]

Thanks, Ray ! So it's fixed upstream in 3.4.3 / 3.3.9, which confirms John's experience. we should push this to the next Samba SRU.

[Clint Byrum (clint-fewbar)]

Since Jaunty is EOL, closing Jaunty task as Won't Fix.