ntlm_auth --helper-protocol=squid-2.5-ntlmssp report segfault

[sense]

System version: Ubuntu 14.04.4 LTS
Squid version: 2:4.3.8+dfsg-0ubuntu0.14.04.2
Winbind version: 2:4.3.8+dfsg-0ubuntu0.14.04.2 upgrade to 2:4.3.9+dfsg-0ubuntu0.14.04.1

My ubuntu server installed Squid to perform http_proxy. Squid auth the Active Directory users(Win2003) by winbind, settings like follow:

...
auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
...

Everything is ok until I upgrade winbind from 2:4.3.8+dfsg-0ubuntu0.14.04.2 to 2:4.3.9+dfsg-0ubuntu0.14.04.1 today. Squid can't auth the AD users anymore. There are some error log in the cache.log:

...
2016/05/05 13:50:45| ERROR: NTLM Authentication Helper '0x7fb54812af68' crashed!.
2016/05/05 13:50:45| ERROR: NTLM Authentication validating user. Error returned 'BH Internal error'
...

And there are some error log in the /var/log/syslog:

...
May 5 09:26:08 ocelot kernel: [ 187.793014] ntlm_auth[4543]: segfault at 8 ip 00007f10aad619b0 sp 00007ffc1ed0e778 error 4 in libsamba-security.so.0[7f10aad56000+1b000]
May 5 09:26:08 ocelot kernel: [ 188.207378] ntlm_auth[5062]: segfault at 8 ip 00007f41ada4d9b0 sp 00007ffebc1a1108 error 4 in libsamba-security.so.0[7f41ada42000+1b000]
May 5 09:26:08 ocelot kernel: [ 188.438501] ntlm_auth[4535]: segfault at 8 ip 00007f73fa26a9b0 sp 00007ffc1c63f808 error 4 in libsamba-security.so.0[7f73fa25f000+1b000]
May 5 09:26:35 ocelot kernel: [ 214.949867] ntlm_auth[5063]: segfault at 8 ip 00007fc1b5c459b0 sp 00007fffd3d5b398 error 4 in libsamba-security.so.0[7fc1b5c3a000+1b000]
May 5 09:26:35 ocelot kernel: [ 215.026850] ntlm_auth[5067]: segfault at 8 ip 00007f782ba5c9b0 sp 00007ffe59d96aa8 error 4 in libsamba-security.so.0[7f782ba51000+1b000]
...

And now I downgraded winbind to 2:4.1.6+dfsg-1ubuntu2(I can't downgrade it to 2:4.3.8+dfsg-0ubuntu0.14.04.2, it keeps telling me "Unable to find a version..."), Squid auth_param ntlm works again.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[ewaldmire (ewaldmire)]

This broke web browsing for my 325 employees this morning. Trying to research how to roll back to 2:4.1.6+dfsg-1ubuntu2. Please provide instructions if you can to help others affected. Thanks for reporting this and your help!

[Mich (michelebalazs)]

Yep, we had to turn off single signon

[Marc Deslauriers (mdeslaur)]

Just so I can try and reproduce this in a test environment, are you authenticating to a Windows AD, or to a Samba AD?

[Mich (michelebalazs)]

Windows AD

i have rolled back and all good now.

[Marc Deslauriers (mdeslaur)]

I can't seem to reproduce this issue. Can someone get a backtrace?

Are you able to reproduce it by using ntlm_auth directly on the command line, like so?:

$ sudo ntlm_auth --helper-protocol=squid-2.5-basic
MYDOMAIN\testuser mypassword

$ sudo ntlm_auth --helper-protocol=squid-2.5-ntlmssp
MYDOMAIN\testuser mypassword

[ewaldmire (ewaldmire)]

When I run the commands on the command line they don't exit and pressing return gives:

ERR

Pressing "?" gives (for ntlmssp, not basic):

BH Query invalid

Typing about anything else gives me (again, only for ntlmssp, not basic):

BH SPNEGO request invalid prefix

I tried installing gdb and creating a backtrace, but I'm not sure what I'm doing - can't see to get anything useful. I'm reading this may be because apport is enabled?

It looks like apport creates files under /var/crash/_usr_bin_ntlm_auth.13.crash automatically - I've attached this hoping it has what you need.

[Patrick McKenna (phmckenna)]

I am seeing the same errors. I am using ntlm_auth for Moodle access.

When attempting to use NTLM, I show this in the processes:

www-data 9866 9856 0 08:52 ? 00:00:00 [ntlm_auth] <defunct>

[Paul Strinati (paul-strinati)]

Same problem. Am running:

Ubuntu Server 14.04.1 LTS
Windbind: 2:4.3.9+dfsg-0ubuntu0.14.04.1
Samba: 2:4.3.9+dfsg-0ubuntu0.14.04.1
Squid3: 3.3.8-1ubuntu6.6

Authenticating against Active Directory - has been working really well for the last 18 months, then stopped working about a week ago.

Errors in cache.log:
2016/05/09 06:20:07| Too few ntlmauthenticator processes are running (need 1/10)
2016/05/09 06:20:07| Starting new helpers
2016/05/09 06:20:07| helperOpenServers: Starting 1/10 'ntlm_auth' processes
2016/05/09 06:20:07| ERROR: NTLM Authentication Helper '0x7f313ea68318' crashed!.
2016/05/09 06:20:07| ERROR: NTLM Authentication validating user. Error returned 'BH Internal error'
2016/05/09 06:20:08| WARNING: ntlmauthenticator #1 exited

Errors in syslog:
May 9 06:20:09 optsquidproxy kernel: [228590.127125] ntlm_auth[8850]: segfault at 8 ip 00007f201ec729b0 sp 00007ffda249aae8 error 4 in libsamba-security.so.0[7f201ec67000+1b000]

Squid is using pure NTLM authentication (taken from squid.conf):
### pure ntlm authentication
auth_param ntlm program /usr/bin/ntlm_auth --diagnostics --helper-protocol=squid-2.5-ntlmssp --domain=MYDOMAIN
auth_param ntlm children 10
auth_param ntlm keep_alive off

[Raulo (raulo-olapodrido)]

running an Apache2 server with NTLM authentication against an AD, stopped working with 500 Internal Server error since the Samba upgrade.

Apache config:
AuthType NTLM
AuthName "..."
NTLMAuth on
NTLMAuthHelper "/usr/bin/ntlm_auth --helper-protocol=squid-.5-ntlmssp"
NTLMBasicAuthoritative on
require valid-user

Syslog:
May 9 08:41:17 intranet-gvp kernel: [438844.377831] ntlm_auth[23011]: segfault at 8 ip 00007f512efcf9b0 sp 00007ffc8017cc78 error 4 in libsamba-security.so.0[7f512efc4000+1b000]
May 9 08:41:23 intranet-gvp kernel: [438849.956323] ntlm_auth[23148]: segfault at 8 ip 00007f39b08e99b0 sp 00007fff4ba86bb8 error 4 in libsamba-security.so.0[7f39b08de000+1b000]
May 9 08:41:30 intranet-gvp kernel: [438856.430960] ntlm_auth[23240]: segfault at 8 ip 00007f96a55309b0 sp 00007ffe0a7eaa98 error 4 in libsamba-security.so.0[7f96a5525000+1b000]
May 9 08:43:30 intranet-gvp kernel: [438977.462065] ntlm_auth[25264]: segfault at 8 ip 00007f874faf29b0 sp 00007ffd417d0478 error 4 in libsamba-security.so.0[7f874fae7000+1b000]
May 9 08:45:03 intranet-gvp kernel: [439070.043363] ntlm_auth[28559]: segfault at 8 ip 00007fb5af4769b0 sp 00007ffcc2b84918 error 4 in libsamba-security.so.0[7fb5af46b000+1b000]
May 9 08:47:12 intranet-gvp kernel: [439199.384723] ntlm_auth[30675]: segfault at 8 ip 00007f357d1439b0 sp 00007ffef1ea7c98 error 4 in libsamba-security.so.0[7f357d138000+1b000]
May 9 08:47:25 intranet-gvp kernel: [439211.944010] ntlm_auth[30822]: segfault at 8 ip 00007f89a24e49b0 sp 00007ffffb32b3c8 error 4 in libsamba-security.so.0[7f89a24d9000+1b000]
May 9 08:50:12 intranet-gvp kernel: [439379.146404] ntlm_auth[1121]: segfault at 8 ip 00007fdb6f7d19b0 sp 00007ffcf44e5728 error 4 in libsamba-security.so.0[7fdb6f7c6000+1b000]

After rollback to Samba packages 4.3.8, the system is running again.

$ pgrep ntlm -a
2192 ntlm_auth
4951 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
4952 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
4953 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
6074 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
6503 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
6596 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
15376 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
17037 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
17149 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp

[Jacek Kuzemczak (t-jacok-y)]

We're getting the same thing on 16.04 - narrowed it down to the same segfault. Here's a strace of the apache process which should be useful, it seems to work fine on the first challenge but then fail on the second

strace: Process 6911 attached
accept4(4, {sa_family=AF_INET6, sin6_port=htons(42500), inet_pton(AF_INET6, "::ffff:10.10.0.210", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 26
gettimeofday({1462796465, 907646}, NULL) = 0
getsockname(26, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "::ffff:10.10.0.24", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
fcntl(26, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(26, F_SETFL, O_RDWR|O_NONBLOCK) = 0
gettimeofday({1462796465, 908578}, NULL) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810dac000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810daa000
read(26, "GET /ntlm-auth HTTP/1.1\r\nHost: jmk-dev-web\r\nUser-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nReferer: http://jmk-dev-web/\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\n\r\n", 8000) = 354
gettimeofday({1462796465, 909153}, NULL) = 0
gettimeofday({1462796465, 909345}, NULL) = 0
gettimeofday({1462796465, 909434}, NULL) = 0
stat("/var/www/jmk/ntlm-auth", {st_dev=makedev(202, 1), st_ino=263914, st_mode=S_IFDIR|0770, st_nlink=2, st_uid=101143, st_gid=33, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=2016/05/09-09:16:32, st_mtime=2016/04/22-12:27:32.672865000, st_ctime=2016/04/22-12:27:32.672865000}) = 0
open("/var/www/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/var/www/jmk/.htaccess", O_RDONLY|O_CLOEXEC) = 27
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810da8000
fstat(27, {st_dev=makedev(202, 1), st_ino=912136, st_mode=S_IFREG|0770, st_nlink=1, st_uid=101143, st_gid=33, st_blksize=4096, st_blocks=8, st_size=28, st_atime=2016/05/09-10:29:18.820558786, st_mtime=2016/05/09-10:29:15.964543846, st_ctime=2016/05/09-10:29:15.964543846}) = 0
read(27, "AddHandler php7.0-fcgi .php\n", 4096) = 28
read(27, "", 4096) = 0
gettimeofday({1462796465, 910687}, NULL) = 0
close(27) = 0
open("/var/www/jmk/ntlm-auth/.htaccess", O_RDONLY|O_CLOEXEC) = 27
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810da6000
fstat(27, {st_dev=makedev(202, 1), st_ino=269927, st_mode=S_IFREG|0770, st_nlink=1, st_uid=101143, st_gid=33, st_blksize=4096, st_blocks=8, st_size=170, st_atime=2016/05/09-09:16:32.004000000, st_mtime=2016/04/22-13:54:12.424865000, st_ctime=2016/04/22-13:54:12.424865000}) = 0
read(27, "AuthName \"NTLM Test\"\r\nNTLMAuth on\r\nNTLMAuthHelper \"/usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp\"\r\nNTLMBasicAuthoritative on\r\nAuthType NTLM\r\nRequire valid-user\r\n", 4096) = 170
read(27, "", 4096) = 0
close(27) = ...

[Marc Deslauriers (mdeslaur)]

Could someone who is experiencing this regression with Samba 4.3.9 please report it to the Samba developers here:

https://bugzilla.samba.org/

Please attach the bug report with this one once it's been filed.

I'd file the bug myself, but they may require more details on the problematic environment.

Thanks!

[Paul Strinati (paul-strinati)]

Have submitted to the Samba developers: https://bugzilla.samba.org/show_bug.cgi?id=11912

[Paul Carroll (pcarroll)]

Could anyone give a quick guide on how to downgrade samba/winbind to workaround this issue while it is fixed?

[sense (opaperjam)]

My downgrade command:

aptitude -V install winbind=2:4.1.6+dfsg-1ubuntu2 samba=2:4.1.6+dfsg-1ubuntu2 libwbclient0=2:4.1.6+dfsg-1ubuntu2 samba-libs=2:4.1.6+dfsg-1ubuntu2 python-samba=2:4.1.6+dfsg-1ubuntu2 samba-vfs-modules=2:4.1.6+dfsg-1ubuntu2 libldb1=1:1.1.16-1 samba-common=2:4.1.6+dfsg-1ubuntu2 samba-common-bin=2:4.1.6+dfsg-1ubuntu2 samba-dsdb-modules=2:4.1.6+dfsg-1ubuntu2 python-ldb=1:1.1.16-1 libnss-winbind=2:4.1.6+dfsg-1ubuntu2 libpam-winbind=2:4.1.6+dfsg-1ubuntu2

[Sebastien Bacher (seb128)]

Upstream has a patch in their bug now

[Marc Deslauriers (mdeslaur)]

I have uploaded test packages that include the upstream fix to the following PPA:

https://launchpad.net/~mdeslaur/+archive/ubuntu/testing

Once they finish building, could someone please test them and confirm it fixes the issue?

Once someone has tested them successfully, I will publish them as security updates.

Thanks!

[Paul Strinati (paul-strinati)]

Just tried and still getting the same problem:
samba:
  Installed: 2:4.3.9+dfsg-0ubuntu0.14.04.2~ppa1
  Candidate: 2:4.3.9+dfsg-0ubuntu0.14.04.2~ppa1
  Version table:
 *** 2:4.3.9+dfsg-0ubuntu0.14.04.2~ppa1 0
        500 http://ppa.launchpad.net/mdeslaur/testing/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     2:4.3.9+dfsg-0ubuntu0.14.04.1 0
        500 http://optubunturepository.MYDOMAIN.net/ubuntu/ trusty-updates/main amd64 Packages
        500 http://optubunturepository.MYDOMAIN.net/ubuntu/ trusty-security/main amd64 Packages
     2:4.1.6+dfsg-1ubuntu2 0
        500 http://optubunturepository.MYDOMAIN.net/ubuntu/ trusty/main amd64 Packages

syslog:
May 10 14:31:11 optsquidproxy kernel: [ 206.928248] ntlm_auth[2264]: segfault at 8 ip 00007f68e2aba9b0 sp 00007fff384ec2c8 error 4 in libsamba-security.so.0[7f68e2aaf000+1b000]

cache.log:
2016/05/10 14:32:42| WARNING: ntlmauthenticator #1 exited
2016/05/10 14:32:42| Too few ntlmauthenticator processes are running (need 1/10)
2016/05/10 14:32:42| Starting new helpers
2016/05/10 14:32:42| helperOpenServers: Starting 1/10 'ntlm_auth' processes
2016/05/10 14:32:42| ERROR: NTLM Authentication Helper '0x7f8368efb268' crashed!.
2016/05/10 14:32:42| ERROR: NTLM Authentication validating user. Error returned 'BH Internal error'

[Marc Deslauriers (mdeslaur)]

Thanks for testing Paul. Did you update all the binary packages in the PPA, including winbind and samba-libs, and did you reboot or restart all processes including squid?

[Paul Strinati (paul-strinati)]

Yep - added your testing PPA, then:

sudo apt-get update
sudo apt-get upgrade samba (which brought in all dependencies)

Then rebooted the server for good measure :)

The above error messages were from after the reboot, and I'm still getting issues when trying to authenticate via the squid proxy.

[Marc Deslauriers (mdeslaur)]

Thanks Paul. Could you please add a comment to the upstream bug?

[Paul Strinati (paul-strinati)]

Done :)

[Stefan Metzmacher (metze)]

It seems there're two similar bugs, I've created
https://bugzilla.samba.org/show_bug.cgi?id=11914 to track the 2nd problem

[Marc Deslauriers (mdeslaur)]

I have uploaded a test package for trusty that includes the proposed fix for Samba bug #11914 to the following PPA:

https://launchpad.net/~mdeslaur/+archive/ubuntu/testing

[Paul Strinati (paul-strinati)]

Thanks Marc - appears to work for me!

[Launchpad Janitor (janitor)]

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu0.14.04.3

---------------
samba (2:4.3.9+dfsg-0ubuntu0.14.04.3) trusty-security; urgency=medium

  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.
  * debian/rules: work around amd64 build failure (LP: #1585174)

 -- Marc Deslauriers <email address hidden> Tue, 24 May 2016 07:47:59 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu0.16.04.2

---------------
samba (2:4.3.9+dfsg-0ubuntu0.16.04.2) xenial-security; urgency=medium

  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.

 -- Marc Deslauriers <email address hidden> Fri, 20 May 2016 07:31:37 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu0.15.10.2

---------------
samba (2:4.3.9+dfsg-0ubuntu0.15.10.2) wily-security; urgency=medium

  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.

 -- Marc Deslauriers <email address hidden> Fri, 20 May 2016 08:09:44 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu1

---------------
samba (2:4.3.9+dfsg-0ubuntu1) yakkety; urgency=medium

  * SECURITY REGRESSION: Updated to 4.3.9 to fix multiple regressions in
    the previous security updates. (LP: #1577739)
    - debian/control: bump tevent Build-Depends to 0.9.28.
  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.

 -- Marc Deslauriers <email address hidden> Wed, 25 May 2016 09:29:15 -0400