ntlm_auth --helper-protocol=squid-2.5-ntlmssp report segfault

Bug #1578576 reported by sense on 2016-05-05
108
This bug affects 18 people
Affects Status Importance Assigned to Milestone
samba
Unknown
Unknown
samba (Ubuntu)
High
Ubuntu Security Team
Trusty
High
Marc Deslauriers
Wily
High
Marc Deslauriers
Xenial
High
Marc Deslauriers

Bug Description

System version: Ubuntu 14.04.4 LTS
Squid version: 2:4.3.8+dfsg-0ubuntu0.14.04.2
Winbind version: 2:4.3.8+dfsg-0ubuntu0.14.04.2 upgrade to 2:4.3.9+dfsg-0ubuntu0.14.04.1

My ubuntu server installed Squid to perform http_proxy. Squid auth the Active Directory users(Win2003) by winbind, settings like follow:

...
auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
...

Everything is ok until I upgrade winbind from 2:4.3.8+dfsg-0ubuntu0.14.04.2 to 2:4.3.9+dfsg-0ubuntu0.14.04.1 today. Squid can't auth the AD users anymore. There are some error log in the cache.log:

...
2016/05/05 13:50:45| ERROR: NTLM Authentication Helper '0x7fb54812af68' crashed!.
2016/05/05 13:50:45| ERROR: NTLM Authentication validating user. Error returned 'BH Internal error'
...

And there are some error log in the /var/log/syslog:

...
May 5 09:26:08 ocelot kernel: [ 187.793014] ntlm_auth[4543]: segfault at 8 ip 00007f10aad619b0 sp 00007ffc1ed0e778 error 4 in libsamba-security.so.0[7f10aad56000+1b000]
May 5 09:26:08 ocelot kernel: [ 188.207378] ntlm_auth[5062]: segfault at 8 ip 00007f41ada4d9b0 sp 00007ffebc1a1108 error 4 in libsamba-security.so.0[7f41ada42000+1b000]
May 5 09:26:08 ocelot kernel: [ 188.438501] ntlm_auth[4535]: segfault at 8 ip 00007f73fa26a9b0 sp 00007ffc1c63f808 error 4 in libsamba-security.so.0[7f73fa25f000+1b000]
May 5 09:26:35 ocelot kernel: [ 214.949867] ntlm_auth[5063]: segfault at 8 ip 00007fc1b5c459b0 sp 00007fffd3d5b398 error 4 in libsamba-security.so.0[7fc1b5c3a000+1b000]
May 5 09:26:35 ocelot kernel: [ 215.026850] ntlm_auth[5067]: segfault at 8 ip 00007f782ba5c9b0 sp 00007ffe59d96aa8 error 4 in libsamba-security.so.0[7f782ba51000+1b000]
...

And now I downgraded winbind to 2:4.1.6+dfsg-1ubuntu2(I can't downgrade it to 2:4.3.8+dfsg-0ubuntu0.14.04.2, it keeps telling me "Unable to find a version..."), Squid auth_param ntlm works again.

sense (opaperjam) on 2016-05-05
description: updated
sense (opaperjam) on 2016-05-05
description: updated
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in samba (Ubuntu):
status: New → Confirmed
ewaldmire (ewaldmire) wrote :

This broke web browsing for my 325 employees this morning. Trying to research how to roll back to 2:4.1.6+dfsg-1ubuntu2. Please provide instructions if you can to help others affected. Thanks for reporting this and your help!

Mich (michelebalazs) wrote :

Yep, we had to turn off single signon

Changed in samba (Ubuntu):
assignee: nobody → Ubuntu Security Team (ubuntu-security)
importance: Undecided → High
Marc Deslauriers (mdeslaur) wrote :

Just so I can try and reproduce this in a test environment, are you authenticating to a Windows AD, or to a Samba AD?

Mich (michelebalazs) wrote :

Windows AD

i have rolled back and all good now.

Marc Deslauriers (mdeslaur) wrote :

I can't seem to reproduce this issue. Can someone get a backtrace?

Are you able to reproduce it by using ntlm_auth directly on the command line, like so?:

$ sudo ntlm_auth --helper-protocol=squid-2.5-basic
MYDOMAIN\testuser mypassword

$ sudo ntlm_auth --helper-protocol=squid-2.5-ntlmssp
MYDOMAIN\testuser mypassword

ewaldmire (ewaldmire) wrote :

When I run the commands on the command line they don't exit and pressing return gives:

ERR

Pressing "?" gives (for ntlmssp, not basic):

BH Query invalid

Typing about anything else gives me (again, only for ntlmssp, not basic):

BH SPNEGO request invalid prefix

I tried installing gdb and creating a backtrace, but I'm not sure what I'm doing - can't see to get anything useful. I'm reading this may be because apport is enabled?

It looks like apport creates files under /var/crash/_usr_bin_ntlm_auth.13.crash automatically - I've attached this hoping it has what you need.

sense (opaperjam) on 2016-05-06
description: updated
Patrick McKenna (phmckenna) wrote :

I am seeing the same errors. I am using ntlm_auth for Moodle access.

When attempting to use NTLM, I show this in the processes:

www-data 9866 9856 0 08:52 ? 00:00:00 [ntlm_auth] <defunct>

Paul Strinati (paul-strinati) wrote :

Same problem. Am running:

Ubuntu Server 14.04.1 LTS
Windbind: 2:4.3.9+dfsg-0ubuntu0.14.04.1
Samba: 2:4.3.9+dfsg-0ubuntu0.14.04.1
Squid3: 3.3.8-1ubuntu6.6

Authenticating against Active Directory - has been working really well for the last 18 months, then stopped working about a week ago.

Errors in cache.log:
2016/05/09 06:20:07| Too few ntlmauthenticator processes are running (need 1/10)
2016/05/09 06:20:07| Starting new helpers
2016/05/09 06:20:07| helperOpenServers: Starting 1/10 'ntlm_auth' processes
2016/05/09 06:20:07| ERROR: NTLM Authentication Helper '0x7f313ea68318' crashed!.
2016/05/09 06:20:07| ERROR: NTLM Authentication validating user. Error returned 'BH Internal error'
2016/05/09 06:20:08| WARNING: ntlmauthenticator #1 exited

Errors in syslog:
May 9 06:20:09 optsquidproxy kernel: [228590.127125] ntlm_auth[8850]: segfault at 8 ip 00007f201ec729b0 sp 00007ffda249aae8 error 4 in libsamba-security.so.0[7f201ec67000+1b000]

Squid is using pure NTLM authentication (taken from squid.conf):
### pure ntlm authentication
auth_param ntlm program /usr/bin/ntlm_auth --diagnostics --helper-protocol=squid-2.5-ntlmssp --domain=MYDOMAIN
auth_param ntlm children 10
auth_param ntlm keep_alive off

Raulo (raulo-olapodrido) wrote :

running an Apache2 server with NTLM authentication against an AD, stopped working with 500 Internal Server error since the Samba upgrade.

Apache config:
AuthType NTLM
AuthName "..."
NTLMAuth on
NTLMAuthHelper "/usr/bin/ntlm_auth --helper-protocol=squid-.5-ntlmssp"
NTLMBasicAuthoritative on
require valid-user

Syslog:
May 9 08:41:17 intranet-gvp kernel: [438844.377831] ntlm_auth[23011]: segfault at 8 ip 00007f512efcf9b0 sp 00007ffc8017cc78 error 4 in libsamba-security.so.0[7f512efc4000+1b000]
May 9 08:41:23 intranet-gvp kernel: [438849.956323] ntlm_auth[23148]: segfault at 8 ip 00007f39b08e99b0 sp 00007fff4ba86bb8 error 4 in libsamba-security.so.0[7f39b08de000+1b000]
May 9 08:41:30 intranet-gvp kernel: [438856.430960] ntlm_auth[23240]: segfault at 8 ip 00007f96a55309b0 sp 00007ffe0a7eaa98 error 4 in libsamba-security.so.0[7f96a5525000+1b000]
May 9 08:43:30 intranet-gvp kernel: [438977.462065] ntlm_auth[25264]: segfault at 8 ip 00007f874faf29b0 sp 00007ffd417d0478 error 4 in libsamba-security.so.0[7f874fae7000+1b000]
May 9 08:45:03 intranet-gvp kernel: [439070.043363] ntlm_auth[28559]: segfault at 8 ip 00007fb5af4769b0 sp 00007ffcc2b84918 error 4 in libsamba-security.so.0[7fb5af46b000+1b000]
May 9 08:47:12 intranet-gvp kernel: [439199.384723] ntlm_auth[30675]: segfault at 8 ip 00007f357d1439b0 sp 00007ffef1ea7c98 error 4 in libsamba-security.so.0[7f357d138000+1b000]
May 9 08:47:25 intranet-gvp kernel: [439211.944010] ntlm_auth[30822]: segfault at 8 ip 00007f89a24e49b0 sp 00007ffffb32b3c8 error 4 in libsamba-security.so.0[7f89a24d9000+1b000]
May 9 08:50:12 intranet-gvp kernel: [439379.146404] ntlm_auth[1121]: segfault at 8 ip 00007fdb6f7d19b0 sp 00007ffcf44e5728 error 4 in libsamba-security.so.0[7fdb6f7c6000+1b000]

After rollback to Samba packages 4.3.8, the system is running again.

$ pgrep ntlm -a
2192 ntlm_auth
4951 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
4952 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
4953 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
6074 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
6503 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
6596 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
15376 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
17037 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
17149 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp

Jacek Kuzemczak (t-jacok-y) wrote :
Download full text (274.9 KiB)

We're getting the same thing on 16.04 - narrowed it down to the same segfault. Here's a strace of the apache process which should be useful, it seems to work fine on the first challenge but then fail on the second

strace: Process 6911 attached
accept4(4, {sa_family=AF_INET6, sin6_port=htons(42500), inet_pton(AF_INET6, "::ffff:10.10.0.210", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 26
gettimeofday({1462796465, 907646}, NULL) = 0
getsockname(26, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "::ffff:10.10.0.24", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
fcntl(26, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(26, F_SETFL, O_RDWR|O_NONBLOCK) = 0
gettimeofday({1462796465, 908578}, NULL) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810dac000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810daa000
read(26, "GET /ntlm-auth HTTP/1.1\r\nHost: jmk-dev-web\r\nUser-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nReferer: http://jmk-dev-web/\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\n\r\n", 8000) = 354
gettimeofday({1462796465, 909153}, NULL) = 0
gettimeofday({1462796465, 909345}, NULL) = 0
gettimeofday({1462796465, 909434}, NULL) = 0
stat("/var/www/jmk/ntlm-auth", {st_dev=makedev(202, 1), st_ino=263914, st_mode=S_IFDIR|0770, st_nlink=2, st_uid=101143, st_gid=33, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=2016/05/09-09:16:32, st_mtime=2016/04/22-12:27:32.672865000, st_ctime=2016/04/22-12:27:32.672865000}) = 0
open("/var/www/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/var/www/jmk/.htaccess", O_RDONLY|O_CLOEXEC) = 27
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810da8000
fstat(27, {st_dev=makedev(202, 1), st_ino=912136, st_mode=S_IFREG|0770, st_nlink=1, st_uid=101143, st_gid=33, st_blksize=4096, st_blocks=8, st_size=28, st_atime=2016/05/09-10:29:18.820558786, st_mtime=2016/05/09-10:29:15.964543846, st_ctime=2016/05/09-10:29:15.964543846}) = 0
read(27, "AddHandler php7.0-fcgi .php\n", 4096) = 28
read(27, "", 4096) = 0
gettimeofday({1462796465, 910687}, NULL) = 0
close(27) = 0
open("/var/www/jmk/ntlm-auth/.htaccess", O_RDONLY|O_CLOEXEC) = 27
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810da6000
fstat(27, {st_dev=makedev(202, 1), st_ino=269927, st_mode=S_IFREG|0770, st_nlink=1, st_uid=101143, st_gid=33, st_blksize=4096, st_blocks=8, st_size=170, st_atime=2016/05/09-09:16:32.004000000, st_mtime=2016/04/22-13:54:12.424865000, st_ctime=2016/04/22-13:54:12.424865000}) = 0
read(27, "AuthName \"NTLM Test\"\r\nNTLMAuth on\r\nNTLMAuthHelper \"/usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp\"\r\nNTLMBasicAuthoritative on\r\nAuthType NTLM\r\nRequire valid-user\r\n", 4096) = 170
read(27, "", 4096) = 0
close(27) = ...

Marc Deslauriers (mdeslaur) wrote :

Could someone who is experiencing this regression with Samba 4.3.9 please report it to the Samba developers here:

https://bugzilla.samba.org/

Please attach the bug report with this one once it's been filed.

I'd file the bug myself, but they may require more details on the problematic environment.

Thanks!

Paul Strinati (paul-strinati) wrote :

Have submitted to the Samba developers: https://bugzilla.samba.org/show_bug.cgi?id=11912

Paul Carroll (pcarroll) wrote :

Could anyone give a quick guide on how to downgrade samba/winbind to workaround this issue while it is fixed?

sense (opaperjam) wrote :

My downgrade command:

aptitude -V install winbind=2:4.1.6+dfsg-1ubuntu2 samba=2:4.1.6+dfsg-1ubuntu2 libwbclient0=2:4.1.6+dfsg-1ubuntu2 samba-libs=2:4.1.6+dfsg-1ubuntu2 python-samba=2:4.1.6+dfsg-1ubuntu2 samba-vfs-modules=2:4.1.6+dfsg-1ubuntu2 libldb1=1:1.1.16-1 samba-common=2:4.1.6+dfsg-1ubuntu2 samba-common-bin=2:4.1.6+dfsg-1ubuntu2 samba-dsdb-modules=2:4.1.6+dfsg-1ubuntu2 python-ldb=1:1.1.16-1 libnss-winbind=2:4.1.6+dfsg-1ubuntu2 libpam-winbind=2:4.1.6+dfsg-1ubuntu2

Thomas (t.c) on 2016-05-10
tags: added: regression-update
Sebastien Bacher (seb128) wrote :

Upstream has a patch in their bug now

Changed in samba (Ubuntu Trusty):
assignee: nobody → Marc Deslauriers (mdeslaur)
Changed in samba (Ubuntu Wily):
assignee: nobody → Marc Deslauriers (mdeslaur)
Changed in samba (Ubuntu Xenial):
assignee: nobody → Marc Deslauriers (mdeslaur)
Changed in samba (Ubuntu Trusty):
status: New → Confirmed
Changed in samba (Ubuntu Wily):
status: New → Confirmed
Changed in samba (Ubuntu Xenial):
status: New → Confirmed
Changed in samba (Ubuntu Trusty):
importance: Undecided → High
Changed in samba (Ubuntu Wily):
importance: Undecided → High
Changed in samba (Ubuntu Xenial):
importance: Undecided → High
Marc Deslauriers (mdeslaur) wrote :

I have uploaded test packages that include the upstream fix to the following PPA:

https://launchpad.net/~mdeslaur/+archive/ubuntu/testing

Once they finish building, could someone please test them and confirm it fixes the issue?

Once someone has tested them successfully, I will publish them as security updates.

Thanks!

Paul Strinati (paul-strinati) wrote :

Just tried and still getting the same problem:
samba:
  Installed: 2:4.3.9+dfsg-0ubuntu0.14.04.2~ppa1
  Candidate: 2:4.3.9+dfsg-0ubuntu0.14.04.2~ppa1
  Version table:
 *** 2:4.3.9+dfsg-0ubuntu0.14.04.2~ppa1 0
        500 http://ppa.launchpad.net/mdeslaur/testing/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     2:4.3.9+dfsg-0ubuntu0.14.04.1 0
        500 http://optubunturepository.MYDOMAIN.net/ubuntu/ trusty-updates/main amd64 Packages
        500 http://optubunturepository.MYDOMAIN.net/ubuntu/ trusty-security/main amd64 Packages
     2:4.1.6+dfsg-1ubuntu2 0
        500 http://optubunturepository.MYDOMAIN.net/ubuntu/ trusty/main amd64 Packages

syslog:
May 10 14:31:11 optsquidproxy kernel: [ 206.928248] ntlm_auth[2264]: segfault at 8 ip 00007f68e2aba9b0 sp 00007fff384ec2c8 error 4 in libsamba-security.so.0[7f68e2aaf000+1b000]

cache.log:
2016/05/10 14:32:42| WARNING: ntlmauthenticator #1 exited
2016/05/10 14:32:42| Too few ntlmauthenticator processes are running (need 1/10)
2016/05/10 14:32:42| Starting new helpers
2016/05/10 14:32:42| helperOpenServers: Starting 1/10 'ntlm_auth' processes
2016/05/10 14:32:42| ERROR: NTLM Authentication Helper '0x7f8368efb268' crashed!.
2016/05/10 14:32:42| ERROR: NTLM Authentication validating user. Error returned 'BH Internal error'

Marc Deslauriers (mdeslaur) wrote :

Thanks for testing Paul. Did you update all the binary packages in the PPA, including winbind and samba-libs, and did you reboot or restart all processes including squid?

Paul Strinati (paul-strinati) wrote :

Yep - added your testing PPA, then:

sudo apt-get update
sudo apt-get upgrade samba (which brought in all dependencies)

Then rebooted the server for good measure :)

The above error messages were from after the reboot, and I'm still getting issues when trying to authenticate via the squid proxy.

Marc Deslauriers (mdeslaur) wrote :

Thanks Paul. Could you please add a comment to the upstream bug?

Paul Strinati (paul-strinati) wrote :

Done :)

Stefan Metzmacher (metze) wrote :

It seems there're two similar bugs, I've created
https://bugzilla.samba.org/show_bug.cgi?id=11914 to track the 2nd problem

Marc Deslauriers (mdeslaur) wrote :

I have uploaded a test package for trusty that includes the proposed fix for Samba bug #11914 to the following PPA:

https://launchpad.net/~mdeslaur/+archive/ubuntu/testing

Paul Strinati (paul-strinati) wrote :

Thanks Marc - appears to work for me!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu0.14.04.3

---------------
samba (2:4.3.9+dfsg-0ubuntu0.14.04.3) trusty-security; urgency=medium

  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.
  * debian/rules: work around amd64 build failure (LP: #1585174)

 -- Marc Deslauriers <email address hidden> Tue, 24 May 2016 07:47:59 -0400

Changed in samba (Ubuntu Trusty):
status: Confirmed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu0.16.04.2

---------------
samba (2:4.3.9+dfsg-0ubuntu0.16.04.2) xenial-security; urgency=medium

  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.

 -- Marc Deslauriers <email address hidden> Fri, 20 May 2016 07:31:37 -0400

Changed in samba (Ubuntu Xenial):
status: Confirmed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu0.15.10.2

---------------
samba (2:4.3.9+dfsg-0ubuntu0.15.10.2) wily-security; urgency=medium

  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.

 -- Marc Deslauriers <email address hidden> Fri, 20 May 2016 08:09:44 -0400

Changed in samba (Ubuntu Wily):
status: Confirmed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu1

---------------
samba (2:4.3.9+dfsg-0ubuntu1) yakkety; urgency=medium

  * SECURITY REGRESSION: Updated to 4.3.9 to fix multiple regressions in
    the previous security updates. (LP: #1577739)
    - debian/control: bump tevent Build-Depends to 0.9.28.
  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.

 -- Marc Deslauriers <email address hidden> Wed, 25 May 2016 09:29:15 -0400

Changed in samba (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.