Unable to log in with AD account after update

[Lyle Dietz]

[Impact]

The pam_winbind.so module is unusable in zesty. It won't load because of missing symbols:

Jun 21 13:17:05 zesty-pamwinbind-1677329 systemd: PAM unable to dlopen(pam_winbind.so): /lib/security/pam_winbind.so: cannot open shared object file: No such file or directory

This is due to the (re)introduction of patch fix-1584485.patch which changes the way this module is built, trying to statically link some libraries. That linking was incorrectly done.

The patch was subsequently removed, but later added back again by mistake during a sync.

A new version of the patch exists (https://code.launchpad.net/~ahasenack/ubuntu/+source/samba/+git/samba/+merge/323767), but upstream (Samba and Debian) isn't very fond of such a change and asked me to submit it for discussion to the samba-technical mailing list (https://lists.samba.org/archive/samba-technical/2017-June/121139.html).

That was done, but since this could take some time, we decided it's best to revert the patch again.

[Test Case]

In a zesty machine/container:
 * sudo apt install libpam-winbind winbind samba
 * tail -f /var/log/auth.log
 * perform a login on this machine. Via ssh, for example
 * the broken version will log this:
Jun 21 13:17:05 zesty-pamwinbind-1677329 systemd: PAM unable to dlopen(pam_winbind.so): /lib/security/pam_winbind.so: cannot open shared object file: No such file or directory
 * The fixed version will load pam_winbind.so just fine, but won't log anything (unless you fully setup winbind). It's easier to add "debug" to the pam_winbind.so lines in /etc/pam.d/common-* files and repeat the login, then you get to see it being loaded in the logs:
Jun 21 17:48:52 zesty-pamwinbind-1677329 sshd[18052]: pam_winbind(sshd:session): [pamh: 0x56460f355740] ENTER: pam_sm_open_session (flags: 0x0000)
Jun 21 17:48:52 zesty-pamwinbind-1677329 sshd[18052]: pam_winbind(sshd:session): [pamh: 0x56460f355740] LEAVE: pam_sm_open_session returning 0 (PAM_SUCCESS)

[Regression Potential]

This reversal has been done before and worked. Right now, the biggest regression potential is to add the broken patch back again.

Reversing this patch will also reintroduce bug #1584485, but I think the configuration that leads to that bug is asking for trouble and I stated as such in a comment (https://bugs.launchpad.net/ubuntu/+source/samba/+bug/1584485/comments/43). "winbind" should be listed after "files" or "compat", not before.

That being said, it is my opinion that having a working pam_winbind module benefits more users than the amount of users that could be affected by the particular configuration that leads to #1584485.

[Other Info]

Sorry for keeping both bugs open (#1644428 and #1677329), but the history on this issue is a bit complicated with multiple SRUs and regressions.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Björn Ramberg (bjoern-ramberg)]

Can confirm this, attached a log snippet on how the auth.log looks after upgrading to this version.

..
Nov 24 11:02:12 thismachine sudo: PAM unable to dlopen(pam_winbind.so): /lib/security/pam_winbind.so: cannot open shared object file: No such file or directory
Nov 24 11:02:12 thismachine sudo: PAM adding faulty module: pam_winbind.so
..

Replacing the new pam_winbind.so with one from previous version does work.

[Robert Euhus (euhus-liste1)]

I think the error message in the log is misleading. the problem is definitely not that /lib/security/pam_winbind.so is missing, since this file did not exist in 4.3.11+dfsg-0ubuntu0.14.04.1 which worked. The module is in the same place as before: /lib/x86_64-linux-gnu/security/pam_winbind.so

But instead I think that something went wrong with the statically linking patch which was the only change introduced there:
http://launchpadlibrarian.net/294673937/samba_2%3A4.3.11+dfsg-0ubuntu0.14.04.1_2%3A4.3.11+dfsg-0ubuntu0.14.04.2.diff.gz

Regards,
Robert Euhus

[Robert Euhus (euhus-liste1)]

Sorry, forgot the link to the bug which caused this change:
https://bugs.launchpad.net/ubuntu/+source/samba/+bug/1584485

[Louis Bouchard (louis)]

Hello,

The safest route for the time being is to revert back to 2:4.3.11+dfsg-0ubuntu0.14.04.1 as outlined in the other bug while we investigate the issue.

[Martin Pitt (pitti)]

Hello Lyle, or anyone else affected,

Accepted samba into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/samba/2:4.3.11+dfsg-0ubuntu0.14.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Martin Pitt (pitti)]

For the record, this just needs to be verified to fix the regression and still generally work. This should *not* be delayed for the usual 7 day maturing period as this fixes a regression.

[Robie Basak (racb)]

Please note that 2:4.3.11+dfsg-0ubuntu0.14.04.3 reverts the change introduced by 2:4.3.11+dfsg-0ubuntu0.14.04.2; apart from the changelog it is identical to 2:4.3.11+dfsg-0ubuntu0.14.04.1.

Please could someone affected by the regression introduced by 2:4.3.11+dfsg-0ubuntu0.14.04.2 confirm that 2:4.3.11+dfsg-0ubuntu0.14.04.3 from proposed fixes the regression?

We should also perform regular verification to make sure that this doesn't inadvertently break something else (eg. some non-determinism in the build, or changes in the build environment). Once that's done, we can release 2:4.3.11+dfsg-0ubuntu0.14.04.3 to trusty-updates, putting users back to a position before the regression.

[Robert Euhus (euhus-liste1)]

Is there any way I can help debugging this problem? I have no idea how to find out, what exactly is missing from the pam_winbind.so module. But I'm willing to learn! :)

I have no problem rebuilding packages to try out patches, or reinstalling from scratch. Please also let me know if You need any other info or logfiles.

[Björn Ramberg (bjoern-ramberg)]

Impressively fast guys, hats off for that!

"Please could someone affected by the regression introduced by 2:4.3.11+dfsg-0ubuntu0.14.04.2 confirm that 2:4.3.11+dfsg-0ubuntu0.14.04.3 from proposed fixes the regression?"

Just tested the trusty-proposed version (ubuntu0.14.04.3) and I can not see the issue anymore.

[Support Lifecycle (support-lifecycle-software-com)]

"Please could someone affected by the regression introduced by 2:4.3.11+dfsg-0ubuntu0.14.04.2 confirm that 2:4.3.11+dfsg-0ubuntu0.14.04.3 from proposed fixes the regression?" I can confirm this worked for us too, we had a bad day due to this :( glad it's over.

[Lyle Dietz (ritterwolf)]

Tested 2:4.3.11+dfsg-0ubuntu0.14.04.3 and it works for me.

[Robie Basak (racb)]

Thank you for the verifications! Could someone please also verify that "regular" samba is working using 2:4.3.11+dfsg-0ubuntu0.14.04.3, as well as the specific winbind regression we're addressing here? Some combination of smbclient, regular file sharing server, expected behaviour from the file manager GUI, etc.

I just want to make sure we've checked samba generally before releasing this, in case only winbind behaviour has been tested. I don't want to introduce a second regression.

When confirming, please describe what aspect you tested.

Thanks!

[Björn Ramberg (bjoern-ramberg)]

From a machine running 4.3.11+dfsg-0ubuntu0.14.04.3, I just tested smbclient towards DFS shares / windows 10 share, cifs mount of DFS share, sharing a local folder with ACL set (net usershare) and then connecting to if from another machine running 16.04 with samba4 2:4.3.11+dfsg-0ubuntu0.16.04.1.

All works as expected from my side.
Thanks guys!

[Launchpad Janitor (janitor)]

This bug was fixed in the package samba - 2:4.3.11+dfsg-0ubuntu0.14.04.3

---------------
samba (2:4.3.11+dfsg-0ubuntu0.14.04.3) trusty; urgency=high

  * Revert to version prior to the 2:4.3.11+dfsg-0ubuntu0.14.04.2
    which is causing regression with statically linked libpam_winbind.
    Removes d/p/fix-1584485.patch. LP: #1644428

 -- Louis Bouchard <email address hidden> Thu, 24 Nov 2016 15:40:40 +0100

[Robie Basak (racb)]

The verification of the Stable Release Update for samba has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Robie Basak (racb)]

Thank you to everyone involved in helping verify this fix.

[Robert Euhus (euhus-liste1)]

Thanks for the quick action!

[Björn Ramberg (bjoern-ramberg)]

Indeed, thanks a lot for very quick action.

[Martino Dell'Ambrogio (tillo)]

This seems to affect 2:4.3.11+dfsg-0ubuntu0.16.04.2 too, as I had exactly the same issue and I fixed it by downgrading everything sourced from samba to version 2:4.3.11+dfsg-0ubuntu0.16.04.1.

[Louis Bouchard (louis)]

Martino,

This is probably because you upgraded from xenial-proposed which has the broken package. This one hasn't been released and should not be used.

 samba | 2:4.3.11+dfsg-0ubuntu0.16.04.1 | xenial-updates | source, amd64, arm64, armhf, i386, powerpc, ppc64el, s390x
 samba | 2:4.3.11+dfsg-0ubuntu0.16.04.2 | xenial-proposed | source, amd64, arm64, armhf, i386, powerpc, ppc64el, s390x

I will see if it can be removed from the xenial-proposed archive

[Michael Iatrou (michael.iatrou)]

The same issue is observed on Xenial:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial

ii libpam-gnome-keyring:amd64 3.18.3-0ubuntu2 amd64
ii libpam-modules:amd64 1.1.8-3.2ubuntu2 amd64
ii libpam-modules-bin 1.1.8-3.2ubuntu2 amd64
ii libpam-pwquality:amd64 1.3.0-0ubuntu1 amd64
ii libpam-runtime 1.1.8-3.2ubuntu2 all
ii libpam-sss:amd64 1.13.4-1ubuntu1.2 amd64
ii libpam-systemd:amd64 229-4ubuntu16 amd64
ii libpam0g:amd64 1.1.8-3.2ubuntu2 amd64
ii python-samba 2:4.3.11+dfsg-0ubuntu0.16.04.3 amd64
ii samba 2:4.3.11+dfsg-0ubuntu0.16.04.3 amd64
ii samba-common 2:4.3.11+dfsg-0ubuntu0.16.04.3 all
ii samba-common-bin 2:4.3.11+dfsg-0ubuntu0.16.04.3 amd64
ii samba-dsdb-modules 2:4.3.11+dfsg-0ubuntu0.16.04.3 amd64
ii samba-libs:amd64 2:4.3.11+dfsg-0ubuntu0.16.04.3 amd64
ii samba-vfs-modules 2:4.3.11+dfsg-0ubuntu0.16.04.3 amd64

From /var/log/auth.log:
10:48:56 gcc-ubuntu-tst1 sudo: gcc : problem with defaults entries ; TTY=pts/18 ; PWD=/etc/sssd ;
10:48:56 gcc-ubuntu-tst1 sudo: PAM unable to dlopen(pam_winbind.so): /lib/security/pam_winbind.so: cannot open shared object file: No such file or directory
10:48:56 gcc-ubuntu-tst1 sudo: PAM adding faulty module: pam_winbind.so
10:48:56 gcc-ubuntu-tst1 sudo: gcc : TTY=pts/18 ; PWD=/etc/sssd ; USER=root ; COMMAND=/bin/more sssd.conf
10:48:56 gcc-ubuntu-tst1 sudo: pam_unix(sudo:session): session opened for user root by gcc(uid=0)
10:48:56 gcc-ubuntu-tst1 sudo: pam_unix(sudo:session): session closed for user root

[Tim Ritberg (xpert-reactos)]

Some problem on 17.04:

Apr 29 00:30:29 X1 systemd: PAM adding faulty module: pam_winbind.so
Apr 29 00:30:29 X1 systemd: pam_unix(systemd-user:session): session opened for user nobody by (uid=0)
Apr 29 00:31:46 X1 su[4553]: PAM unable to dlopen(pam_winbind.so): /lib/security/pam_winbind.so: Kann die Shared-Object-Datei nicht öffnen: Datei oder Verzeichnis nicht gefunden

[Martin Soentgenrath (m-soentgenrath)]

Also affects Version 2:4.5.8+dfsg-0ubuntu0.17.04.1

[jMurr (jmurchik)]

If manually create symlink /lib/x86_64-linux-gnu/security/ to /lib/security/ there is another error in auth.log:
 PAM unable to dlopen(pam_winbind.so): /lib/security/pam_winbind.so: undefined symbol: wbcCtxFree
 PAM adding faulty module: pam_winbind.so

[Christian Ehrhardt  (paelzer)]

Not sure if this should be a new bug, opening a zesty task for now until we are sure.
Summary: pam_winbind.so fails to open
Related: bug 1584485
Repro:
 $ apt-get install samba winbind libnss-winbind libpam-winbind
 $ sed -i -e 's/^passwd: compat/passwd: winbind compat/' /etc/nsswitch.conf
 $ systemctl restart smbd nmbd winbind
 $ tail -n 5 /var/log/auth.log
May 5 06:31:43 zesty-test groupadd[19598]: group added to /etc/group: name=sambashare, GID=118
May 5 06:31:43 zesty-test groupadd[19598]: group added to /etc/gshadow: name=sambashare
May 5 06:31:43 zesty-test groupadd[19598]: new group: name=sambashare, GID=118
May 5 06:32:08 zesty-test sudo: PAM unable to dlopen(pam_winbind.so): /lib/security/pam_winbind.so: cannot open shared object file: No such file or directory
May 5 06:32:08 zesty-test sudo: PAM adding faulty module: pam_winbind.so

Also the path is different as outlined before:
dpkg -S pam_winbind.so
libpam-winbind:amd64: /lib/x86_64-linux-gnu/security/pam_winbind.so

Confirmed at least on zesty for now, but I'm no windbind/pam expert so I struggle to see what is going on without diving much deeper - this might even be just the wrong way to set it up ... ? :-/

[Mario Lipinski (mario.lipinski)]

Since this bug was marked as "fixed" for the earlier release, I opened #1677329. Since I am not familiar with launchpad I failed to setup proper relationships for Zesty. So far I would say that libpam-winbind is completely broken on Zesty.

[Andreas Hasenack (ahasenack)]

I asked upstream (Debian and Samba) for a review of this patch:

https://lists.samba.org/archive/samba-technical/2017-June/121139.html

That could take a while, so until that happens, I'm proposing a different MP to fix this for now and that is to revert the broken patch one more time.

[Launchpad Janitor (janitor)]

This bug was fixed in the package samba - 2:4.5.8+dfsg-2ubuntu2

---------------
samba (2:4.5.8+dfsg-2ubuntu2) artful; urgency=medium

  * Add extra DEP8 tests to samba (LP: #1696823):
    - d/t/control: enable the new DEP8 tests
    - d/t/smbclient-anonymous-share-list: list available shares anonymously
    - d/t/smbclient-authenticated-share-list: list available shares using
      an authenticated connection
    - d/t/smbclient-share-access: create a share and download a file from it
    - d/t/cifs-share-access: access a file in a share using cifs
  * Ask the user if we can run testparm against the config file. If yes,
    include its stderr and exit status in the bug report. Otherwise, only
    include the exit status. (LP: #1694334)
  * If systemctl is available, use it to query the status of the smbd
    service before trying to reload it. Otherwise, keep the same check
    as before and reload the service based on the existence of the
    initscript. (LP: #1579597)
  * Remove d/p/fix-1584485.patch as it builds a broken pam_winbind
    module. There is a fixed version of that patch attached to
    #1677329 but it has not been vetted yet, so for now it's best
    to revert (again) so that pam_winbind can be used.
    (LP: #1677329, LP: #1644428)

 -- Andreas Hasenack <email address hidden> Mon, 19 Jun 2017 10:49:29 -0700

[Andrew Reis (areis422)]

Confirmed having this problem on fresh Zesty install:

Dell R410

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.04
Release: 17.04
Codename: zesty

$ dpkg -l | grep 'samba\|winbind\|nss' | awk '{print $2":"$3}'
libnss-resolve:amd64:232-21ubuntu4
libnss-winbind:amd64:2:4.5.8+dfsg-0ubuntu0.17.04.2
libnss3:amd64:2:3.28.4-0ubuntu0.17.04.2
libpam-winbind:amd64:2:4.5.8+dfsg-0ubuntu0.17.04.2
libwbclient0:amd64:2:4.5.8+dfsg-0ubuntu0.17.04.2
python-samba:2:4.5.8+dfsg-0ubuntu0.17.04.2
samba:2:4.5.8+dfsg-0ubuntu0.17.04.2
samba-common:2:4.5.8+dfsg-0ubuntu0.17.04.2
samba-common-bin:2:4.5.8+dfsg-0ubuntu0.17.04.2
samba-dsdb-modules:2:4.5.8+dfsg-0ubuntu0.17.04.2
samba-libs:amd64:2:4.5.8+dfsg-0ubuntu0.17.04.2
samba-vfs-modules:2:4.5.8+dfsg-0ubuntu0.17.04.2
winbind:2:4.5.8+dfsg-0ubuntu0.17.04.2

[Andrew Reis (areis422)]

Updated packages to latest.
Also having this issue. After symlinking /lib/x86_64-linux-gnu/security to /lib/security, receiving:

Jul 19 12:05:41 hostname login[28882]: PAM unable to dlopen(pam_winbind.so): /lib/security/pam_winbind.so: undefined symbol: wbcCtxFree
Jul 19 12:05:41 hostname login[28882]: PAM adding faulty module: pam_winbind.so

user@hostname:/lib$ dpkg -l | grep 'samba\|winbind' | awk '{print $2" --- "$3}'
libnss-winbind:amd64 --- 2:4.5.8+dfsg-0ubuntu0.17.04.4
libpam-winbind:amd64 --- 2:4.5.8+dfsg-0ubuntu0.17.04.4
libwbclient0:amd64 --- 2:4.5.8+dfsg-0ubuntu0.17.04.4
python-samba --- 2:4.5.8+dfsg-0ubuntu0.17.04.4
samba --- 2:4.5.8+dfsg-0ubuntu0.17.04.4
samba-common --- 2:4.5.8+dfsg-0ubuntu0.17.04.4
samba-common-bin --- 2:4.5.8+dfsg-0ubuntu0.17.04.4
samba-dsdb-modules --- 2:4.5.8+dfsg-0ubuntu0.17.04.4
samba-libs:amd64 --- 2:4.5.8+dfsg-0ubuntu0.17.04.4
samba-vfs-modules --- 2:4.5.8+dfsg-0ubuntu0.17.04.4
winbind --- 2:4.5.8+dfsg-0ubuntu0.17.04.4

[Andreas Hasenack (ahasenack)]

The MP has been up for weeks. Here is a debdiff.

[piviul (piviul)]

I have the same problem. The only way to have winbind works again is to install libpam-winbind from scratch? There are no binaries for ubuntu zesty?

Piviul

[Andreas Hasenack (ahasenack)]

The fix is stuck in the sponsoring queue, which is huge: http://reqorts.qa.ubuntu.com/reports/sponsoring/

I'll try again to ping someone

[Andreas Hasenack (ahasenack)]

Package was sponsored, now the sru team needs to take a look.

[Brian Murray (brian-murray)]

Hello Lyle, or anyone else affected,

Accepted samba into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/samba/2:4.5.8+dfsg-0ubuntu0.17.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andrew Reis (areis422)]

Confirmed working:

Distributor ID: Ubuntu
Description: Ubuntu 17.04
Release: 17.04
Codename: zesty

libnss-winbind:amd64 2:4.5.8+dfsg-0ubuntu0.17.04.5
libpam-winbind:amd64 2:4.5.8+dfsg-0ubuntu0.17.04.5
libwbclient0:amd64 2:4.5.8+dfsg-0ubuntu0.17.04.5
python-samba 2:4.5.8+dfsg-0ubuntu0.17.04.5
samba 2:4.5.8+dfsg-0ubuntu0.17.04.5
samba-common 2:4.5.8+dfsg-0ubuntu0.17.04.5
samba-common-bin 2:4.5.8+dfsg-0ubuntu0.17.04.5
samba-dsdb-modules 2:4.5.8+dfsg-0ubuntu0.17.04.5
samba-libs:amd64 2:4.5.8+dfsg-0ubuntu0.17.04.5
samba-vfs-modules 2:4.5.8+dfsg-0ubuntu0.17.04.5
winbind 2:4.5.8+dfsg-0ubuntu0.17.04.5

[Andreas Hasenack (ahasenack)]

@drew-reis, thanks for the verification. Can you please update the bug tags according to the instructions in comment #36?

[Launchpad Janitor (janitor)]

This bug was fixed in the package samba - 2:4.5.8+dfsg-0ubuntu0.17.04.5

---------------
samba (2:4.5.8+dfsg-0ubuntu0.17.04.5) zesty; urgency=medium

  * Remove the fix for LP #1584485 as it builds a broken pam_winbind
    module. There is a revised version of that patch attached to
    #1584485 but it has not been vetted yet, so for now it's best
    to revert (again) so that pam_winbind can be used.
    (LP: #1677329, LP: #1644428)
    - d/p/fix-1584485.patch: drop
    - d/rules: remove winbind static build option

 -- Andreas Hasenack <email address hidden> Thu, 13 Jul 2017 14:44:16 -0300

[Andrew Bartlett (abartlet)]

This was finally fixed properly upstream with this massive patch set https://bugzilla.samba.org/show_bug.cgi?id=14780 for Samba 4.16. A very good reason to upgrade to this release where possible.

By building a properly static (in terms of samba libs at least) pam_winbind.so, there should be no more unresolved symbols.