Upgrade from Samba 4.13 to 4.15 results in "dlopen(pam_winbind.so): /lib/security/pam_winbind.so: cannot open shared object file: No such file or directory"

[Andrew Martin]

On Ubuntu 20.04 I was previously running Samba 2:4.13.17~dfsg-0ubuntu1.20.04.5 but today the upgrade to 2:4.15.13+dfsg-0ubuntu0.20.04.1 for USN-5936-1 appears to have introduced the following regression when using winbind and libpam-winbind to perform authentication:

PAM unable to dlopen(pam_winbind.so): /lib/security/pam_winbind.so: cannot open shared object file: No such file or directory
PAM adding faulty module: pam_winbind.so

The PAM configuration is as follows:

/etc/nsswitch.conf:
passwd: files winbind
shadow: files winbind
group: files winbind

/etc/pam.d files configured with "pam-auth-update --force" and selecting winbind as one of the enabled options.

Creating a symlink at /lib/security/pam_winbind.so that points at /lib/x86_64-linux-gnu/security/pam_winbind.so does not resolve the problem.

This appears to be the same issue described in #1644428 and #1584485. As reported in those tickets, the upstream fix was introduced in 4.16 so it seems that 4.15 is unusable with this patch applied.

[Marc Deslauriers (mdeslaur)]

Unfortunately, pam_winbind doesn't gracefully handle version upgrades properly. All processes using pam most likely need to be restarted to load the new libraries. Was that server rebooted after being updated to the new version?

[Andreas Hasenack (ahasenack)]

The path looks odd, shouldn't that be /lib/x86_64-linux-gnu/security/pam_winbind.so ? Or more generaly, /lib/$arch/security/pam_winbind.so

[Marc Deslauriers (mdeslaur)]

What's the output of grep pam_winbind /etc/pam.d/* ?

[Marc Deslauriers (mdeslaur)]

pam appears to try /lib/$arch/security/pam_winbind.so first, and if that fails, because of missing symbols in the update, it then falls back to trying /lib/security/pam_winbind.so and then it fails again, but it only logs the second attempt. The wrong path is a red herring.

[Steve Langasek (vorlon)]

On Fri, Mar 10, 2023 at 04:25:56PM -0000, Andreas Hasenack wrote:
> The path looks odd, shouldn't that be /lib/x86_64-linux-
> gnu/security/pam_winbind.so ? Or more generaly,
> /lib/$arch/security/pam_winbind.so

The multiarch implementation in pam is a bit annoying, it tries both
/lib/$arch/security/$module and /lib/security/$module (for
backwards-compatibility) and only reports the error message based on the
last path it tries so always shows the non-multiarch path in the error
message.

So /lib/x86_64-linux-gnu/security/pam_winbind.so may be present on the
filesystem but not loadable due to ELF symbol resolution issues.

[Andrew Martin (asmartin)]

> Unfortunately, pam_winbind doesn't gracefully handle version upgrades properly. All processes using pam most likely need to be restarted to load the new libraries. Was that server rebooted after being updated to the new version?

You were correct - after a reboot the server is now functioning correctly.

[Marc Deslauriers (mdeslaur)]

Thanks for updating the bug to let us know.

Unfortunately, there's nothing much we can do on our end to prevent this from happening again next time we are required to do a version upgrade. Thankfully, it looks like this is fixed upstream in 4.16, so hopefully this will be the last version to expose this unfortunate behaviour.

I am closing this bug report.