Too loose Depends constraints in sssd-common causes critical crash

[Max]

Ubuntu Security team recently released ".11" of SSSD.

`sssd` correctly specifies a hard dependency on "sssd-common (= 2.2.3-3ubuntu0.11)"

`sssd-common` only specifies "libsss-certmap0 (>= 2.2.3)", which means `libsss-certmap0` could stay on 2.2.3-3ubuntu0*.10*

This causes the following critical error:

> sssd_pam[10488]: /usr/libexec/sssd/sssd_pam: /lib/x86_64-linux-gnu/libsss_certmap.so.0: version `SSS_CERTMAP_0.2' not found (required by /usr/libexec/sssd/sssd_pam)

The user-experience is a immediate crash to black screen, lost work due to session termination, and inability to log in again until `libsss-certmap0` has been updated.

The issue can be reproduced on an Ubuntu 20.04 machine connected to AD running the previous .10 release of SSSD, and upgraded using "apt install sssd".

Note that a blanket "apt upgrade" is not always possible, as it will restart services such as Docker and Databases, which may have critical impact on production. It should be the responsibility of "Depends" to specify exactly what the software needs to function correctly. In this case it was a breaking change between 2.2.3-3ubuntu0.10 and 2.2.3-3ubuntu0.11 in `libsss-certmap0`, so the "Depends" of sssd-common should be updated to reflect that minimum requirement.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sergio Durigan Junior (sergiodj)]

Thank you for taking the time to report this bug.

I can confirm it locally. Inside a Focal LXD container, install sssd 2.2.3-3ubuntu0.10 (the best way is to use pull-lp-debs here, since the version is not present in the archive anymore), then run "apt install sssd" to upgrade the package to 2.2.3-3ubuntu0.10, and try to run sssd_pam by hand:

# /usr/libexec/sssd/sssd_pam
/usr/libexec/sssd/sssd_pam: /lib/x86_64-linux-gnu/libsss_certmap.so.0: version `SSS_CERTMAP_0.2' not found (required by /usr/libexec/sssd/sssd_pam)

[Bas van den Heuvel (2-bas)]

I fixed it (temporary) by downgrading and holding the following ssd packages (downloaded from http://nl.archive.ubuntu.com/ubuntu/pool/main/s/sssd/):

-rw-r--r-- 1 root root 10656 apr 21 2020 libipa-hbac0_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 32908 apr 21 2020 libsss-certmap0_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 14916 apr 21 2020 libsss-idmap0_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 46012 apr 21 2020 python3-sss_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 4220 apr 21 2020 sssd_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 115524 apr 21 2020 sssd-ad_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 67612 apr 21 2020 sssd-ad-common_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 1028864 apr 21 2020 sssd-common_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 210744 apr 21 2020 sssd-ipa_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 13596 apr 21 2020 sssd-krb5_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 75768 apr 21 2020 sssd-krb5-common_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 31092 apr 21 2020 sssd-ldap_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 34540 apr 21 2020 sssd-proxy_2.2.3-3_amd64.deb
-rw-r--r-- 1 root root 94584 apr 21 2020 sssd-tools_2.2.3-3_amd64.deb

then running

dpkg -i *.deb

and the:

ls -1 *.deb | awk -F'_' '{ print $1" hold" }' | dpkg --set-selections

[None Given (ptech)]

This has affected several users with FreeIPA accounts on their local Ubuntu machines. Without local account access they have had to reboot with a live CD and swap the libsss-certmap0 file manually to be able to boot the system at all. So far the two cases where it's happened, one was affected by an unattended upgrade failure, where it updated sssd-common but then the service failed somehow which left libsss-certmap in an incompatible state. The second was a user manually updated a different package which only updated sssd and sssd-common but not libsss-certmap0. Immediately after doing that they were unable to run sudo commands to update the libsss-certmap0 package. We are triaging by having all users manually update both packages to the specific version at the same time.

[Sergio Durigan Junior (sergiodj)]

Thank you for the extra information. We are working on this bug and should have an update soon.

[Marc Deslauriers (mdeslaur)]

Thanks for reporting this issue.

Security updates aren't intended to be installed manually, and that is not a scenario that is tested when we publish them. Doing a full "apt upgrade" is the recommended way to install security updates.

That being said, not being able to login after manually installing packages definitely isn't an ideal scenario, and we will certainly look into tightening the dependencies to make sure this no longer happens.

[Andreas Hasenack (ahasenack)]

@ptech I did a test with unattended-upgrades in focal, and it did the right thing. It pulled all the necessary sssd packages, including libsss-certmap0. You mentioned unattended-upgrades had a failure, was it unrelated to sssd?

[Marc Deslauriers (mdeslaur)]

I have uploaded a package with tightened dependencies to build in the security team PPA here:

https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+packages

Once it is done building, if someone can confirm that fixes the regression, that would be helpful. I'll also test it tomorrow morning and if the problem seems fixed, I will release it.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sssd - 2.2.3-3ubuntu0.12

---------------
sssd (2.2.3-3ubuntu0.12) focal-security; urgency=medium

  * Fix crash with mismatched packages (LP: #2023598)
    - debian/control: add a versioned dependency on libsss-certmap0 to the
      sssd-common package.

 -- Marc Deslauriers <email address hidden> Thu, 15 Jun 2023 18:16:57 -0400

[None Given (ptech)]

@ahasenack The system with the unattended-upgrade failure was only logged in the apt history.log. It was recorded as "Sub-process /usr/bin/dpkg returned an error code (1)". On another of our systems that didn't have issues, unattended-upgrades updated sssd-common without errors and on a subsequent unattended-upgrade run, about a minute later, it updated libsss-certmap0 successfully. This leads me to believe the one that ran into errors during sssd-common update likely had some other issue with the apt install process unrelated to the out of order update dependency issue but that left libsss-certmap0 in an incompatible state which prevented the system from booting.

[Andreas Hasenack (ahasenack)]

The terminal log would have more information about that dpkg error.