SSSD authentication fails after upgrade to 1.11.8-0ubuntu0.2

Bug #1640805 reported by Florin Baca on 2016-11-10
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
sssd
Fix Released
Unknown
sssd (Ubuntu)
High
Unassigned
Trusty
Undecided
Unassigned

Bug Description

[Impact]

 * sssd authentication fails if only allow rules are used and
   simple_allow_groups is set but group verification fails.

 * That is a regression to the status before the last SRU update.

 * The fix is a backport of the upstream fix for
   https://fedorahosted.org/sssd/ticket/2519

[Test Case]

 * Set up sssd (can be quite complex) with only allow rules and
   simple_allow_rules config being set.

 * Then authenticate a user/group combo where the group verification fails

 * This should succeed, but does no more since last update

[Regression Potential]

 * Since this is a backport of a single fix, but lacks the other changes
   that got upstream in-between there is the chance we missed in
   code-review that there is more context needed to be backported that
   could now cause other authentication issues.

[Other Info]

 * Setup is so complex we likely have to rely on the reporter for
   verification.

 * Reporter tested with various config options for his issue - it would be
   kind if he could do so as well for the proposed verification.

---

During the first half of September, the Ubuntu sssd package has been updated from 1.11.5-1ubuntu3 to 1.11.8-0ubuntu0.2. We use sssd for authentication on a few dev servers and all our Ubuntu workstations. After the first systems began upgrading we noticed people are no longer able to login. Using the ui you're simply redirected to the login screen. With ssh the connection is closed right away:

$ ssh username@nv-hostname04
username@nv-hostname04's password:
Connection closed by x.x.x.244

In the auth log we can see the following:

Nov 9 09:33:10 nv-hostname04 sshd[5397]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.250 user=username
Nov 9 09:33:10 nv-hostname04 sshd[5397]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.250 user=username
Nov 9 09:33:10 nv-hostname04 sshd[5397]: pam_sss(sshd:account): Access denied for user username: 4 (System error)
Nov 9 09:33:10 nv-hostname04 sshd[5397]: Failed password for username from x.x.x.250 port 54210 ssh2
Nov 9 09:33:10 nv-hostname04 sshd[5397]: fatal: Access denied for user username by PAM account configuration [preauth]

Once I have downgraded the packages to the previous version everything works fine again:

apt-get install -y --force-yes sssd=1.11.5-1ubuntu3 sssd-common=1.11.5-1ubuntu3 sssd-ad=1.11.5-1ubuntu3 sssd-ipa=1.11.5-1ubuntu3 sssd-krb5=1.11.5-1ubuntu3 sssd-ldap=1.11.5-1ubuntu3 sssd-proxy=1.11.5-1ubuntu3 python-sss=1.11.5-1ubuntu3 libsss-idmap0=1.11.5-1ubuntu3 sssd-ad-common=1.11.5-1ubuntu3 sssd-krb5-common=1.11.5-1ubuntu3 sssd-krb5-common=1.11.5-1ubuntu3 libipa-hbac0=1.11.5-1ubuntu3 sssd-ad-common=1.11.5-1ubuntu3 sssd-krb5-common=1.11.5-1ubuntu3 libsss-idmap0=1.11.5-1ubuntu3 sssd-krb5-common=1.11.5-1ubuntu3
echo 'sssd hold' | dpkg --set-selections

I started enabling sssd debug logs, starting from 3 up to 7. It seems the problem is directly related to the fact that sssd cannot resolve the name of a few groups. The users are part of different mailing lists which we don't want listed on our Linux pcs.

(Wed Nov 9 09:33:10 2016) [sssd[be[ads.domain.com]]] [simple_resolve_group_done] (0x0040): Refresh failed
(Wed Nov 9 09:33:10 2016) [sssd[be[ads.domain.com]]] [simple_check_get_groups_next] (0x0040): Could not resolve name of group with GID 2099314
(Wed Nov 9 09:33:10 2016) [sssd[be[ads.domain.com]]] [simple_access_check_done] (0x0040): Could not collect groups of user username

I also noticed this is directly related to the simple_allow_groups module that we use to allow login for specific groups. Here's what I have tried and confirmed it fixes the issue:

1. comment out the line "simple_allow_groups = ", restart sssd => authentication works
2. change the "ldap_group_search_base" to include all un-resolvable groups, restart sssd => authentication works.

For the sake of testing, I used the sssd/updates ppa to install version 1.12.5-1~trusty1 of the sssd. I can confirm in this version everything works as expected. So basically:

broken: 1.11.8-0ubuntu0.2
good: 1.11.5-1ubuntu3
good: 1.12.5-1~trusty1

I looked at the upstream merges Ubuntu has done for 1.11.8, there are around 5-6 changes but I cannot figure out which one introduced the error.

The direct issue from sssd which describes the exact same issue is found at: https://fedorahosted.org/sssd/ticket/2519 .

Thanks for your pre-analysis on this Florin.

It seems it is an upstream issue - just as you pointed out with the linked ticket.
And that matches exactly what you report (and makes sense that neither me nor you find one of the changes Ubuntu introduced to be related to the issue).

IMHO this happened this way:
good: 1.11.5-1ubuntu3
> upstream introduced issue, picked up by the bugfix release to 1.11.8 by us
broken: 1.11.8-0ubuntu0.2
> upstream fixed in 1.12.3 according to the ticket.
good: 1.12.5-1~trusty1

I checked on the fixes themself since a bugtracker eventually is only text, but it matches:
$ git tag --contains 79f12880 | head -n 1
sssd-1_12_3
82a958e6 is the same as above but to dev branch
$ git tag --contains 45a089a7 | head -n 1
sssd-1_12_5

So 1.12.5 that you tested from ppa should work and that is what you report.

I doubt we want to pull in all of 1.12.5 into trusty - the whole reasons to pull 1.11.8 (I assume) was that it was meant to be a bugfix only release.

So one has to evaluate if the patches would apply.
While checking that I realized that I accidentally created almost the full fix - so we could as well test it. But since I have no knowledge of sssd nor a setup I need you for that.

So I created a ppa at https://launchpad.net/~paelzer/+archive/ubuntu/trusty-sssd-bug-1640805 and would like to have:
- a verification from you on the ppa
- a comment from Timo as he did the update to 1.11.8

Changed in sssd (Ubuntu):
status: New → Triaged
importance: Undecided → High
Florin Baca (fbaca) wrote :

That was very fast Christian, thanks! I have just tested on a freshly installed Ubuntu and I can confirm the new ppa package fixed the issue:

[ansible | CLIENTS] ssh username@x.x.x.143
username@x.x.x.143's password:
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-101-generic x86_64)

 * Documentation: https://help.ubuntu.com/

Last login: Fri Nov 11 10:19:42 2016 from x.x.x.250
username@nv-hostname06:~$ dpkg -l |grep sssd |awk '{print $2,$3}'
sssd 1.11.8-0ubuntu0.3~ppa1
sssd-ad 1.11.8-0ubuntu0.3~ppa1
sssd-ad-common 1.11.8-0ubuntu0.3~ppa1
sssd-common 1.11.8-0ubuntu0.3~ppa1
sssd-ipa 1.11.8-0ubuntu0.3~ppa1
sssd-krb5 1.11.8-0ubuntu0.3~ppa1
sssd-krb5-common 1.11.8-0ubuntu0.3~ppa1
sssd-ldap 1.11.8-0ubuntu0.3~ppa1
sssd-proxy 1.11.8-0ubuntu0.3~ppa1

On Fri, Nov 11, 2016 at 10:36 AM, Florin Baca <email address hidden>
wrote:

> That was very fast Christian, thanks! I have just tested on a freshly
> installed Ubuntu and I can confirm the new ppa package fixed the issue
>

Great - thanks for the fast verify as well.
FYI I'm waiting for a comment of Timo before I push for an SRU on this
then.

Timo Aaltonen (tjaalton) wrote :

thanks, looks good.. go ahead with the sru

Changed in sssd:
status: Unknown → Fix Released

Fixed in Ubuntu(dev) since Xenial, nominating for Trusty SRU

description: updated
Changed in sssd (Ubuntu):
status: Triaged → Fix Released

Note for the SRU Review - something in the packaging here is odd (or it is me, the monday morning or something broken in the (my?) tooling)
Please take an extra eye on the changes.
I created a debdiff and tried all I could to make it a clean one, but I had hidden files/dirs sneak into them more then once.
Here an example on what I see and atm I'm not sure if is is me or the package:
See this for more http://paste.ubuntu.com/23474596/.

I discussed on #ubuntu-devel, but we didn't find a solution to it - OTOH it doesn't seem critical either so I'd leave it up to your expertise.

Attaching the debdiff as reference if there is anything to fix up.

Subscribing sponsors as I can't upload back to Trusty and given the mentioned weirdness on the packaging it is worth an extra eye to look at it.

Sponsors seem to be very busy these days, I pinged Timo to hope we get this into Trusty Proposed sometime soon.

Timo Aaltonen (tjaalton) wrote :

uploaded to the queue

Changed in sssd (Ubuntu Trusty):
status: New → In Progress

Hello Florin, or anyone else affected,

Accepted sssd into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sssd/1.11.8-0ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in sssd (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Florin Baca (fbaca) wrote :

Hi Robie,

Thanks for the info, I have upgraded the proposed package on a broken system and I hereby confirm it solves the issue as expected.

Current versions:

sssd 1.11.8-0ubuntu0.3
sssd-ad 1.11.8-0ubuntu0.3
sssd-ad-common 1.11.8-0ubuntu0.3
sssd-common 1.11.8-0ubuntu0.3
sssd-ipa 1.11.8-0ubuntu0.3
sssd-krb5 1.11.8-0ubuntu0.3
sssd-krb5-common 1.11.8-0ubuntu0.3
sssd-ldap 1.11.8-0ubuntu0.3
sssd-proxy 1.11.8-0ubuntu0.3

Per former comment, setting verification done

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sssd - 1.11.8-0ubuntu0.3

---------------
sssd (1.11.8-0ubuntu0.3) trusty; urgency=medium

  * d/p/fix-upstream-2519.diff SSSD should not fail authentication when only
    allow rules are used (LP: #1640805)

 -- Christian Ehrhardt <email address hidden> Fri, 11 Nov 2016 09:23:35 +0100

Changed in sssd (Ubuntu Trusty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for sssd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.