Activity log for bug #1684295

Date Who What changed Old value New value Message
2017-04-19 21:20:18 Pamela Skutnik bug added bug
2017-04-19 21:20:18 Pamela Skutnik attachment added sssd.conf file https://bugs.launchpad.net/bugs/1684295/+attachment/4865180/+files/sssd.conf.txt
2017-07-26 21:37:41 Andreas Hasenack sssd (Ubuntu): status New Incomplete
2017-10-04 04:17:33 Launchpad Janitor sssd (Ubuntu): status Incomplete Expired
2017-10-04 20:19:32 Pamela Skutnik attachment added sssd.conf https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+attachment/4962331/+files/sssd.conf
2017-10-04 21:08:20 Andreas Hasenack sssd (Ubuntu): status Expired New
2017-10-25 18:24:11 Pamela Skutnik attachment added sssd_tpad.log https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+attachment/4995131/+files/sssd_tpad.log
2017-10-25 18:24:41 Pamela Skutnik attachment added sssd.log https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+attachment/4995132/+files/sssd.log
2017-11-02 19:30:26 Pamela Skutnik attachment added _usr_lib_x86_64-linux-gnu_sssd_sssd_be.0.crash https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+attachment/5002554/+files/_usr_lib_x86_64-linux-gnu_sssd_sssd_be.0.crash
2017-11-03 12:43:59 Andreas Hasenack sssd (Ubuntu): status New Triaged
2017-11-03 12:44:03 Andreas Hasenack sssd (Ubuntu): importance Undecided High
2017-11-03 12:44:20 Andreas Hasenack tags bitesize server-next
2017-11-03 12:51:52 Andreas Hasenack bug added subscriber Ubuntu Server Team
2017-11-06 12:40:31 Steve Buehrle bug added subscriber Steve Buehrle
2017-11-06 17:22:50 Andreas Hasenack sssd (Ubuntu): assignee Andreas Hasenack (ahasenack)
2017-11-06 17:22:54 Andreas Hasenack sssd (Ubuntu): status Triaged In Progress
2017-11-06 17:39:51 Andreas Hasenack description This is Ubuntu 16.04.2 LTS sssd is configured to connect to two domains, our TPAD directory and Active Directory. sssd starts up at boot time. As soon as I ssh login (with any id, AD, TPAD or local), sssd fails with the error message in the title. After that, we can only login with local ids, not TPAD or AD ids. **************** Here is the output from systemctl status sssd after the failure: root@dcmilphlum128:~# systemctl status sssd â sssd.service - System Security Services Daemon Loaded: loaded (/lib/systemd/system/sssd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2017-04-19 16:40:08 EDT; 7min ago Process: 119143 ExecStart=/usr/sbin/sssd -D -f (code=exited, status=0/SUCCESS) Main PID: 119145 (code=exited, status=1/FAILURE) Apr 19 16:39:47 dcmilphlum128.edc.nam.gm.com sssd[be[119187]: Starting up Apr 19 16:39:51 dcmilphlum128.edc.nam.gm.com sssd[be[119191]: Starting up Apr 19 16:39:57 dcmilphlum128.edc.nam.gm.com sssd[be[119206]: Starting up Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[119145]: Exiting the SSSD. Could not restart critical service [tpad]. Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[119149]: Shutting down Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[119148]: Shutting down Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[be[119146]: Shutting down Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com systemd[1]: sssd.service: Unit entered failed state. Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com systemd[1]: sssd.service: Failed with result 'exit-code'. ****************** Also, in kern.log I have four of these (I have retries set to 3): Apr 19 16:39:59 dcmilphlum128 kernel: [ 6205.937807] sssd_be[12218]: segfault at 0 ip 00007fecb32b6b94 sp 00007ffce49a2230 error 4 in libsss_util.so[7fecb32a2000+6c000] Apr 19 16:40:02 dcmilphlum128 kernel: [ 6206.980725] sssd_be[12253]: segfault at 0 ip 00007f302de29b94 sp 00007fffca943cc0 error 4 in libsss_util.so[7f302de15000+6c000] Apr 19 16:40:05 dcmilphlum128 kernel: [ 6211.036205] sssd_be[12256]: segfault at 0 ip 00007fd196169b94 sp 00007ffd624249f0 error 4 in libsss_util.so[7fd196155000+6c000] Apr 19 16:40:07 dcmilphlum128 kernel: [ 6225.081902] sssd_be[12257]: segfault at 0 ip 00007fd1f669bb94 sp 00007ffdd8e5bf80 error 4 in libsss_util.so[7fd1f6687000+6c000] ******************* My sssd package are at 1.13.4: sssd 1.13.4-1ubuntu1.1 amd64 sssd-ad 1.13.4-1ubuntu1.1 amd64 sssd-ad-common 1.13.4-1ubuntu1.1 amd64 sssd-common 1.13.4-1ubuntu1.1 amd64 sssd-ipa 1.13.4-1ubuntu1.1 amd64 sssd-krb5 1.13.4-1ubuntu1.1 amd64 sssd-krb5-common 1.13.4-1ubuntu1.1 amd64 sssd-ldap 1.13.4-1ubuntu1.1 amd64 sssd-proxy 1.13.4-1ubuntu1.1 amd64 sssd-tools 1.13.4-1ubuntu1.1 amd64 *************** I upgraded all the sssd packages to 1.13.4-1ubuntu1.4 and had the same problem. I downgraded them to 1.12.5-2 and was NOT able to reproduce the problem. I attached my sssd.conf file. [Impact] * An explanation of the effects of the bug on users and * justification for backporting the fix to the stable release. * In addition, it is helpful, but not required, to include an explanation of how the upload fixes this bug. [Test Case] * detailed instructions how to reproduce the bug * these should allow someone who is not familiar with the affected package to reproduce the bug and verify that the updated package fixes the problem. [Regression Potential] * discussion of how regressions are most likely to manifest as a result of this change. * It is assumed that any SRU candidate patch is well-tested before upload and has a low overall risk of regression, but it's important to make the effort to think about what ''could'' happen in the event of a regression. * This both shows the SRU team that the risks have been considered, and provides guidance to testers in regression-testing the SRU. [Other Info] * Anything else you think is useful to include * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board * and address these questions in advance
2017-11-06 17:44:55 Andreas Hasenack description [Impact] * An explanation of the effects of the bug on users and * justification for backporting the fix to the stable release. * In addition, it is helpful, but not required, to include an explanation of how the upload fixes this bug. [Test Case] * detailed instructions how to reproduce the bug * these should allow someone who is not familiar with the affected package to reproduce the bug and verify that the updated package fixes the problem. [Regression Potential] * discussion of how regressions are most likely to manifest as a result of this change. * It is assumed that any SRU candidate patch is well-tested before upload and has a low overall risk of regression, but it's important to make the effort to think about what ''could'' happen in the event of a regression. * This both shows the SRU team that the risks have been considered, and provides guidance to testers in regression-testing the SRU. [Other Info] * Anything else you think is useful to include * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board * and address these questions in advance [Impact] In this particular configuration, when ldap_rfc2307_fallback_to_local_users is set to true in /etc/sss/sssd.conf and a local user is a member of an ldap group and does not exist in the directory (other scenarios are possible), the sssd_be process segfaults and logins might be prevented. The original scenario is a bit more complex and involves setting up an Active Directory server, but with the help from the bug reporter (thanks @pam-s!) we managed to narrow it down to this simple test case. [Test Case]  * detailed instructions how to reproduce the bug  * these should allow someone who is not familiar with the affected    package to reproduce the bug and verify that the updated package fixes    the problem. [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance
2017-11-06 17:47:18 Andreas Hasenack description [Impact] In this particular configuration, when ldap_rfc2307_fallback_to_local_users is set to true in /etc/sss/sssd.conf and a local user is a member of an ldap group and does not exist in the directory (other scenarios are possible), the sssd_be process segfaults and logins might be prevented. The original scenario is a bit more complex and involves setting up an Active Directory server, but with the help from the bug reporter (thanks @pam-s!) we managed to narrow it down to this simple test case. [Test Case]  * detailed instructions how to reproduce the bug  * these should allow someone who is not familiar with the affected    package to reproduce the bug and verify that the updated package fixes    the problem. [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Impact] In this particular configuration, when ldap_rfc2307_fallback_to_local_users is set to true in /etc/sss/sssd.conf and a local user is a member of an ldap group and does not exist in the directory (other scenarios are possible), the sssd_be process segfaults and logins might be prevented. The original scenario is a bit more complex and involves setting up an Active Directory server, but with the help from the bug reporter (thanks @pam-s!) we managed to narrow it down to this simple test case. [Test Case] # Install the packages. When prompted, choose any password for the ldap admin $ sudo apt update; sudo apt install sssd slapd # create the sssd config $ sudo tee /etc/sssd/sssd.conf <<EOF [sssd] config_file_version = 2 services = nss, pam domains = LDAP [domain/LDAP] id_provider = ldap ldap_uri = ldap://localhost ldap_search_base = dc=example,dc=com ldap_rfc2307_fallback_to_local_users = True EOF $ sudo chmod 0600 /etc/sssd/sssd.conf # reconfigure slapd for domain example.com, organization example. For the rest, accept defaults $ sudo dpkg-reconfigure slapd # add the base ldif. When prompted, use the password you chose when reconfiguring slapd earlier $ ldapadd -x -D cn=admin,dc=example,dc=com -W <<EOF dn: ou=People,dc=example,dc=com ou: People objectClass: organizationalUnit dn: ou=Group,dc=example,dc=com ou: Group objectClass: organizationalUnit dn: cn=ldapusers,ou=Group,dc=example,dc=com cn: ldapusers objectClass: posixGroup gidNumber: 10000 memberUid: localuser EOF adding new entry "ou=People,dc=example,dc=com" adding new entry "ou=Group,dc=example,dc=com" adding new entry "cn=ldapusers,ou=Group,dc=example,dc=com" # create a localuser with that name $ sudo useradd -M localuser # restart sssd $ sudo service sssd restart # take note of the sssd_be process id: $ pidof sssd_be 15474 # in one terminal, keep tailing /var/log/syslog $ sudo tail -f /var/log/syslog # in another terminal, run this id command. It will possibly hang for a bit, and won't show the "ldapusers" group membership $ id localuser (hangs a bit) uid=1001(localuser) gid=1001(localuser) groups=1001(localuser) # /var/log/syslog will emit messages like these, about a crash and sssd_be restarting (if you don't have apport installed, you will just see the "starting up" bit about sssd_be): Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 systemd[1]: Starting Apport crash forwarding receiver... Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 sssd[be[LDAP]]: Starting up Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 systemd[1]: Started Apport crash forwarding receiver. # verify that the sssd_be process id changed, confirming that it crashed and was restarted: $ pidof sssd_be 15485 # install the fixed packages from proposed $ apt install/dist-upgrade .... # repeat the id command. Now it finishes quickly, shows the "ldapusers" group membership, and there won't be any sign of an sssd_be restart in /var/log/syslog: $ id localuser uid=1001(localuser) gid=1001(localuser) groups=1001(localuser),10000(ldapusers) [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance
2017-11-06 17:57:01 Andreas Hasenack description [Impact] In this particular configuration, when ldap_rfc2307_fallback_to_local_users is set to true in /etc/sss/sssd.conf and a local user is a member of an ldap group and does not exist in the directory (other scenarios are possible), the sssd_be process segfaults and logins might be prevented. The original scenario is a bit more complex and involves setting up an Active Directory server, but with the help from the bug reporter (thanks @pam-s!) we managed to narrow it down to this simple test case. [Test Case] # Install the packages. When prompted, choose any password for the ldap admin $ sudo apt update; sudo apt install sssd slapd # create the sssd config $ sudo tee /etc/sssd/sssd.conf <<EOF [sssd] config_file_version = 2 services = nss, pam domains = LDAP [domain/LDAP] id_provider = ldap ldap_uri = ldap://localhost ldap_search_base = dc=example,dc=com ldap_rfc2307_fallback_to_local_users = True EOF $ sudo chmod 0600 /etc/sssd/sssd.conf # reconfigure slapd for domain example.com, organization example. For the rest, accept defaults $ sudo dpkg-reconfigure slapd # add the base ldif. When prompted, use the password you chose when reconfiguring slapd earlier $ ldapadd -x -D cn=admin,dc=example,dc=com -W <<EOF dn: ou=People,dc=example,dc=com ou: People objectClass: organizationalUnit dn: ou=Group,dc=example,dc=com ou: Group objectClass: organizationalUnit dn: cn=ldapusers,ou=Group,dc=example,dc=com cn: ldapusers objectClass: posixGroup gidNumber: 10000 memberUid: localuser EOF adding new entry "ou=People,dc=example,dc=com" adding new entry "ou=Group,dc=example,dc=com" adding new entry "cn=ldapusers,ou=Group,dc=example,dc=com" # create a localuser with that name $ sudo useradd -M localuser # restart sssd $ sudo service sssd restart # take note of the sssd_be process id: $ pidof sssd_be 15474 # in one terminal, keep tailing /var/log/syslog $ sudo tail -f /var/log/syslog # in another terminal, run this id command. It will possibly hang for a bit, and won't show the "ldapusers" group membership $ id localuser (hangs a bit) uid=1001(localuser) gid=1001(localuser) groups=1001(localuser) # /var/log/syslog will emit messages like these, about a crash and sssd_be restarting (if you don't have apport installed, you will just see the "starting up" bit about sssd_be): Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 systemd[1]: Starting Apport crash forwarding receiver... Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 sssd[be[LDAP]]: Starting up Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 systemd[1]: Started Apport crash forwarding receiver. # verify that the sssd_be process id changed, confirming that it crashed and was restarted: $ pidof sssd_be 15485 # install the fixed packages from proposed $ apt install/dist-upgrade .... # repeat the id command. Now it finishes quickly, shows the "ldapusers" group membership, and there won't be any sign of an sssd_be restart in /var/log/syslog: $ id localuser uid=1001(localuser) gid=1001(localuser) groups=1001(localuser),10000(ldapusers) [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Impact] In this particular configuration, when ldap_rfc2307_fallback_to_local_users is set to true in /etc/sss/sssd.conf and a local user is a member of an ldap group and does not exist in the directory (other scenarios are possible), the sssd_be process segfaults and logins might be prevented. The original scenario is a bit more complex and involves setting up an Active Directory server, but with the help from the bug reporter (thanks @pam-s!) we managed to narrow it down to this simple test case. [Test Case] # Install the packages. When prompted, choose any password for the ldap admin $ sudo apt update; sudo apt install sssd slapd # create the sssd config $ sudo tee /etc/sssd/sssd.conf <<EOF [sssd] config_file_version = 2 services = nss, pam domains = LDAP [domain/LDAP] id_provider = ldap ldap_uri = ldap://localhost ldap_search_base = dc=example,dc=com ldap_rfc2307_fallback_to_local_users = True EOF $ sudo chmod 0600 /etc/sssd/sssd.conf # reconfigure slapd for domain example.com, organization example. For the rest, accept defaults $ sudo dpkg-reconfigure slapd # add the base ldif. When prompted, use the password you chose when reconfiguring slapd earlier $ ldapadd -x -D cn=admin,dc=example,dc=com -W <<EOF dn: ou=People,dc=example,dc=com ou: People objectClass: organizationalUnit dn: ou=Group,dc=example,dc=com ou: Group objectClass: organizationalUnit dn: cn=ldapusers,ou=Group,dc=example,dc=com cn: ldapusers objectClass: posixGroup gidNumber: 10000 memberUid: localuser EOF adding new entry "ou=People,dc=example,dc=com" adding new entry "ou=Group,dc=example,dc=com" adding new entry "cn=ldapusers,ou=Group,dc=example,dc=com" # create a localuser with that name $ sudo useradd -M localuser # restart sssd $ sudo service sssd restart # take note of the sssd_be process id: $ pidof sssd_be 15474 # in one terminal, keep tailing /var/log/syslog $ sudo tail -f /var/log/syslog # in another terminal, run this id command. It will possibly hang for a bit, and won't show the "ldapusers" group membership $ id localuser (hangs a bit) uid=1001(localuser) gid=1001(localuser) groups=1001(localuser) # /var/log/syslog will emit messages like these, about a crash and sssd_be restarting (if you don't have apport installed, you will just see the "starting up" bit about sssd_be): Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 systemd[1]: Starting Apport crash forwarding receiver... Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 sssd[be[LDAP]]: Starting up Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 systemd[1]: Started Apport crash forwarding receiver. # verify that the sssd_be process id changed, confirming that it crashed and was restarted: $ pidof sssd_be 15485 # install the fixed packages from proposed $ apt install/dist-upgrade .... # repeat the id command. Now it finishes quickly, shows the "ldapusers" group membership, and there won't be any sign of an sssd_be restart in /var/log/syslog: $ id localuser uid=1001(localuser) gid=1001(localuser) groups=1001(localuser),10000(ldapusers) [Regression Potential] The patch is very specific, but given in how many different ways sssd can be configured, it would really help if users actually tested the package from proposed in their deployments. Specially considering it's a login service. That being said, the patch is applied in the 1.13, 1,14 and current 1.15 series upstream and is more than a year old by now. It could rely on other changes that I missed, though, but at least one I chose to ignore (see [other info]). [Other Info] The exact upstream patch wasn't applied (https://pagure.io/SSSD/sssd/c/5a0fb268e836e600d864ded7de5d935946ae6c61), because it relied on dropping an unused parameter from sdap_fallback_local_user(), namely the *opts struct pointer (https://pagure.io/SSSD/sssd/c/77f960ab32c2d2245fed55671f24af287ea0ba50). It is indeed not used, but I rather not drop it for an SRU because I don't know if some library could be using it, and also because a new upstream version for this series (1.13.5) wasn't released yet with this change.
2017-11-07 18:36:38 Launchpad Janitor merge proposal linked https://code.launchpad.net/~ahasenack/ubuntu/+source/sssd/+git/sssd/+merge/333333
2017-11-08 15:50:14 Christian Ehrhardt  nominated for series Ubuntu Xenial
2017-11-08 15:50:14 Christian Ehrhardt  bug task added sssd (Ubuntu Xenial)
2017-11-08 15:50:20 Christian Ehrhardt  sssd (Ubuntu Xenial): status New In Progress
2017-11-08 15:50:28 Christian Ehrhardt  sssd (Ubuntu Xenial): assignee Andreas Hasenack (ahasenack)
2017-11-08 15:50:31 Christian Ehrhardt  sssd (Ubuntu): status In Progress Fix Released
2017-11-08 15:50:34 Christian Ehrhardt  sssd (Ubuntu Xenial): importance Undecided High
2017-11-09 16:21:05 Łukasz Zemczak sssd (Ubuntu Xenial): status In Progress Fix Committed
2017-11-09 16:21:07 Łukasz Zemczak bug added subscriber Ubuntu Stable Release Updates Team
2017-11-09 16:21:10 Łukasz Zemczak bug added subscriber SRU Verification
2017-11-09 16:21:14 Łukasz Zemczak tags bitesize server-next bitesize server-next verification-needed verification-needed-xenial
2017-11-13 13:11:27 Andreas Hasenack tags bitesize server-next verification-needed verification-needed-xenial bitesize server-next verification-done-xenial verification-needed
2017-11-20 17:18:33 Launchpad Janitor sssd (Ubuntu Xenial): status Fix Committed Fix Released
2017-11-20 17:18:42 Brian Murray removed subscriber Ubuntu Stable Release Updates Team