Activity log for bug #1664566

Date Who What changed Old value New value Message
2017-02-14 13:34:58 Michael Smith bug added bug
2017-02-14 13:41:43 Michael Smith description Hi, I'm in an environment with several Active Directory sites, each with a domain controller. When remote sites' DCs are unreachable because of a VPN outage, password authentication is slow or fails. tcpdump shows the system is trying to talk to the other sites' domain controllers, and timing out. sssd-common installs the locator plugin at /usr/lib/x86_64-linux-gnu/krb5/plugins/krb5/sssd_krb5_locator_plugin.so. But I can see in strace that Kerberos apps are looking for plugins in /usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5 instead (libkrb5 vs krb5). open("/usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory) As a result, Kerberos doesn't respect SSSD's Active Directory site selection. As a workaround, if I copy /usr/lib/x86_64-linux-gnu/krb5/plugins/krb5 to /usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5, site selection works as expected. Mailing list ref: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org/thread/UUMFE5T376D3NJLNHQSJZAJCPM35KRED/ Hi, I'm on Ubuntu 16.04 LTS, sssd-common 1.13.4-1ubuntu1.1, libkrb5-3 1.13.2+dfsg-5. I'm in an environment with several Active Directory sites, each with a domain controller. When remote sites' DCs are unreachable because of a VPN outage, password authentication is slow or fails. tcpdump shows the system is trying to talk to the other sites' domain controllers, and timing out. sssd-common installs the locator plugin at /usr/lib/x86_64-linux-gnu/krb5/plugins/krb5/sssd_krb5_locator_plugin.so. But I can see in strace that Kerberos apps are looking for plugins in /usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5 instead (libkrb5 vs krb5). open("/usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory) As a result, Kerberos doesn't respect SSSD's Active Directory site selection. As a workaround, if I copy /usr/lib/x86_64-linux-gnu/krb5/plugins/krb5 to /usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5, site selection works as expected. Mailing list ref: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org/thread/UUMFE5T376D3NJLNHQSJZAJCPM35KRED/
2017-02-14 13:45:42 Michael Smith tags apport-collected uec-images xenial
2017-02-14 13:45:43 Michael Smith description Hi, I'm on Ubuntu 16.04 LTS, sssd-common 1.13.4-1ubuntu1.1, libkrb5-3 1.13.2+dfsg-5. I'm in an environment with several Active Directory sites, each with a domain controller. When remote sites' DCs are unreachable because of a VPN outage, password authentication is slow or fails. tcpdump shows the system is trying to talk to the other sites' domain controllers, and timing out. sssd-common installs the locator plugin at /usr/lib/x86_64-linux-gnu/krb5/plugins/krb5/sssd_krb5_locator_plugin.so. But I can see in strace that Kerberos apps are looking for plugins in /usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5 instead (libkrb5 vs krb5). open("/usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory) As a result, Kerberos doesn't respect SSSD's Active Directory site selection. As a workaround, if I copy /usr/lib/x86_64-linux-gnu/krb5/plugins/krb5 to /usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5, site selection works as expected. Mailing list ref: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org/thread/UUMFE5T376D3NJLNHQSJZAJCPM35KRED/ Hi, I'm on Ubuntu 16.04 LTS, sssd-common 1.13.4-1ubuntu1.1, libkrb5-3 1.13.2+dfsg-5. I'm in an environment with several Active Directory sites, each with a domain controller. When remote sites' DCs are unreachable because of a VPN outage, password authentication is slow or fails. tcpdump shows the system is trying to talk to the other sites' domain controllers, and timing out. sssd-common installs the locator plugin at /usr/lib/x86_64-linux-gnu/krb5/plugins/krb5/sssd_krb5_locator_plugin.so. But I can see in strace that Kerberos apps are looking for plugins in /usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5 instead (libkrb5 vs krb5). open("/usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory) As a result, Kerberos doesn't respect SSSD's Active Directory site selection. As a workaround, if I copy /usr/lib/x86_64-linux-gnu/krb5/plugins/krb5 to /usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5, site selection works as expected. Mailing list ref: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org/thread/UUMFE5T376D3NJLNHQSJZAJCPM35KRED/ --- ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 DistroRelease: Ubuntu 16.04 JournalErrors: Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. Users in the 'systemd-journal' group can see all messages. Pass -q to turn off this notice. No journal files were opened due to insufficient permissions. Package: sssd 1.13.4-1ubuntu1.1 PackageArchitecture: amd64 ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcVersionSignature: Ubuntu 4.4.0-47.68-generic 4.4.24 Tags: xenial uec-images Uname: Linux 4.4.0-47-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True
2017-02-14 13:45:44 Michael Smith attachment added Dependencies.txt https://bugs.launchpad.net/bugs/1664566/+attachment/4818794/+files/Dependencies.txt
2017-02-17 09:47:40 Timo Aaltonen sssd (Ubuntu): status New Fix Committed
2017-04-06 12:07:41 Launchpad Janitor sssd (Ubuntu): status Fix Committed Fix Released
2017-07-06 15:06:08 Andreas Hasenack nominated for series Ubuntu Xenial
2017-07-06 15:21:05 Nish Aravamudan bug task added sssd (Ubuntu Xenial)
2017-07-06 15:21:35 Andreas Hasenack tags apport-collected uec-images xenial apport-collected server-next uec-images xenial
2017-07-12 21:22:43 Andreas Hasenack sssd (Ubuntu Xenial): status New Confirmed
2017-07-12 21:22:56 Andreas Hasenack sssd (Ubuntu Xenial): importance Undecided Medium
2017-07-12 21:23:31 Andreas Hasenack sssd (Ubuntu Xenial): assignee Andreas Hasenack (ahasenack)
2017-07-12 21:23:47 Andreas Hasenack bug added subscriber Andreas Hasenack
2017-07-21 17:10:23 Andreas Hasenack sssd (Ubuntu Xenial): status Confirmed In Progress
2017-07-21 21:20:32 Launchpad Janitor merge proposal linked https://code.launchpad.net/~ahasenack/ubuntu/+source/sssd/+git/sssd/+merge/327922
2017-07-21 21:36:30 Andreas Hasenack description Hi, I'm on Ubuntu 16.04 LTS, sssd-common 1.13.4-1ubuntu1.1, libkrb5-3 1.13.2+dfsg-5. I'm in an environment with several Active Directory sites, each with a domain controller. When remote sites' DCs are unreachable because of a VPN outage, password authentication is slow or fails. tcpdump shows the system is trying to talk to the other sites' domain controllers, and timing out. sssd-common installs the locator plugin at /usr/lib/x86_64-linux-gnu/krb5/plugins/krb5/sssd_krb5_locator_plugin.so. But I can see in strace that Kerberos apps are looking for plugins in /usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5 instead (libkrb5 vs krb5). open("/usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory) As a result, Kerberos doesn't respect SSSD's Active Directory site selection. As a workaround, if I copy /usr/lib/x86_64-linux-gnu/krb5/plugins/krb5 to /usr/lib/x86_64-linux-gnu/krb5/plugins/libkrb5, site selection works as expected. Mailing list ref: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org/thread/UUMFE5T376D3NJLNHQSJZAJCPM35KRED/ --- ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 DistroRelease: Ubuntu 16.04 JournalErrors: Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. Users in the 'systemd-journal' group can see all messages. Pass -q to turn off this notice. No journal files were opened due to insufficient permissions. Package: sssd 1.13.4-1ubuntu1.1 PackageArchitecture: amd64 ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcVersionSignature: Ubuntu 4.4.0-47.68-generic 4.4.24 Tags: xenial uec-images Uname: Linux 4.4.0-47-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True [Impact] Users cannot rely on the sssd krb5 locator plugin. Effect varies from slow logins (client trying to reach many different KDCs instead of directly the one specified by sssd configuration) to failed logins. The bug is simple, and so is the fix. The plugin was installed in the wrong directory. [Test Case] This test case does not reproduce the exact case reported by the user, but is good enough to prove that the plugin is not loaded in the broken package, and is loaded just fine in the fixed package. * install the packages on a xenial system. I suggest using LXD: $ sudo apt install sssd krb5-kdc krb5-admin-server libpam-sss For the kerberos prompts, answer: - default kerberos realm: EXAMPLE.COM - kerberos servers: just hit enter - administrative server: just hit enter * create the EXAMPLE.COM realm. Use any password during the creation, it doesn't matter: $ sudo krb5_newrealm * create the ubuntu principal in the EXAMPLE.COM realm with a password of "ubuntu". Note: please make sure your local ubuntu user uses a different password, or has none at all. When we login succesfully later, we want to be sure it was via kerberos, and not the local user. $ sudo kadmin.local -q "addprinc -pw ubuntu ubuntu@EXAMPLE.COM" * configure the krb5 libraries to use a fake realm by default - edit /etc/krb5.conf - replace the default_realm value in [libdefaults] with LOCALHOST (just so it fails quickly): [libdefaults] default_realm = LOCALHOST - do not restart the kerberos services * Create the sssd configuration file /etc/sssd/sssd.conf with these contents: """ [sssd] config_file_version = 2 services = pam domains = kerberos.example.com [pam] [domain/kerberos.example.com] id_provider = proxy proxy_lib_name = files auth_provider = krb5 krb5_server = YOURADDRESS krb5_realm = EXAMPLE.COM """ - replace YOURADDRESS with the IP of your test container or VM (do not use 127.0.0.1) - IMPORTANT: sudo chmod 0600 /etc/sssd/sssd.conf * Start sssd: $ sudo systemctl start sssd.service * in one terminal: $ tail -f /var/log/syslog * in another terminal, run: $ sudo login (or just become root and run login) * attempt to login as ubuntu with the kerberos password created earlier "ubuntu"): $ sudo login xenial-sssd-krb5-locator-1664566 login: ubuntu Password: Login incorrect * observe that syslog complains about not finding the the KDC for the EXAMPLE.COM realm: Jul 21 21:03:40 xenial-sssd-krb5-locator-1664566 [sssd[krb5_child[13628]]]: Cannot find KDC for realm "EXAMPLE.COM" * /var/log/auth will report a general PAM error with no specifics * install the fixed packages from proposed * retry the login as ubuntu: - login succeeds - no errors in /var/log/syslog - /var/log/auth will report a good login via pam_sss * run klist to verify you have the kerberos tgt: $ klist Ticket cache: FILE:/tmp/krb5cc_1000_XTpaOo Default principal: ubuntu@EXAMPLE.COM Valid starting Expires Service principal 07/21/2017 21:05:26 07/22/2017 07:05:26 krbtgt/EXAMPLE.COM@EXAMPLE.COM renew until 07/22/2017 21:05:26 - run kdestroy followed by kinit to verify you will NOT get the tgt, because /etc/krb5.conf is still specifying the incorrect realm: $ kdestroy $ kinit kinit: Cannot find KDC for realm "LOCALHOST" while getting initial credentials This proves that the krb5 locator sssd plugin was loaded and worked, because it found the right realm and its KDC via the sssd configuration only. [Regression Potential] The fix is just placing the plugin in the correct directory. Users have already been using a workaround of symlinking the file, or even copying it manually over to the right place. The directory where the plugin was located, and where it is located now, is private, and no other packages will be loading it other than SSSD. So changing its location should not affect other software installed on the system. [Other Info] None at this time.
2017-07-21 21:36:52 Andreas Hasenack description [Impact] Users cannot rely on the sssd krb5 locator plugin. Effect varies from slow logins (client trying to reach many different KDCs instead of directly the one specified by sssd configuration) to failed logins. The bug is simple, and so is the fix. The plugin was installed in the wrong directory. [Test Case] This test case does not reproduce the exact case reported by the user, but is good enough to prove that the plugin is not loaded in the broken package, and is loaded just fine in the fixed package. * install the packages on a xenial system. I suggest using LXD: $ sudo apt install sssd krb5-kdc krb5-admin-server libpam-sss For the kerberos prompts, answer: - default kerberos realm: EXAMPLE.COM - kerberos servers: just hit enter - administrative server: just hit enter * create the EXAMPLE.COM realm. Use any password during the creation, it doesn't matter: $ sudo krb5_newrealm * create the ubuntu principal in the EXAMPLE.COM realm with a password of "ubuntu". Note: please make sure your local ubuntu user uses a different password, or has none at all. When we login succesfully later, we want to be sure it was via kerberos, and not the local user. $ sudo kadmin.local -q "addprinc -pw ubuntu ubuntu@EXAMPLE.COM" * configure the krb5 libraries to use a fake realm by default - edit /etc/krb5.conf - replace the default_realm value in [libdefaults] with LOCALHOST (just so it fails quickly): [libdefaults] default_realm = LOCALHOST - do not restart the kerberos services * Create the sssd configuration file /etc/sssd/sssd.conf with these contents: """ [sssd] config_file_version = 2 services = pam domains = kerberos.example.com [pam] [domain/kerberos.example.com] id_provider = proxy proxy_lib_name = files auth_provider = krb5 krb5_server = YOURADDRESS krb5_realm = EXAMPLE.COM """ - replace YOURADDRESS with the IP of your test container or VM (do not use 127.0.0.1) - IMPORTANT: sudo chmod 0600 /etc/sssd/sssd.conf * Start sssd: $ sudo systemctl start sssd.service * in one terminal: $ tail -f /var/log/syslog * in another terminal, run: $ sudo login (or just become root and run login) * attempt to login as ubuntu with the kerberos password created earlier "ubuntu"): $ sudo login xenial-sssd-krb5-locator-1664566 login: ubuntu Password: Login incorrect * observe that syslog complains about not finding the the KDC for the EXAMPLE.COM realm: Jul 21 21:03:40 xenial-sssd-krb5-locator-1664566 [sssd[krb5_child[13628]]]: Cannot find KDC for realm "EXAMPLE.COM" * /var/log/auth will report a general PAM error with no specifics * install the fixed packages from proposed * retry the login as ubuntu: - login succeeds - no errors in /var/log/syslog - /var/log/auth will report a good login via pam_sss * run klist to verify you have the kerberos tgt: $ klist Ticket cache: FILE:/tmp/krb5cc_1000_XTpaOo Default principal: ubuntu@EXAMPLE.COM Valid starting Expires Service principal 07/21/2017 21:05:26 07/22/2017 07:05:26 krbtgt/EXAMPLE.COM@EXAMPLE.COM renew until 07/22/2017 21:05:26 - run kdestroy followed by kinit to verify you will NOT get the tgt, because /etc/krb5.conf is still specifying the incorrect realm: $ kdestroy $ kinit kinit: Cannot find KDC for realm "LOCALHOST" while getting initial credentials This proves that the krb5 locator sssd plugin was loaded and worked, because it found the right realm and its KDC via the sssd configuration only. [Regression Potential] The fix is just placing the plugin in the correct directory. Users have already been using a workaround of symlinking the file, or even copying it manually over to the right place. The directory where the plugin was located, and where it is located now, is private, and no other packages will be loading it other than SSSD. So changing its location should not affect other software installed on the system. [Other Info] None at this time. [Impact] Users cannot rely on the sssd krb5 locator plugin. Effect varies from slow logins (client trying to reach many different KDCs instead of directly the one specified by sssd configuration) to failed logins. The bug is simple, and so is the fix. The plugin was installed in the wrong directory. [Test Case] This test case does not reproduce the exact scenario reported by the user, but is good enough to prove that the plugin is not loaded in the broken package, and is loaded just fine in the fixed package. * install the packages on a xenial system. I suggest using LXD: $ sudo apt install sssd krb5-kdc krb5-admin-server libpam-sss For the kerberos prompts, answer: - default kerberos realm: EXAMPLE.COM - kerberos servers: just hit enter - administrative server: just hit enter * create the EXAMPLE.COM realm. Use any password during the creation, it doesn't matter: $ sudo krb5_newrealm * create the ubuntu principal in the EXAMPLE.COM realm with a password of "ubuntu". Note: please make sure your local ubuntu user uses a different password, or has none at all. When we login succesfully later, we want to be sure it was via kerberos, and not the local user. $ sudo kadmin.local -q "addprinc -pw ubuntu ubuntu@EXAMPLE.COM" * configure the krb5 libraries to use a fake realm by default - edit /etc/krb5.conf - replace the default_realm value in [libdefaults] with LOCALHOST (just so it fails quickly):   [libdefaults]       default_realm = LOCALHOST - do not restart the kerberos services * Create the sssd configuration file /etc/sssd/sssd.conf with these contents: """ [sssd] config_file_version = 2 services = pam domains = kerberos.example.com [pam] [domain/kerberos.example.com] id_provider = proxy proxy_lib_name = files auth_provider = krb5 krb5_server = YOURADDRESS krb5_realm = EXAMPLE.COM """ - replace YOURADDRESS with the IP of your test container or VM (do not use 127.0.0.1) - IMPORTANT: sudo chmod 0600 /etc/sssd/sssd.conf * Start sssd: $ sudo systemctl start sssd.service * in one terminal: $ tail -f /var/log/syslog * in another terminal, run: $ sudo login (or just become root and run login) * attempt to login as ubuntu with the kerberos password created earlier "ubuntu"): $ sudo login xenial-sssd-krb5-locator-1664566 login: ubuntu Password: Login incorrect * observe that syslog complains about not finding the the KDC for the EXAMPLE.COM realm: Jul 21 21:03:40 xenial-sssd-krb5-locator-1664566 [sssd[krb5_child[13628]]]: Cannot find KDC for realm "EXAMPLE.COM" * /var/log/auth will report a general PAM error with no specifics * install the fixed packages from proposed * retry the login as ubuntu: - login succeeds - no errors in /var/log/syslog - /var/log/auth will report a good login via pam_sss * run klist to verify you have the kerberos tgt: $ klist Ticket cache: FILE:/tmp/krb5cc_1000_XTpaOo Default principal: ubuntu@EXAMPLE.COM Valid starting Expires Service principal 07/21/2017 21:05:26 07/22/2017 07:05:26 krbtgt/EXAMPLE.COM@EXAMPLE.COM         renew until 07/22/2017 21:05:26 - run kdestroy followed by kinit to verify you will NOT get the tgt, because /etc/krb5.conf is still specifying the incorrect realm: $ kdestroy $ kinit kinit: Cannot find KDC for realm "LOCALHOST" while getting initial credentials This proves that the krb5 locator sssd plugin was loaded and worked, because it found the right realm and its KDC via the sssd configuration only. [Regression Potential] The fix is just placing the plugin in the correct directory. Users have already been using a workaround of symlinking the file, or even copying it manually over to the right place. The directory where the plugin was located, and where it is located now, is private, and no other packages will be loading it other than SSSD. So changing its location should not affect other software installed on the system. [Other Info] None at this time.
2017-07-21 21:41:46 Andreas Hasenack bug added subscriber Ubuntu Sponsors Team
2017-08-17 22:48:59 Brian Murray sssd (Ubuntu Xenial): status In Progress Fix Committed
2017-08-17 22:49:00 Brian Murray bug added subscriber Ubuntu Stable Release Updates Team
2017-08-17 22:49:03 Brian Murray bug added subscriber SRU Verification
2017-08-17 22:49:06 Brian Murray tags apport-collected server-next uec-images xenial apport-collected server-next uec-images verification-needed verification-needed-xenial xenial
2017-08-21 12:19:16 Andreas Hasenack tags apport-collected server-next uec-images verification-needed verification-needed-xenial xenial apport-collected server-next uec-images verification-done-xenial verification-needed xenial
2017-08-28 14:13:54 Launchpad Janitor sssd (Ubuntu Xenial): status Fix Committed Fix Released
2017-08-28 14:14:05 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team