Bug #1675118 “Setting locale breaks sss_ssh_authorizedkeys: set_...” : Bugs : sssd package : Ubuntu

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2017-03-23:

#1

Hi Graham, this is unlikely to be an AppArmor issue, the usual results from AppArmor blocking access to a resource will be a Permission Denied response rather than Input/Output error.

Just in case I'm wrong though, please paste in any DENIED messages from AppArmor in your kernel's dmesg buffer.

Thanks

Revision history for this message

Steve Beattie (sbeattie) wrote on 2017-03-23:

#2

Hi Graham, thanks for reporting your issue, sorry you're having problems.

First, this is not an apparmor issue; likely it's an issue with sssd (so adjusting there).

Second, sssd in trusty did have an update published for it on 15 March 2017 https://launchpad.net/ubuntu/+source/sssd/1.11.8-0ubuntu0.5 (an update from https://launchpad.net/ubuntu/+source/sssd/1.11.8-0ubuntu0.3), so perhaps your previous successful deployment was before that update made it out.

There was also a recent glibc security update released, but that did not touch locale processing code directly. It would be good to verify that reverting to https://launchpad.net/ubuntu/+source/eglibc/2.19-0ubuntu6.9 does not address the issue.

The primary commits referenced in the upstream bug report are (for sssd 1.13 branch):
https://pagure.io/SSSD/sssd/c/4815471669a25566f6772c228c104a206ffa37f7?branch=sssd-1-13
https://pagure.io/SSSD/sssd/c/76ab3eb947f4d6fe6555d8ea0ae97dc3966f02ac?branch=sssd-1-13

affects:

apparmor (Ubuntu) → sssd (Ubuntu)

Revision history for this message

Graham Leggett (minfrin-y) wrote on 2017-03-23:

#3

I thought I set this as an sssd bug, sorry about that.

I have many machines (in the tens to hundreds) across an estate all of whom have the same sssd configuration against LDAP. Machines that came up this morning worked with respect to LDAP, the machine that was brought up yesterday and referenced in this bug report still does not. All machines are orchestrated in the same way by the same version controlled mechanism.

It looks like in my case there are a number of bugs happening simultaneously:

- Running sss_ssh_authorizedkeys from the command line fails with "set_locale() failed (5): Input/output error" when it should not fail. Backporting the fixes above should fix this.

- There are errors in auth.log saying "error: AuthorizedKeysCommand /usr/bin/sss_ssh_authorizedkeys returned status 1"

- sss_ssh_authorizedkeys is broken in that it does not log anything to syslog on failure OR ssh is broken in that when it invokes sss_ssh_authorizedkeys the output of sss_ssh_authorizedkeys is lost. For this reason it is impossible to tell for certain whether the "status 1" is caused by the set_locale failed, or something else.

For a start, it looks like sss_ssh_authorizedkeys need to log properly, something it does not do now.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-03-24:

#4

> "Machines that came up this morning worked with respect to LDAP"

Very interesting, but that makes it hard to differ from a transient issue that happened just once or if on the given day the combination of packages was in a state to cause this :-/

> "the machine that was brought up yesterday and referenced in this bug report still does not."

So at least we could check to go debugging this one as long as it is broken.

The code throwing that msg is
53 ret = set_locale();
54 if (ret != EOK) {
55 DEBUG(SSSDBG_CRIT_FAILURE,
56 "set_locale() failed (%d): %s\n", ret, strerror(ret));
57 ERROR("Error setting the locale\n");
58 ret = EXIT_FAILURE;
59 goto fini;
60 }

So it actually logs about as much as it can which makes up the error message you reported.

The fix referenced seems to apply back to trusty, but since all other machines work fine we have the problem that as soon as we change *anything* we might consider it being the fix while it is not.

Also since neither you nor I can recreate a new system to debug I'd much more like to keep the broken system broken and debug there for now.

Could you follow [1] to add debug symbols.
Then install at least sssd-common-dbgsym (actually all sssd-*-dbgsym might be safer to catch all). As well as libc6-dbg libc6-dbgsym.

I read that a direct call to /usr/bin/sss_ssh_authorizedkeys fails, so you could now run
$ gdb /usr/bin/sss_ssh_authorizedkeys
b set_locale
run <yourusername>

This will get you to the function set_locale of sssd.
It has three real things it does, step #1 is to find which one is failing for you.
Just enter "n" to step through.
One of the thee setlocale, bindtextdomain, textdomain will be the one throwing the I/O error.

I don't see how the setlocale would fail, but the two textdomain related calls at least do I/O.

Once you know which one fails you can start over and use "s" to step into.
Some gdb basics are at [2], but this can quickly get complex depending on your experience with it. Do this as deep as you can and this should bring us closer to understand what your I/O error actually is about.

If anything I'd see LOCALEDIR in the I/O error scope, but that should be /usr/share/locale for you.
Could you check just to be sure there is no random I/O or path issue what an ls to that path gives you and if there is a subdir for your locale?
I love that sssd not even isntalls own bits there :-/ at least what I see with:
$ dpkg -L $(dpkg -l '*sssd*' | awk '/ii/ {print $2}' | xargs) | grep locale

[1]: https://wiki.ubuntu.com/Debug%20Symbol%20Packages
[2]: https://sourceware.org/gdb/onlinedocs/gdb/Continuing-and-Stepping.html#Continuing-and-Stepping

> "Machines that came up this morning worked with respect to LDAP"

Very interesting, but that makes it hard to differ from a transient issue that happened just once or if on the given day the combination of packages was in a state to cause this :-/

> "the machine that was brought up yesterday and referenced in this bug report still does not."

So at least we could check to go debugging this one as long as it is broken.

The code throwing that msg is
 53     ret = set_locale();                                                          
 54     if (ret != EOK) {                                                            
 55         DEBUG(SSSDBG_CRIT_FAILURE,                                               
 56               "set_locale() failed (%d): %s\n", ret, strerror(ret));             
 57         ERROR("Error setting the locale\n");                                     
 58         ret = EXIT_FAILURE;                                                      
 59         goto fini;                                                               
 60     }