Recent accountsservice update causes login window to hang

[Stephen Jahl]

Hi,

The recent 0.6.35-0ubuntu7.1 update (see https://bugs.launchpad.net/ubuntu/+source/accountsservice/+bug/1290785) seems to have removed the UID_MAX check, which could be used to hide users from the login window. In a large organization where many thousands of users are present on our systems, we relied on the UID_MAX setting in /etc/login.defs to prevent accounts-daemon from trying to frob all our accounts. accounts-daemon's performance is so poor, that not having the UID_MAX check causes lightdm to hang for 5-10mins while accounts-daemon tries to look for something in every users home folder.

I've already done things like set the login window to require the user to enter their username and password rather than list all users, but the login window still hangs while accounts-daemon is doing its thing.

The behavior I see, is:

1) just after boot, there's a black screen, right after the plymouth splash disappears, for 2-3 minutes before lightdm appears
2) You type a username into the username field and press 'Tab'
3) lightdm appears to hang for 5-10mins before the cursor moves to the password field. During this time, I see an accounts-daemon spawn for each user on the system, sequentially (in alphabetical order). Once the last user's accounts-daemon process finishes, the cursor moves to the password field.

This not only happens on first boot, but every time a user logs out and the system returns to the login window.

Any chance we can have the UID_MAX check back? Or some equivalent? This is causing a serious problem for us!

Thanks!

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ioannis Papadopoulos (ioannis-herbert-papadopoulos)]

Hello,

I am also affected by this bug, having ~6000 user accounts on my system.
I observe exactly the same things as reported in the original bug report.

I think that lightdm should not consult accountsservice at all,
when the number of existing users is large (e.g. more than 100, could be set in the lightdm configuration files).
So, probably this is more a lightdm issue.

Cheers,
Yannis

System info (lsb_release -a):
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty

[Steve Langasek (vorlon)]

Robert, have you seen this regression reported against the accountsservice SRU?

[Derek Poon (dpoon)]

I have found something close to a root cause, and I observe this hang 100% of the time under the following conditions:

1. Packages libnss-ldap_264-2.2ubuntu4.14.04.1 and accountsservice_0.6.35-0ubuntu7.1 are installed.
2. /etc/nsswitch.conf contains "passwd: files ldap"
3. An LDAP user has logged in recently, as recorded in /var/log/wtmp.
4. dbus and lightdm are started on boot.

In lightdm_10.10.5-0ubuntu1, user-list.c makes a DBus call for the method org.freedesktop.Accounts.ListCachedUsers.[1] This call times out. You can confirm that ListCachedUsers fails by running `/usr/sbin/dbus-send --system --print-reply --dest=org.freedesktop.Accounts /org/freedesktop/Accounts org.freedesktop.Accounts.ListCachedUsers`.

DBus tries to dispatch the message to accounts-daemon. However, accounts-daemon isn't listening for requests because it hasn't fully started up yet. Rather, it is stuck in the user_locale_validate() function, waiting for the /usr/share/language-tools/language-validate shell script to complete.[2] Note that the language-validate mechanism is an Ubuntu customization, not present in the upstream accountsservice 0.6.35.

The language-validate shell script runs not as root, but as an LDAP user. Bash never gets a chance to execute its first command, since it gets stuck during initialization with a getpwuid() call for that LDAP user that never returns. (To see that, I suggest hacking user_language_validate() to insert an ltrace or strace.) There is a race condition: this early in the boot process, libnss-ldap lookups just hang.

On the other hand, libnss-ldapd, which bills itself as the successor fork to libnss-ldap,[3] does not suffer from such a problem. Unfortunately, if you install ldap-auth-config_0.5.3, it will tend to pull in libnss-ldap as an automatic dependency rather than libnss-ldapd.[4]

In summary, executing `killall language-validate` after boot serves as a workaround, as it lets accounts-daemon continue its initialization. Running `service dbus restart` is a more forceful workaround, as it lets accounts-daemon launch after conditions are ready for LDAP lookups. The problem occurs with libnss-ldap_264-2.2ubuntu4.14.04.1, and it goes away when libnss-ldapd_0.8.13-3 is installed instead. That, to me, is the most sensible solution.

 [1]: http://bazaar.launchpad.net/~lightdm-team/lightdm/1.10/view/1995/common/user-list.c#L887
 [2]: http://bazaar.launchpad.net/~robert-ancell/accountsservice/lp1290785/view/head:/debian/patches/0010-set-language.patch#L60
 [3]: http://arthurdejong.org/nss-pam-ldapd/
 [4]: http://packages.ubuntu.com/trusty/ldap-auth-client

[Derek Poon (dpoon)]

Why lightdm behaves the way it does…

In common/user-list.c, load_users() calls load_passwd_file(). The intended code path is is for load_passwd_file() to make use of the org.freedesktop.Accounts.ListCachedUsers DBus call.[1] However, due to the accounts-daemon failure[2], it falls back on the slow code path, which calls load_passwd_file(), which then calls make_passwd_user() for every single user.[3] That results in a check for the existence of ~/.face and ~/.face.icon for potentially thousands of LDAP users, possibly with NFS-mounted home directories!

  [1]: https://bazaar.launchpad.net/~lightdm-team/lightdm/1.10/view/1995/common/user-list.c#L908
  [2]: https://bugs.launchpad.net/ubuntu/+source/accountsservice/+bug/1350393/comments/4
  [3]: https://bazaar.launchpad.net/~lightdm-team/lightdm/1.10/view/1995/common/user-list.c#L264

[Gunnar Hjalmarsson (gunnarhj)]

Hi!

I'm currently working with bug #1443178, and I suspect that the solution I propose there may help address the performance issue discussed above.

The idea is to save the language and locale settings, based on the system default, at the creation of a new user. That way those values would be handy available in accountsservice's keyfile, and e.g. the expensive function user_language_validate() would no longer be called for each user at each login.

[Gunnar Hjalmarsson (gunnarhj)]

Version 0.6.35-0ubuntu7.2 of accountsservice is now available in trusty-proposed, and the performance ought to have improved significantly due to the change on a system with many users.

@Stephen, @Ioannis and @Derek: It would be great if you could install the new version, and let us know about your observations as regards performance.

[Derek Poon (dpoon)]

This may be a duplicate of bug 996791.
https://bugs.launchpad.net/ubuntu/+source/accountsservice/+bug/996791

[Derek Poon (dpoon)]

@Gunnar, 0.6.35-0ubuntu7.2 may be improve performance in some cases, but it does nothing to avoid the infinite hang that I characterized in comment #4. The fact that it calls user_language_validate() just once in the context of an LDAP user when the LDAP client isn't fully ready will cause accounts-daemon to fail completely. Therefore, your revised patch should make no difference to the failure mode I described.

[Gunnar Hjalmarsson (gunnarhj)]

Right, Derek. Actually, shouldn't the LDAP issue you described be reported as a separate libnss-ldap bug?