performance of accounts-daemon is very poor

Bug #941673 reported by Peter Gray on 2012-02-26
58
This bug affects 10 people
Affects Status Importance Assigned to Milestone
accountsservice
Unknown
Medium
accountsservice (Ubuntu)
High
Unassigned
Oneiric
Medium
Unassigned
Precise
High
Unassigned

Bug Description

Version 11.10

The performance of accounts-daemon is unacceptable with large password and/or group files. We have appoximately 40000 users in the passwords file and 16000 lines in the group file. Having looked at the source, it appears to me the problem is that for each user is pulls from the password database (via getpwXXX) its then calls getgroups to return the users list of groups.

Getgroups must make a complete pass through the group file to determine the groups for the user. When initializing this results in
reading the groups file 40000 times.

I think a better solution is to build a inverted group file map then first time getgroups is called by reading the entire group database and building a data structure indexed by username where a users group list can by found in a single lookup.
Of couse, an even better solution would be for getgroups to do that itself, but that would require much larger code changes.

I tried installed nscd to see if this would make a difference but no dice.

At the moment, 11.10 is unusable for us because of this problem.

Hope this is all the info you need.

Regards, pdg

<email address hidden>

Peter Gray (pdg-o) wrote :

Sorry, I said getgroups in my original post, and I should have said getgrouplist.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in accountsservice (Ubuntu):
status: New → Confirmed
Sebastien Bacher (seb128) wrote :

Thank you for your bug report, that seems like the sort of issues that should be discussed upstream, somebody should report a bug on https://bugs.freedesktop.org/enter_bug.cgi?product=accountsservice

While the issue is an important one it's limited to some big deployments and those can probably uninstall accountsservices (lightdm will fallback if it's not sure), ideally we would work on the issue but we have limited resources and it's not on the priority list

Changed in accountsservice (Ubuntu):
importance: Undecided → High

Original report: https://bugs.launchpad.net/bugs/941673

The performance of accounts-daemon is unacceptable with large password and/or group files. We have appoximately 40000 users in the passwords file and 16000 lines in the group file. Having looked at the source, it appears to me the problem is that for each user is pulls from the password database (via getpwXXX) its then calls getgrouplist to return the users list of groups.

Getgroups must make a complete pass through the group file to determine the groups for the user. When initializing this results in
reading the groups file 40000 times.

I think a better solution is to build a inverted group file map then first time getgroups is called by reading the entire group database and building a data structure indexed by username where a users group list can by found in a single lookup.
Of couse, an even better solution would be for getgroups to do that itself, but that would require much larger code changes.

I tried installed nscd to see if this would make a difference but no
dice.

Vadim Rutkovsky (roignac) wrote :

Thank you for your bug report. This bug has been reported to the developers of the software. You can track it and make comments at https://bugs.freedesktop.org/show_bug.cgi?id=48177

Changed in accountsservice (Ubuntu):
status: Confirmed → Triaged
Changed in accountsservice:
importance: Unknown → Medium
status: Unknown → Confirmed

AccountsService sort of assumes users in /etc/passwd are "local users" and that the 40000 user network login cases are covered by nis/ldap/etc

I guess we should do some sort of sanity check and treat /etc/passwd users as remote in the case the check fails. This way we sidestep all the costly work entirely.

maybe we could just bail if there are greater than N found users. The problem then is finding a suitable value of N.

Leonardo Borda (lborda) wrote :

Hi Guys,

As a workaround for the time being one could do the following (tested on 12.04):

Remove accountsservice
sudo apt-get remove accountsservice

Edit /etc/lightdm/lightdm.conf and add the following line :
greeter-hide-users=true

Then restart lightdm or reboot the machine:
sudo service lightdm restart

Leonardo Borda

Julian Clark (julian.clark) wrote :

My setup is Ubuntu 12.04 with Gnome + Compiz in use instead of Unity.

I am not quite confident with the suggested workaround. The package "accountsservice" appears to be a key dependency for some seemingly important packages. In my case, those packages are "gnome-control-center," "language-selector-common," and "language-selector-gnome." The package "ubuntu-desktop" (which is a commonly-installed package) has "gnome-control-center" as a dependency. I cannot see how uninstalling "accountsservice" will not significantly reduce overall functionality and/or break something.

Chris J Arges (arges) on 2012-06-20
Changed in accountsservice (Ubuntu Oneiric):
status: New → Triaged
Changed in accountsservice (Ubuntu Precise):
status: New → Triaged
importance: Undecided → High
Changed in accountsservice (Ubuntu Oneiric):
importance: Undecided → Medium
Peter Gray (pdg-o) wrote :

FYI,

I think I have a workaround for this problem by putting

UID_MIN 9999999

into /etc/gdm/custom.conf

This seems fragile since I am not sure the interaction of accountsservices and custom.conf is stable or documented but it appears to solve the problem. Of course, this is not a general solution for all sites, but happens to work for me.

The attachment "a temporary hackish workaround/patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch

Created attachment 65941
proposed patch

limit the no of user listed by a-s to UID_MAX. Also, sets user-limit to true when this condition is meet, allowing apps like gdm/lightdm to enable manual login option.

Talked with Ritesh about this bug today on IRC. He's going to look into using fgetgrent to create the hash table of local groups and use that.

marek keram (makory1) on 2012-08-25
Changed in accountsservice (Ubuntu):
status: Triaged → Fix Released
Changed in accountsservice (Ubuntu Oneiric):
status: Triaged → Fix Released
Changed in accountsservice (Ubuntu Precise):
status: Triaged → Fix Released

Created attachment 66171
proposed patch

Changed in accountsservice (Ubuntu Precise):
status: Fix Released → Triaged
Changed in accountsservice (Ubuntu Oneiric):
status: Fix Released → Triaged
Changed in accountsservice (Ubuntu):
status: Fix Released → Triaged
Changed in accountsservice (Ubuntu Precise):
milestone: none → ubuntu-12.04.2

ritz, any chance you could get this applying against master (preferably as a git-format-patch formatted patch) ?

Changed in accountsservice (Ubuntu Precise):
assignee: nobody → Ritesh Khadgaray (khadgaray)
Changed in accountsservice (Ubuntu):
assignee: nobody → Ritesh Khadgaray (khadgaray)
status: Triaged → In Progress
Changed in accountsservice (Ubuntu Precise):
status: Triaged → In Progress
Tatsis Dimitrios (dimitrist16) wrote :

Our setup is affected by this bug when the ldap server is offline. What is the proper way to disable accounts service and 'fallback lightdm' ? I removed accountsservice and everything works fine except that a handful of packets depend on it (Ubuntu 10.04)

* The following packages will be REMOVED:
* accountsservice gdm gnome-control-center indicator-datetime indicator-power language-selector-common
* language-selector-gnome ubuntu-desktop ubuntu-standard

I really don't see why all these dependencies exist. Isn't there supposed to be an alternative ?

any updates on this?

On Fri, Jan 04, 2013 at 04:20:15PM -0000, Rstrode wrote:
> any updates on this?

Errr... I am not working on this. I just reported it.

I managed a work around so the problem went away for me.

Regards,
pdg

Created attachment 73763
patch

Hi halfline

  I have rewritten the code to use hashtable instead of list, leading to lot faster startup time on my test system ( <1s for ~40k users). I am currently testing the code ( add/remove users option ). This is for an older version of a-s (0.6.15) , will update the code against head and post.

  Anything which does need the user list ( gnome-control-center user manager/ indicator-session/..) still breaks .

  Personally I believe, the best solution would be to allow user to select an upper limit ( and list only cached users from login history ), and broadcast "user-limit-reached" option. The broadcast could be used by applications to display warning or change their behaviour ( such as disable user list in gdm ).

Created attachment 74177
patch

initial patch, against latest release.

still need to work quirk against entry_generator_wtmp. It calls getpwnam for every entry ( even if it is a duplicate entry ).

Colin Watson (cjwatson) on 2013-02-13
Changed in accountsservice (Ubuntu Precise):
milestone: ubuntu-12.04.2 → ubuntu-12.04.3

any chance you could update this again for git master?

Changed in accountsservice (Ubuntu Precise):
assignee: Ritesh Khadgaray (khadgaray) → nobody
Changed in accountsservice (Ubuntu):
assignee: Ritesh Khadgaray (khadgaray) → nobody
Rolf Leggewie (r0lf) wrote :

oneiric has seen the end of its life and is no longer receiving any updates. Marking the oneiric task for this ticket as "Won't Fix".

Changed in accountsservice (Ubuntu Oneiric):
status: Triaged → Won't Fix

Created attachment 124799
user: check if user is in wheel more efficiently

We currently get all the groups a user belongs to in one pass,
then check each one to see if it's wheel.

It's much more efficient to just get the wheel group and check if
any of its members are the user.

Created attachment 124800
daemon: get local users from /etc/shadow not /etc/passwd

For some sites, it's common practice to rsync around large
/etc/passwd files containing the password entries for remote
users. That means accountsservices' "assume /etc/passwd is local
users" heuristic falls over.

This commit changes it to only treat users in /etc/shadow as local.

Created attachment 124801
daemon: don't call getspnam for local users

We're already iterating over the whole shadow file, so
just cache the entries instead of calling getspname a
few lines later.

Created attachment 124802
daemon: constrain max local users to 50

Systems with tens of thousands of users don't want all those users
showing up in the user list.

Set a cap at an even 50, which should cover the lion's share of use
cases well. Of course, if a user not in the list explicitly
logs in (from Not Listed? or whatever) they get added to the list.

Created attachment 124803
daemon: don't source user list from wtmp

wtmp can get rather large on some systems from ssh logins.
Furthermore it's pretty much completely redundant given the user
cache in /var/lib/AccountService

This commit changes the wtmp code to only get used for maintaining
login frequency and accounting, not for generating new users.

i'm going to push these patches to master. They're loosely based on ritz work in this bug, but they diverge in some significant ways:

1) i set a limit on the number of local users returned (ala comment 2)
2) i redid the wheel check to avoid needing an admin user cache
3) i cache shadow information a little differently and for a shorter period of time
4) i get "local user list" from /etc/shadow now instead of /etc/passwd (since /etc/passwd gets rsynced around)

Attachment 124799 pushed as c85b41d - user: check if user is in wheel more efficiently
Attachment 124800 pushed as 98f4287 - daemon: get local users from /etc/shadow not /etc/passwd
Attachment 124801 pushed as 14ca424 - daemon: don't call getspnam for local users
Attachment 124802 pushed as 20c0315 - daemon: constrain max local users to 50
Attachment 124803 pushed as a2f448b - daemon: don't source user list from wtmp

Unconditioned #include <shadow.h> breaks the build on FreeBSD.

FreeBSD doesn't have shadow.h and /etc/shadow. Hashed passwords are stored in /etc/master.passwd, which is a /etc/passwd with additional fields. There is no API specifically made to read /etc/master.passwd because getpwent on FreeBSD automatically switches to use /etc/spwd.db (the binary form of /etc/master.passwd) instead of /etc/pwd.db (the binary form of /etc/passwd) when the user has permission to read the file.

Same on OpenBSD. Since that change, accountsservice is broken here as well.

is it common practice on BSD to rsync /etc/passwd around? Maybe bsd could go back to using /etc/passwd?

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/accountsservice/accountsservice/issues/39.

Changed in accountsservice:
status: Confirmed → Unknown
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.