X01-localapps pathologically slow with many groups

[Dan Young]

With non-trivial numbers of users/groups (e.g. when the server LDAP authenticates), X01-localapps takes many, many minutes to return, blocking the session from starting.

The attached patch rewrites the portion of the shell script that does the group deduplication in awk.

[Stéphane Graber (stgraber)]

Is the current code still slow for you?

I remember us doing multiple iterations of that specific scripts to try to get it right (both properly parsing complex group names and being fast at the same time).

I don't necessarily have something against rewriting it in awk but experience showed that this specific script is very fragile and we can't easily regression test it, so the least we change it, the better.

[Dan Young (danyoung)]

I hope to have the opportunity to test next week.

To be honest, we just applied our patch everywhere necessary after this bug went unattended for so long.

[Launchpad Janitor (janitor)]

[Expired for LTSP because there has been no activity for 60 days.]

[Lenin (gagarin)]

This bugreport is still valid, I can confirm it in our setup of LTSP server (two load balanced servers, ldap/nis users (about 4000)): Ubuntu 12.04.2 LTS.

Since we don't use local apps (yet), we just rename X01-localapps to x01-localapps skipping it's use.

[Lenin (gagarin)]

Details on the time delay when user logs in:
On Alix hardware it's 4-5 minutes.
On Zotac hardware it's one minute.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Axel Beckert (xtaran)]

This is also a major issue for us, so we had to disable it, too. Took 53 seconds on an ALIX. Bootup is now from around a minute down to a few seconds.

[Axel Beckert (xtaran)]

(We're also running Precise on our LTSP servers and clients.)

[Alkis Georgopoulos (alkisg)]

> Details on the time delay when user logs in:
> On Alix hardware it's 4-5 minutes.
> On Zotac hardware it's one minute.

Does the attached patch make it faster? And, how much faster?

(i.e. is it slow because the LDAP calls are slow, or because the shell is much slower than awk?)

[Lenin (gagarin)]

i haven't tried. we just do what i posted at #5

[Alkis Georgopoulos (alkisg)]

I rewrote X01-localapps so that it only reads the groups that the user is a member of,
instead of all of the available groups,
could someone test it before I commit it?

It should be way faster, and it should support spaces and other LDAP/AD characters:
http://paste.debian.net/65929/

[Alkis Georgopoulos (alkisg)]

Fixed a typo, better use that one instead:
http://bazaar.launchpad.net/~alkisg/+junk/ltsp/view/head:/client/localapps/ldm-rc.d/X01-localapps

[Alkis Georgopoulos (alkisg)]

Fix committed in http://bazaar.launchpad.net/~ltsp-upstream/ltsp/ltsp-trunk/revision/2498.

[Alkis Georgopoulos (alkisg)]

A note about a change in behavior with the committed fix:

Previously, if a user U was a member of group G,
then all other users U1 U2 U3... that were members of that G group were also created locally in the LTSP client.

Now they aren't, and that should make the script execute in msecs even if thousands of users are in the same G group (e.g. a group named "users" that contains everyone).

But if someone has a use case where he still needs the old behavior, please mention it in this bug report.

[Alkis Georgopoulos (alkisg)]

Fix released in LTSP 5.4.6 (http://bazaar.launchpad.net/~ltsp-upstream/ltsp/ltsp-trunk/revision/2502).

[Launchpad Janitor (janitor)]

This bug was fixed in the package ltsp - 5.4.6-1ubuntu1

---------------
ltsp (5.4.6-1ubuntu1) trusty; urgency=low

  * Merge from Debian unstable. Remaining changes:
    - Change default color depth from 16bit to 24bit as 16bit just doesn't
      work when using unity with llvmpipe. This is a workaround and
      alternatives should be looked at to get back to 16bit if possible.
    - Add 40-ltsp as 40-ltsp-client and 40-ltsp-server in Xsession.d for both
      ltsp-client-core and ltsp-server. (LP: #987726)
    - Make ltsp-client-builder depend on console-setup-udeb rather than
      kbd-chooser for Ubuntu.
    - Make ltsp-client recommend zram-config
    - Make ltsp-client depend on dbus, dmz-cursor-theme,
      fonts-dejavu-core, fonts-freefont-ttf, fonts-kacst-one,
      fonts-khmeros-core, fonts-lao, fonts-liberation, fonts-nanum,
      fonts-takao-pgothic, fonts-thai-tlwg, language-pack-en, libgl1-mesa-dri,
      plymouth-theme-ubuntu-logo, plymouth-theme-ubuntu-text,
      ttf-indic-fonts-core, ttf-punjabi-fonts, ttf-ubuntu-font-family,
      ttf-wqy-microhei and wget.
    - Make ltsp-client-core depend on busybox-static, udhcpd and usbutils.
    - Ship a distro-specific dhcpd.conf using the good old
      192.168.0.0/24 subnet instead of upstream's default 192.168.67.0/24.
    - Add fat client support to the udeb.
    - Have the udeb detect Ubuntu flavours with only main and restricted.
    - Fix race condition during image compression in udeb.
    - Patch dhcpd.conf on the fly to match current architecture.
    - Create a squashfs image by default and configure network in d-i.
    - Install ltsp-cluster.
    - Add code for second stage Edubuntu LTSP install.
    - Change watch file to monitor Debian.

ltsp (5.4.6-1) unstable; urgency=low

  * New upstream version:
    - Use help2man to generate man pages (LP #1008053).
    - Updated German translation by Wolfgang Schweer (Closes: #702915).

    - ltsp-client:
      + pulseaudio: Add module-suspend-on-idle to improve stability.
        Thanks to Tom Wallis for reporting the issue and fix.
      + Optimize localapps user and group handling (LP: #357268).
      + Move the localapps cleanup after X terminates (LP: #1093144).
      + ltsp-cleanup: Only remove local users (LP: #1037034).
      + Avoid Xorg crashes caused by nouveau dri (LP: #1072711).
      + Make dbus machine id constant per client.

    - ltsp-build-client:
      + Mount /sys in chroot during installation.
        Thanks to Petter Reinholdtsen (Closes: #721597).
      + Fix ignored --debootstrap-keyring commandline option.

  * ltsp-client-core:
    + Recommend pciutils, used by several screen-session hooks.
    + Remove python-daemon dependency, as jetpipe no longer uses it.
    + Recommend dnsutils or knot-dnsutils, which is used to find a valid DNS
      server (Closes: #704271).
    + Drop custom udev rules for ltsp-sound, which is now installed at runtime.

  * Updated patches:
    - use-test-binary
    - jetpipe-from-ltsp-client-core-init-script
  * Removed patches applied upstream:
    - fix-ltsp-cleanup-bashism
    - fix-ltsp-config-bashisms
    - fix-screen-x-common-bashisms
    - fix-dashisms-using-local-dash
...

[Alkis Georgopoulos (alkisg)]

Fix released in LTSP 5.4.6 (http://bazaar.launchpad.net/~ltsp-upstream/ltsp/ltsp-trunk/revision/2502).