GDM3 hangs, if local home directory is not accessible

[Michal Kukuča]

In Ubuntu 17.10 with GDM 3.26.1-3ubuntu3, the greeter will hang, if local /home directory is not accessible.

Our setup:
- Ubuntu Server (16.04) with DNS, LDAP, Kerberos 5 and NFS servers. The home directories are present on the server and made available via NFS with krb5 authentication. LDAP provides the user data for clients.
- Ubuntu client (17.10), bound to the server, uses AutoFS to automount remote home directories under /home. This makes the otherwise present local user directories unaccessible.

Expected behaviour: After system startup, the greeter should provide a way for users to log in.
Actual behaviour: The system hangs before the greeter displays the user list (but it does display the top menu bar and Ubuntu logo at the bottom).

Additional remarks:
If the settings for local users under /var/lib/AccountsService/users contain SystemAccount=true, the greeter will work as expected (while not displaying local users). This is a workaround, that I'm using right now. BUT: if the network user will log in and invoke the authentication dialog for system-wide settings (e.g. if he will try to add a new printer, or change the system update settings), the system will hang before displaying the expected dialog window. Also if a local user will try to log in by entering his login name and password, the system will hang (it should either produce an error message, or log in without a home directory).

AutoFS uses this line to mount the remote home directories under /home:
* -fstype=nfs4,rw,intr,sec=krb5 FQDN:/home/&

[Sebastien Bacher (seb128)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. The issue you are reporting is an upstream one and it would be nice if somebody having it could send the bug to the developers of the software by following the instructions at https://wiki.ubuntu.com/Bugs/Upstream/GNOME. If you have done so, please tell us the number of the upstream bug (or the link), so we can add a bugwatch that will inform us about its status. Thanks in advance.

[Michal Kukuča (djaeris)]

This bug seems not to be related to GDM at all. This is a problem either with automount, or mount.nfs4.

I did spend some time today digging around this and found out that the hang lasts for several minutes, but then the system springs back to life and the user list in the greeter will get populated.

Related stuff from journald (user testuser is an LDAP user, remote home; user localadmin is local - his home is not accessible):
okt 28 16:07:17 paulus systemd[1]: Starting GNOME Display Manager...
okt 28 16:07:17 paulus systemd[1]: Started Regular background program processing daemon.
okt 28 16:07:17 paulus automount[1566]: Starting automounter version 5.1.2, master map /etc/auto.master
okt 28 16:07:17 paulus automount[1566]: using kernel protocol version 5.02
okt 28 16:07:17 paulus automount[1566]: lookup_nss_read_master: reading master file /etc/auto.master
okt 28 16:07:17 paulus automount[1566]: do_init: parse(sun): init gathered global options: (null)
okt 28 16:07:17 paulus systemd[1]: Started GNOME Display Manager.
okt 28 16:07:17 paulus automount[1566]: lookup_read_master: lookup(file): read entry /home
okt 28 16:07:17 paulus automount[1566]: master_do_mount: mounting /home
okt 28 16:07:17 paulus automount[1566]: automount_path_to_fifo: fifo name /var/run/autofs.fifo-home
okt 28 16:07:17 paulus automount[1566]: lookup_nss_read_map: reading map file /etc/auto.home
okt 28 16:07:17 paulus automount[1566]: do_init: parse(sun): init gathered global options: (null)
okt 28 16:07:18 paulus automount[1566]: mounted indirect on /home with timeout 300, freq 75 seconds
okt 28 16:07:18 paulus automount[1566]: st_ready: st_ready(): state = 0 path /home
okt 28 16:07:18 paulus systemd[1]: Started Automounts filesystems on demand.
okt 28 16:07:18 paulus systemd[1]: Reached target Multi-User System.
okt 28 16:07:18 paulus systemd[1]: Reached target Graphical Interface.
********SNIP
okt 28 16:07:37 paulus automount[1566]: handle_packet: type = 3
okt 28 16:07:37 paulus automount[1566]: handle_packet_missing_indirect: token 2, name testuser, request pid 1805
okt 28 16:07:37 paulus automount[1566]: attempting to mount entry /home/testuser
okt 28 16:07:37 paulus automount[1566]: lookup_mount: lookup(file): looking up testuser
okt 28 16:07:37 paulus automount[1566]: lookup_mount: lookup(file): testuser -> -fstype=nfs4,rw,intr,sec=krb5 petrus.(REDACTED):/home/&
okt 28 16:07:37 paulus automount[1566]: parse_mount: parse(sun): expanded entry: -fstype=nfs4,rw,intr,sec=krb5 petrus.(REDACTED):/home/testuser
okt 28 16:07:37 paulus automount[1566]: parse_mount: parse(sun): gathered options: fstype=nfs4,rw,intr,sec=krb5
okt 28 16:07:37 paulus automount[1566]: parse_mount: parse(sun): dequote("petrus.(REDACTED):/home/testuser") -> petrus.(REDACTED):/home/testuser
okt 28 16:07:37 paulus automount[1566]: parse_mount: parse(sun): core of entry: options=fstype=nfs4,rw,intr,sec=krb5, loc=petrus.(REDACTED):/home/testuser
okt 28 16:07:37 paulus automount[1566]: sun_mount: parse(sun): mounting root /home, mountpoint testuser, what petrus.(REDACTED):/home/testuser, fstype nfs4, options rw,intr,sec=krb5
okt 28 16:07:37 paulus automount[1566]: mount_mount: mount(nfs): root=/home nam...

[Andreas Hasenack (ahasenack)]

How do you fetch your users from ldap, using sssd? Or nss_ldap? Or something else? Does that need authentication, or are anonymous searches allowed?

Does the mount()ing work if you temporarily create the user locally in /etc/{passwd,group,shadow} but not his/her home directory?

This bug is a bit broad in its current state, it would help if you could narrow it down a bit.

[Michal Kukuča (michalmaria)]

We're using SSSD. LDAP does not use authentication for search.

In the mean time we already upgraded to 18.04 and I'm not sure if the bug persists (but will try it again and post the results here - the workaround is still in place). The issue seems to be with the home directory being mounted from the remote host and GDM and other (local) processes trying to read from that directory with root access. That cannot work by default, as NFS mounts with the local root are mapped to nfs_nobody. In order to circumvent this I tried setting the no_root_squash option in the exports file, which should in theory enable local root processes to access /home normally. Well - that ended with another bug filed with nfs-utils, as no_root_squash does not work (it didn't work with 17.10 - that was a known issue with that particular nfs server version, but it seems to be still there with 18.04). I still have one idea that I will try this weekend, but this has become a nice big rabbit hole for me.

[Andreas Hasenack (ahasenack)]

But the terminal login shouldn't suffer from this root-can't-read-home, should it? Didn't you experience this there as well?

If the problem is indeed root-squash, then the login should also fail without autofs. Just mount the home directory first. Can you get this down to a very simple test case with a new machine or vm?

[Michal Kukuča (michalmaria)]

I have finally found some time to look into this, and turns out, it's not a problem anymore (with 18.04 at least). If I set the local user back to normal (so it does appear in the greeter), GDM works normally (it will display the user list, without any freezing). Of course, you cannot log into the local user, because the home directory is simply not there (the remote /home is mounted onto the local /home).

As a side note: when you do attempt a login into the local user (with the correct credentials), you will not get any error message (apart from the journal), which is not a very user-friendly solution - GDM just restarts the greeter.

When I try to login using the terminal, the login will work (as it always did - also with 17.10), but bash will complain, that it cannot chdir into the home directory and access its stuff there.

So again: I posted this bug in October 2017, and I did not know then where to properly post this and now (with Bionic) I can't reproduce the problem. It probably was not a bug with autofs. What's relevant for me now is the root_squash issue here: #1733101

[Michal Kukuča (michalmaria)]

Here's the link to the NFS issue: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1733101

[Andreas Hasenack (ahasenack)]

autofs should still work and mount /home for the terminal user that logged in. Even after you login, and get the bash warning about the home directory not being there: if you access the home directory, it still won't get mounted?

[Michal Kukuča (michalmaria)]

Yes, the /home directory is there - but it's the remote one.

[Launchpad Janitor (janitor)]

[Expired for autofs (Ubuntu) because there has been no activity for 60 days.]