Kerberos, NFS4 and autofs issue

[maxjos]

Ubuntu 9.04 latest update.

When mounting the users home folder over NFS4 using Kerberos with RPCGSSDOPTS="-n" set in /etc/defaults/nfs-common a kerberos ticket is acquired for the NFS service thus allowing for other autofs kerberized mounts to work as well. However, if home is not on kerberos NFS (local) and the user trying to access kerberized NFS exports after logging in, a NFS kerberos ticket will fail to be acquired and the user must go through several manual steps for kerberos to pick up an NFS ticket. This is one way to do it:

$ sudo kinit
$ sudo ls -l /mountpoint

At this point the automount will still fail as now the kerberos ticket is owned by root, however, if you change the owner of the ticket back to the original user, automount will be able to mount/access the kerberized NFS export. As mentioned at the beginning, this is not the case if the users home is NFS mounted as it seems to trigger a function that will automatically make Ubuntu acquire NFS kerberos ticket (machine credentials?). Note I'm not using client keytabs in this setup.

I've added some verbose logging to this to try and figure out what the issue could be but the strange thing is the logs say the same even if it is able to mount: rpc.gssd access denied errors and failed to create krb5 context for uid 0.

Is the mounting process by design? What triggers the mounts to work when $HOME is mounted over NFS and why do they fail if it is not?

PS: this should be pretty easy to replicate if you have a working krb5/nfs4/autofs setup, simply point the /home autofs to somewhere else like e.g. /tmphome. Add RPCGSSDOPTS="-n" in /etc/defaults/nfs-common and restart.

[Russ Allbery (rra-debian)]

I'm not sure what package this is a problem with, but I can say with some certainty that it isn't kerberos-configs. This package only provides the krb5.conf configuration to find the KDCs and do other library initialization.

This sounds like a bug in the NFS v4 userspace processes, if I understand your bug report properly. Am I correct in thinking you already have a Kerberos ticket but no service ticket for NFS is acquired to access the NFS mount? If so, that means the GSS daemon isn't doing what it should.

If the problem is that you have no Kerberos ticket at all, then I don't think there's any bug here, just a configuration error on your system. In order to let the user access directories in Kerberos-authenticated NFS, you need to ensure that a Kerberos ticket is acquired for the user before they try. This will require pam-krb5 be installed and configured so they get a ticket at login.

[maxjos (magne-fortytwo)]

>Am I correct in thinking you already have a Kerberos ticket but no service ticket for NFS is acquired to access the NFS mount? If so, that means the GSS daemon isn't doing what it should.

Yes, this is correct. Service ticket for NFS is missing unless users home is mounted over NFS.

[Russ Allbery (rra-debian)]

Yeah, that sounds like a bug in the NFS userspace portions to me.

The way that I understand this is supposed to work is that the NFS Kerberos support is divided into two components: a userspace daemon that finds the user's ticket cache, grabs credentials where necessary, and loads them into the kernel, and the kernel support which just presents those kernel credentials to the file server.

It sounds like the userspace daemon isn't finding the ticket cache and doing its magic with it.

Just one thing to double-check: when this happens, what's the file name of the user's ticket cache? Does it contain the UID?

[maxjos (magne-fortytwo)]

Yes, the ticket is /tmp/krb5cc_<uid>_<randstring> and the only difference in these two scenarios I describe is that when $HOME is NFS mounted `klist` show that I have the "nfs/<hostname>@REALM" service ticket whilst it does not in the second scenario.

From what I could see in the nfs-utils docs though, RPCGSSDOPTS="-n" means you have to acquire the NFS service ticket as root manually before the user is able to mount any kerberized exports. This leads me to believe something is triggered in the login process (as root) making kerberos able to get the service ticket and insert into my credentials *IF* my home is NFS mounted. Plausible?

I have tried logging in through both gdm and terminal and the behaviour is exactly the same. pam_krb5.so issue?

Here is some proof of concept, first with autofs mouting my home under /net/home/user1:

Ubuntu 9.04 tty1

hostname login: user1
Password:
[ .. snip .. ]

user1@hostname:~# klist
Ticket cache: FILE:/tmp/krb5cc_1001_F4RoT
Default principal: user1@REALM

Valid starting Expires Service principal
04/28/09 17:31:12 04/29/09 03:31:12 krbtgt/REALM@RELAM
    renew until 04/29/09 17:31:12

Kerberos 4 ticket cache: /tmp/tkt1001
klist: You have no tickets cached

user1@hostname:~# ls -l /net/home/user1
ls: cannot access /net/home/user1: No such file or directory
user1@hostname:~# logout

In syslog I see rpc.gssd complain about 'CC file '/tmp/krb5cc_1001_F4RoT owned by 1001, no 0' and automount 'access denied by server while mounting ....'.

Now I log in as root and modify autofs to mount my home at /home/user1 then restart autofs daemon and log in as user1 again:

Ubuntu 9.04 tty1

hostname login: user1
Password:
[ .. snip .. ]

user1@hostname:~# klist
Ticket cache: FILE:/tmp/krb5cc_1001_Ru4r3l
Default principal: user1@REALM

Valid starting Expires Service principal
04/28/09 17:36:12 04/29/09 03:36:12 krbtgt/REALM@RELAM
    renew until 04/29/09 17:36:12
04/28/09 17:36:12 04/29/09 03:36:12 nfs/nfs-server.example.com@RELAM
    renew until 04/29/09 17:36:12

Kerberos 4 ticket cache: /tmp/tkt1001
klist: You have no tickets cached

user1@hostname:~# logout

Magically I now have an NFS service ticket apparently because my $HOME is pointing at the NFS server and I can easily browse my NFS home folder....

In syslog I see something that might explain the behaviour. rpc.gssd says: 'CC file '/tmp/krb5cc_pam_<randstring>' (user1@REALM) passed all checks and has mtime of <unixtime>' and 'using FILE:/tmp/krb5cc_pam_<randstring> as credentials cache for client with uid 0 for server nfs-server.example.com'. Apparently PAM does some magic if it sees your home folder lives on NFS and creates a temporary credentials cache? Does this help to figure out what is causing this and how to fix it?

Thanks
Max

PS: I couldn't copy/paste the above console output so it might have some typos

[Russ Allbery (rra-debian)]

Okay, I understand what's going on now. I'm not sure what to do about
it, though. Hopefully someone else who understands what's going on will
be able to help. I suspect that this ticket needs to be reassigned to
either nfs-common or to autofs.

maxjos <email address hidden> writes:

> From what I could see in the nfs-utils docs though, RPCGSSDOPTS="-n"
> means you have to acquire the NFS service ticket as root manually
> before the user is able to mount any kerberized exports. This leads me
> to believe something is triggered in the login process (as root)
> making kerberos able to get the service ticket and insert into my
> credentials *IF* my home is NFS mounted. Plausible?

Yes.

The issue is that there are two separate operations involved in
Kerberos-authenticated NFS access so far as I can tell from the gssd man
page. (I'm somewhat handicapped here by not using Kerberized NFS; I
find AFS a better solution for what I need to do.)

First, there's mounting of the file system. Normally, this is done
using machine credentials (which presumably means /etc/krb5.keytab).
With the -n option, this can instead be done using root's user-space
credentials.

Second, there's authenticated access to files in the file system once
it's mounted. This is presumably done using the user's own regular
credentials.

The problem, when combined with autofs, is that mounts are done as the
root user rather than as the user triggering the autofs mount, but root
doesn't have the user's credentials. Hence, autofs mounting doesn't
work unless you manually obtain Kerberos credentials as root.

That this happens to work when you're auto-mounting the user's home
directory is purely an accident of implementation in pam-krb5. It
relies on unsupported and undocumented behavior of pam-krb5 and could
well break again in future versions. Due to some implementation
difficulties with how OpenSSH does authentication, pam-krb5 saves a
temporary ticket cache to the file system during the auth stack and then
moves that ticket cache to the user's proper final ticket cache during
the session stack (or setcred call). That temporary ticket cache is
only intended to be used internally by pam-krb5; that it's stored in the
file system at all is only because OpenSSH doesn't pass PAM data around
in a way that would let it be kept only in memory. Because it's only
used internally, pam-krb5 doesn't bother chowning it, which means it's
owned by root.

rpc.gssd happens to find that ticket cache when autofs fires during the
login process and, since it's owned by root, happily uses it to do the
mount. This is actually something of a security weirdness in how
rpc.gssd works -- there's a race condition here in its scan of /tmp for
root-owned tickets and possible logins that could lead it to mounting a
user's home directory using some other user's credentials. This is an
inherent security flaw in how rpc.gssd works, which is probably not
fixable as long as it scans /tmp for ticket caches and uses whatever it
happens to find.

If autofs happens later, after the PAM authentication has successfully
completed, this temporary ticket cache of course no longer exists and
therefore the mount cannot ...

[Timo Aaltonen (tjaalton)]

moving to nfs-utils for now.

[Jan Bernlöhr (janbernloehr)]

This is still an issue with ubuntu 11.10! When the -n option is passed to rpc.gssd it still uses uid=0 to mount the user shares which is stupid and makes kerberos protected homes inaccessible from ubuntu...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.