nfs4+idmap does not map uids correctly when using AUTH_SYS

[Toby Corkindale]

I've observed this bug on Ubuntu Precise (beta) and Oneiric, and also on Debian Squeeze and Debian Testing.
(Kernel versions 2.6.32, 3.0.0 and 3.2.0)
I've found numerous forum posts around the internet from confused users, but no solutions.

The bug in question involves using nfs v4 with the idmapd, with users with the same username but differing uids across the client and server. The idmapping appears to have worked, until you try to write to the directories, at which point it skips the idmapping.
This is a security issue as it will allow users to access files owned by other users unexpectedly.

When listing files or directories on the client, the directories show up as owned by your local user, however attempting to write will result in a Permission Denied error. If you go back to the server and chown the directory to be owned by the uid used on the client, then the client will see the directory as owned by the incorrect user -- but WILL be able to write to it!

The log files for idmapd on both client and server appear to indicate that things are working correctly. eg:

Server's syslog: rpc.idmapd[777]: Server : (user) id "2012" -> name "postie@localdomain"
Client's syslog: rpc.idmapd[870]: Client 0: (user) name "postie@localdomain" -> id "2014"

Running commands on the client:
$ getent passwd postie
postie:x:2014:2014::/home/postie:/bin/bash
$ cd /srv/test
$ ls -l
drwxr-xr-x 2 postie root 4096 Mar 28 11:48 postie
$ ls -ln
drwxr-xr-x 2 2014 0 4096 Mar 28 11:48 postie
$ touch postie/foo
touch: cannot touch `postie/foo': Permission denied

To prove that the mount *is* mounted read-write, I'll change the ownership of the directory on the server to uid 2014, rather than the postie user there (who has uid 2012).

Now I run some commands on the client again:
$ ls -l
drwxr-xr-x 2 nobody root 4096 Mar 28 11:48 postie
$ ls -ln
drwxr-xr-x 2 65534 0 4096 Mar 28 11:48 postie
$ touch postie/foo
# It succeeds!

Any thoughts on this, or if there's a better place to report this bug?

[Toby Corkindale (tjc-wintrmute)]

In case it isn't clear about how this affects security, consider this case:
In a small environment without LDAP, NIS, etc, it's not unusual for the users to be created manually, and sometimes out of order, so their uids don't match between servers.

NFS Server: Has users alice (uid 1001), bob (uid 1002), craig (uid 1003)
NFS Client 1: Has users alice (uid 1001), bob (uid 1002), craig (uid 1003)
NFS Client 2: Has users alice (uid 1001), bob (uid 1003), craig (uid 1002)

Let's say each user's home directory is mounted from the server to the clients with nfs. A pretty normal situation.
On client machine #1, it would be working as expected.
On client machine #2, a sysadmin looking at the mounts would see that they were owned by the expected users.
ie. /home/alice is owned by alice, /home/bob is owned by bob, and /home/craig is owned by craig.

drwxr-x--- 2 alice users 4096 Mar 28 14:43 alice
drwxr-x--- 2 bob users 4096 Mar 30 10:43 bob
drwxr-x--- 2 craig users 4096 Mar 29 12:21 craig

However because the NFS id mapping is bugged, on client #2 with the non-matched UIDs, it will allow bob to execute something like this:
cat /home/craig/top_secret_document
echo "pwned" > /home/craig/.bashrc

[Steve Langasek (vorlon)]

Hi Toby,

On Wed, Mar 28, 2012 at 01:28:14AM -0000, Toby Corkindale wrote:

> The bug in question involves using nfs v4 with the idmapd, with users with
> the same username but differing uids across the client and server. The
> idmapping appears to have worked, until you try to write to the
> directories, at which point it skips the idmapping.

How is /etc/exports configured on your server? Are you using GSSAPI
security?

I don't see the problem you describe using GSSAPI-authenticated NFSv4
shares.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Toby Corkindale (tjc-wintrmute)]

Hi Steve,
I am not using GSSAPI-authed shares here -- just plain IP-based auth.

Eg.
/srv/nfs 192.168.10.0/255.255.255.0(rw,fsid=0,async,no_subtree_check)
/srv/nfs/archive 192.168.10.0/255.255.255.0(rw,async,no_subtree_check)

[Toby Corkindale (tjc-wintrmute)]

/etc/idmapd.conf looks like this at the moment. (I've also tried with Domain set to another value, or with domain commented out so that imapd can assign a default automatically, which is the default in the version of nfs-utils that ships in 12.04)

[General]

Verbosity = 5
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
Domain = localdomain

[Mapping]

Nobody-User = nobody
Nobody-Group = nogroup

[Translation]

Method = nsswitch

[Steve Langasek (vorlon)]

Ok. I think this is almost certainly a bug in the software somewhere, not just a misconfiguration, but I'm of no help in figuring it out; I only use RPCSEC_GSS here because even if this particular issue is an implementation bug, there are plenty of other problems with AUTH_SYS that are fundamental limitations. So in the long term you're better off migrating to RPCSEC_GSS regardless of whether we manage to fix this bug.

[Toby Corkindale (tjc-wintrmute)]

Having scoured the internet some more, I've found a couple more sources that confirm the idmapping works when Kerberos security is enabled.. and that it doesn't work properly without it.
The fact that the usernames are still converted correctly in listing, just not access control, still makes it look a lot like a bug rather than a feature though. (I mean, the results from "ls" should really match the access control, otherwise it's very confusing!)

In the meantime, I'll look into setting up Kerberos, but I'm pretty certain it will be a no-go here -- too much effort for what appears to be too little gain. (We already have a single-sign-on system in place)

cheers,
Toby

[NiGhTHawK (clifford-u)]

We are having the same problem on our servers running 11.10 but not on servers running 10.04.4 and they both configured the same (they are two different enviroments)

Also the two 11.10 boxen are all upto date and running the same kernels (I read someone solved a similar issue by upgrading kernels)

[NiGhTHawK (clifford-u)]

as a workaround I created the attached script (which you will need to update for correct UID and GID values, this can be used on the slave server to "sync" the UID and GID across the servers.

I had to lookup the values of the GID and UID and find a spare value that wasn't used.

*note* I used this in a two machine enviroment and it will need some work to get the UID/GID's matched across more servers.

*disclaimer* The use of this script is at your own discression if you don't under stand what it is doing then please don't use it...

[Toby Corkindale (tjc-wintrmute)]

Thanks Clifford; sadly, it's probably not so useful in environments with two or more NFS clients.
If we really only needed to mount one share per one client, we could use iSCSI or something.

[Philipp Wendler (philw85)]

I found a discussion from 2008 on the net which seems to indicate that this is by design and won't get changed:

http://thread.gmane.org/gmane.linux.nfsv4/7103/focus=7105

Sad, but probably there is nothing that can be done about this (aside from using Kerberos, syncing IDs, or switching to Samba).

[Rudd-O (rudd-o)]

This bug persists on 12.04. It works on Fedora though.

[Ian Wells (ijw-ubuntu)]

Exploring the bug, it seems that rpc.idmap on the server maps a UID to a name for RPC, and on the client does the reverse. The only earthly reason for this is to get the server/client names straight. Unfortunately it isn't applied on all operations - specifically, not when you open a file - and the discrepancy being seen is because of that. (I have to say that regardless of whether you think idmap is doing the thing it's supposed to do the resulting behaviour is just stupid, and incredibly unuseful - if you have perfectly matching IDs then there's no need for a name/id translation at all and if you don't this doesn't solve the problem.)

On Fedora they no longer use rpc.idmap, they use nfsidmap (a different kernel comms mechanism) - I don't know whether this is used in more cases or what but it's possible it behaves differently.

[zwigno (zwigno)]

FYI - Running across this bug when trying to get rpc.idmapd working between Redhat5/6 boxes that have the same UID but different user names. Added fun is that some of the conflicting UIDs are from LDAP and some are from files. Very annoying!

[Toby Corkindale (tjc-wintrmute)]

It sounds like this may have been fixed upstream?
See https://bugzilla.linux-nfs.org/show_bug.cgi?id=226

[rew (r-e-wolff)]

1) Confirmed: Ubuntu 11.04 has the problem, 10.04 does not.
2) I'm running the involved clients "diskless", and apparently I forgot to upgrade the tftpboot kernel for the 11.04, so the clients are now both running the same kernel: 2.6.38-8-generic-pae. The problem does not depend on the kernel.
3) I /HAVE/ matching uids on the server and client.
The server was upgraded to 14.04 and since then the 11.04 client maps all normal-uid-owned-files on the NFS mount to "nobody". This results in sshd not allowing the authorized keys file, and not being able to use your own files.

Now, "it sounds like this may have been fixed upstream" is encouraging. But what do I need to upgrade to get the fix?

What do you guys do to get idmapd to log the translations it is doing? -v option? Is there a neat way to enable it?

I've stopped the idmapd on the client. Same problem.
I've straced the idmapd on the server, and it was not doing anything when the login on the server failed again. Are the idmapd requests cached? How do I flush them?

[Steve Langasek (vorlon)]

11.04 has been unsupported since October 2012. You might want to re-test this with a currently supported release.

[Ancoron Luciferis (ancoron-luciferis)]

Just wanted to add my findings to this as I was also curious to get user/group mapping to work without the need of a Kerberos server.

However, what I saw especially in comments and documentation from upstream is that this was never designed to work that way. Instead, the idmapper only works in the way I expected (sending username@realm instead of UID) when GSS auth is being configured on the NFS server side which (currently) requires that you have a Kerberos server that manages the authentication tickets for both, the server and the client and serves as the trusting third party, so conceptually a little bit similar to the extra authentication services we usually see around the net.

If that is not available to you I am afraid you are stuck with the old auth_sys, which means sticking to same UIDs/GIDs on server and clients.

In the end I have to say that this is quite inflexibly designed and by design excludes home setups where a separate authentication party is not desirable, without any secure alternative.

[Mark Solaris (a-mark-8)]

I can toggle the good and bad behaviour by adding or removing "files" to passwd_compat in /etc/nsswitch.conf

Causes problem:

passwd_compat: files ldap
group_compat: files ldap

host:/root root# getent passwd 1119
host:/root root#

No problem:

passwd_compat: ldap
group_compat: ldap

host:/root root# getent passwd 1119
testuser:*:1119:1000:Test User:/home/testuser:/bin/bash

Adding "files" to that line is suggested in some HOWTOs.

We'll likely remove compat entirely and just go with "files ldap" on the "passwd:" line earlier in /etc/nsswitch.conf, we don't need the NIS-like behvaiours here.

[Aidan Walton (aidan-walton)]

Having just spent several days trying to get to the root cause of this id-mapping issue I am strongly inclined to agree with Ancoron regarding the forced usage of UID & GID with AUTH_SYS.
I have exactly the issue he describes. This is a small home setup and I do not have nor do I desire to set-up and configure a Kerberos infrastructure. I have 3 OSs running. Linux OSX and Windows. The Windows client is a corner case where I am forced to make it emulate a server side UID & GID as it lacks NFSv4 anyway. However the OSX machines and Linux machines where set-up a different times. They have different UID & GIDs for the same named user. I made the move to NFS4 as I was under the impression that id-mapping would allow a setup where the UIDs & GIDs of these machines could stay different to those on the server and by running id-mapping all client requests would contain the string based names to enable the server to map these incoming request to a local UID & GID. No such luck.

Having watched carefully the exchanges on the wire I would suggest that the current implementation is neither working nor consistent in this scenario.

If id-mapping is configured on the server /sys/module/nfsd/parameters/nfs4_disable_idmapping = 0 and the same is also configured on the client in : /sys/module/nfs/parameters/nfs4_disable_idmapping = 0
The client sends all its request only ever using UID & GID in the RPC header. The NFS4 Attr mask Owner and Owner_Group are always blank.
However replies from the server to these same requests always have these NFS4 attr containing the mapped strings. The client honours this and maps correctly to the required client side UID & GID using a call to nfsidmap, which subsequently call nsswitch:
nfs4_name_to_gid: nsswitch->name_to_gid returned 0

So essentially we have one way mapping, working correctly from Server to Client, but not available from Client to Server.

I can understand that there may well be security implications for a simple mapping scenario, but to be blunt, who cares! these are my machines and my network, its not large and I'm not paranoid. If I was truly worried about security then I would implement Kerberos, but I am not.

However due to the fact that the client does not offer NFS4 attr as the string name, the server can never perform the same mapping in the client to server direction.

This forces either matching UID & GID on all clients and server, or separate UID & potentially GIDs on each machine, or use of all_squash in the export to crunch all client to server requests into one user or still further per client IP address based exports with different all_squash options. All of these are sub-optimal.

I would be interested to know if anyone is familiar enough with the RFCs to know whether the use of AUTH_SYS within the RPC header specifically precludes the use of id-mapping by the server for incoming requests. If this is not strictly forbidden then why could the use of AUTH_SYS with string based NFS4 attr not be a perfectly valid possibility. IMHO it would offer flexibility better than what is currently available and much simpler for the home or less security concerned user.