Ubuntu

nfs4+idmap does not map uids correctly when using AUTH_SYS

Reported by Toby Corkindale on 2012-03-28
344
This bug affects 17 people
Affects Status Importance Assigned to Milestone
nfs-utils (Ubuntu)
Medium
Unassigned

Bug Description

I've observed this bug on Ubuntu Precise (beta) and Oneiric, and also on Debian Squeeze and Debian Testing.
(Kernel versions 2.6.32, 3.0.0 and 3.2.0)
I've found numerous forum posts around the internet from confused users, but no solutions.

The bug in question involves using nfs v4 with the idmapd, with users with the same username but differing uids across the client and server. The idmapping appears to have worked, until you try to write to the directories, at which point it skips the idmapping.
This is a security issue as it will allow users to access files owned by other users unexpectedly.

When listing files or directories on the client, the directories show up as owned by your local user, however attempting to write will result in a Permission Denied error. If you go back to the server and chown the directory to be owned by the uid used on the client, then the client will see the directory as owned by the incorrect user -- but WILL be able to write to it!

The log files for idmapd on both client and server appear to indicate that things are working correctly. eg:

Server's syslog: rpc.idmapd[777]: Server : (user) id "2012" -> name "postie@localdomain"
Client's syslog: rpc.idmapd[870]: Client 0: (user) name "postie@localdomain" -> id "2014"

Running commands on the client:
$ getent passwd postie
postie:x:2014:2014::/home/postie:/bin/bash
$ cd /srv/test
$ ls -l
drwxr-xr-x 2 postie root 4096 Mar 28 11:48 postie
$ ls -ln
drwxr-xr-x 2 2014 0 4096 Mar 28 11:48 postie
$ touch postie/foo
touch: cannot touch `postie/foo': Permission denied

To prove that the mount *is* mounted read-write, I'll change the ownership of the directory on the server to uid 2014, rather than the postie user there (who has uid 2012).

Now I run some commands on the client again:
$ ls -l
drwxr-xr-x 2 nobody root 4096 Mar 28 11:48 postie
$ ls -ln
drwxr-xr-x 2 65534 0 4096 Mar 28 11:48 postie
$ touch postie/foo
# It succeeds!

Any thoughts on this, or if there's a better place to report this bug?

visibility: private → public
Toby Corkindale (tjc-wintrmute) wrote :

In case it isn't clear about how this affects security, consider this case:
In a small environment without LDAP, NIS, etc, it's not unusual for the users to be created manually, and sometimes out of order, so their uids don't match between servers.

NFS Server: Has users alice (uid 1001), bob (uid 1002), craig (uid 1003)
NFS Client 1: Has users alice (uid 1001), bob (uid 1002), craig (uid 1003)
NFS Client 2: Has users alice (uid 1001), bob (uid 1003), craig (uid 1002)

Let's say each user's home directory is mounted from the server to the clients with nfs. A pretty normal situation.
On client machine #1, it would be working as expected.
On client machine #2, a sysadmin looking at the mounts would see that they were owned by the expected users.
ie. /home/alice is owned by alice, /home/bob is owned by bob, and /home/craig is owned by craig.

drwxr-x--- 2 alice users 4096 Mar 28 14:43 alice
drwxr-x--- 2 bob users 4096 Mar 30 10:43 bob
drwxr-x--- 2 craig users 4096 Mar 29 12:21 craig

However because the NFS id mapping is bugged, on client #2 with the non-matched UIDs, it will allow bob to execute something like this:
cat /home/craig/top_secret_document
echo "pwned" > /home/craig/.bashrc

Hi Toby,

On Wed, Mar 28, 2012 at 01:28:14AM -0000, Toby Corkindale wrote:

> The bug in question involves using nfs v4 with the idmapd, with users with
> the same username but differing uids across the client and server. The
> idmapping appears to have worked, until you try to write to the
> directories, at which point it skips the idmapping.

How is /etc/exports configured on your server? Are you using GSSAPI
security?

I don't see the problem you describe using GSSAPI-authenticated NFSv4
shares.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Hi Steve,
I am not using GSSAPI-authed shares here -- just plain IP-based auth.

Eg.
/srv/nfs 192.168.10.0/255.255.255.0(rw,fsid=0,async,no_subtree_check)
/srv/nfs/archive 192.168.10.0/255.255.255.0(rw,async,no_subtree_check)

Toby Corkindale (tjc-wintrmute) wrote :

/etc/idmapd.conf looks like this at the moment. (I've also tried with Domain set to another value, or with domain commented out so that imapd can assign a default automatically, which is the default in the version of nfs-utils that ships in 12.04)

[General]

Verbosity = 5
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
Domain = localdomain

[Mapping]

Nobody-User = nobody
Nobody-Group = nogroup

[Translation]

Method = nsswitch

Steve Langasek (vorlon) wrote :

Ok. I think this is almost certainly a bug in the software somewhere, not just a misconfiguration, but I'm of no help in figuring it out; I only use RPCSEC_GSS here because even if this particular issue is an implementation bug, there are plenty of other problems with AUTH_SYS that are fundamental limitations. So in the long term you're better off migrating to RPCSEC_GSS regardless of whether we manage to fix this bug.

summary: - nfs4 allows writes by incorrect users
+ nfs4+idmap does not map uids correctly when using AUTH_SYS
Changed in nfs-utils (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Toby Corkindale (tjc-wintrmute) wrote :

Having scoured the internet some more, I've found a couple more sources that confirm the idmapping works when Kerberos security is enabled.. and that it doesn't work properly without it.
The fact that the usernames are still converted correctly in listing, just not access control, still makes it look a lot like a bug rather than a feature though. (I mean, the results from "ls" should really match the access control, otherwise it's very confusing!)

In the meantime, I'll look into setting up Kerberos, but I'm pretty certain it will be a no-go here -- too much effort for what appears to be too little gain. (We already have a single-sign-on system in place)

cheers,
Toby

NiGhTHawK (clifford-u) wrote :

We are having the same problem on our servers running 11.10 but not on servers running 10.04.4 and they both configured the same (they are two different enviroments)

Also the two 11.10 boxen are all upto date and running the same kernels (I read someone solved a similar issue by upgrading kernels)

NiGhTHawK (clifford-u) wrote :

as a workaround I created the attached script (which you will need to update for correct UID and GID values, this can be used on the slave server to "sync" the UID and GID across the servers.

I had to lookup the values of the GID and UID and find a spare value that wasn't used.

*note* I used this in a two machine enviroment and it will need some work to get the UID/GID's matched across more servers.

*disclaimer* The use of this script is at your own discression if you don't under stand what it is doing then please don't use it...

Toby Corkindale (tjc-wintrmute) wrote :

Thanks Clifford; sadly, it's probably not so useful in environments with two or more NFS clients.
If we really only needed to mount one share per one client, we could use iSCSI or something.

Philipp Wendler (philw85) wrote :

I found a discussion from 2008 on the net which seems to indicate that this is by design and won't get changed:

http://thread.gmane.org/gmane.linux.nfsv4/7103/focus=7105

Sad, but probably there is nothing that can be done about this (aside from using Kerberos, syncing IDs, or switching to Samba).

Rudd-O (rudd-o) wrote :

This bug persists on 12.04. It works on Fedora though.

Ian Wells (ijw-ubuntu) wrote :

Exploring the bug, it seems that rpc.idmap on the server maps a UID to a name for RPC, and on the client does the reverse. The only earthly reason for this is to get the server/client names straight. Unfortunately it isn't applied on all operations - specifically, not when you open a file - and the discrepancy being seen is because of that. (I have to say that regardless of whether you think idmap is doing the thing it's supposed to do the resulting behaviour is just stupid, and incredibly unuseful - if you have perfectly matching IDs then there's no need for a name/id translation at all and if you don't this doesn't solve the problem.)

On Fedora they no longer use rpc.idmap, they use nfsidmap (a different kernel comms mechanism) - I don't know whether this is used in more cases or what but it's possible it behaves differently.

zwigno (zwigno) wrote :

FYI - Running across this bug when trying to get rpc.idmapd working between Redhat5/6 boxes that have the same UID but different user names. Added fun is that some of the conflicting UIDs are from LDAP and some are from files. Very annoying!

It sounds like this may have been fixed upstream?
See https://bugzilla.linux-nfs.org/show_bug.cgi?id=226

rew (r-e-wolff) wrote :

1) Confirmed: Ubuntu 11.04 has the problem, 10.04 does not.
2) I'm running the involved clients "diskless", and apparently I forgot to upgrade the tftpboot kernel for the 11.04, so the clients are now both running the same kernel: 2.6.38-8-generic-pae. The problem does not depend on the kernel.
3) I /HAVE/ matching uids on the server and client.
The server was upgraded to 14.04 and since then the 11.04 client maps all normal-uid-owned-files on the NFS mount to "nobody". This results in sshd not allowing the authorized keys file, and not being able to use your own files.

Now, "it sounds like this may have been fixed upstream" is encouraging. But what do I need to upgrade to get the fix?

What do you guys do to get idmapd to log the translations it is doing? -v option? Is there a neat way to enable it?

I've stopped the idmapd on the client. Same problem.
I've straced the idmapd on the server, and it was not doing anything when the login on the server failed again. Are the idmapd requests cached? How do I flush them?

Steve Langasek (vorlon) wrote :

11.04 has been unsupported since October 2012. You might want to re-test this with a currently supported release.

To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.