idmap should be started by default because mount.nfs now negotiates NFSv4 before NFSv3

[Pedro Lopes]

I have a NFS client/server setup where both client and server are Ubuntu 10.10 (both upgraded from 10.4)

To make UID and GID mapping trivial, I've made sure that clients and server all have the same users and groups with the same IDs. This used to work fine. However now the client is always displaying 4294967294 for the UID and GID but - and here is the strange bit - if I create a file on the client it shows up on the server with the correct user and group (but on the client still with 4294967294).

I posted this originally on the forums, and decided to file this bug when 2 other people reported having the same problem since upgrading to 10.10. In both cases they were using Ubuntu as client and Solaris as server. See thread here: http://ubuntuforums.org/showthread.php?p=9983624

Additional info:

/etc/exports line on the server:
data/fileserver 192.168.1.0/255.255.255.0(rw,sync,no_subtree_check)

/etc/fstab line on the client:
htpc:/data/fileserver /data/fileserver nfs defaults 0 0

Trying to chown a file on the client gives the error "chown: changing ownership of `somefile': Invalid argument"

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: nfs-common 1:1.2.2-1ubuntu1
ProcVersionSignature: Ubuntu 2.6.35-22.34-generic 2.6.35.4
Uname: Linux 2.6.35-22-generic x86_64
NonfreeKernelModules: nvidia
Architecture: amd64
Date: Mon Oct 18 15:40:16 2010
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.utf8
 SHELL=/bin/bash
SourcePackage: nfs-utils

[Martin Felder (martin-felder)]

I can confirm this bug for our mixed environment (Solaris servers using zfs, and Ubuntu clients). It does seriously affect us, since the behaviour of file creation is quite erratic (as far as I can see). Additional observations:

1. No problems with older Solaris servers running NFS3 and NFS2 - seems to be specific to NFS4.

2. Creating dir on NFS4 share from Ubuntu 10.10:

ubuntu1010> md test; ls -l
drwxrwx--- 2 4294967294 4294967294 3 2010-10-20 09:00 test
This shows up correctly on Solaris boxes and Ubuntu 10.04 mounting the same share:
drwxrwx--- 2 martin staff 2 2010-10-20 09:00 test

3. If I change ownership on Ubuntu 10.10, strange things occur:
ubuntu1010> chown martin:staff test; ls -l
drwxrwx--- 2 martin staff 3 2010-10-20 09:00 test
ubuntu1010> cd test
bash: cd: test: Permission denied
--> This is really annoying!

Meanwhile, on the other machines:
ubuntu1004> ls -l
drwxrwx--- 2 4294967294 4294967294 3 2010-10-20 09:00 test
solaris> ls -l
drwxrwx--- 2 nobody nobody 3 2010-10-20 09:00 test

So, apparently my user info does not get transmitted to the server either, which makes changing permissions extremely hazardous.

I thus recommend rating this bug as serious.

[Erkki Seppälä (flux-inside)]

For what it's worth, I resolved the same (?) problem involving a Solaris ZFS server and Ubuntu 10.10 nfs client by using nfs mount option vers=3.

[Nick Schef (scheffer-nicolas)]

Solaris servers, ubuntu 10.10 clients. This used to work fine.

Exact same problem as the one above:
"However now the client is always displaying 4294967294 for the UID and GID but - and here is the strange bit - if I create a file on the client it shows up on the server with the correct user and group (but on the client still with 4294967294)."

This creates other problems with the desktop such as a strange behavior of pulse audio, gdm, thunderbird that thinks my home directory is not owned by me.

[John (w00t-cky)]

same problem here, client 10.10 and server 10.10 gives me UID and GID 4294967294. UID and GID are correct on the server but not on the client.

It works however if i connect an ubuntu 10.04 client to the 10.10 server. So it seems to be a client issue. The solution im using is autofs for all my users, witch cause alot of problem for me so i did a rollback on my servers =(.

[Perry Nguyen (pfnguyen)]

Not a bug it seems, but needs documentation or something somewhere.

10.10 enables NFSv4 and I was able to fix it by enabling nfs idmap on both server and client.

On my solaris server, I edited /etc/default/nfs and set NFSMAPID_DOMAIN to my server's name (domain name, whatever), then did svcadm restart nfs/mapid

On ubuntu, I edited /etc/idmapd.conf and changed Domain to be the same value as what I set for NFSMAPID_DOMAIN, edit /etc/default/nfs-common and set NEED_STATD=no and NEED_IDMAPD=yes, then service rpc_pipefs restart and service idmapd restart, remount your nfs shares (in my case, service autofs restart); and all the uid/gid mapping should now be correct.

[Oebele drijfhout (oebele.drijfhout)]

I believe this is documented already here: https://help.ubuntu.com/community/NFSv4Howto

[Pedro Lopes (paol)]

@Perry:

Thanks, from your suggestion I made the changes to /etc/default/nfs-common and started idmapd, and the problem was resolved.

@Tiemen:

Thanks for the link, but even the wiki authors seem confused about this issue...

So, now it seems clear that it is a default configuration problem rather than a bug. I suggest this bug report should stand as a request to change the default configuration. As it currently stands it breaks NFS setups that were working in previous versions of Ubuntu.

[Adriaan van Nijendaal (adriaan-choam)]

This morning, I had a working NFS set-up on an 10.04 server with 10.10 clients. After upgrading the server to 10.10, I had this problem as well. Setting NEED_STATD=no and NEED_IDMAPD=yes on BOTH ends fixes it, indeed. Thanks a lot for the info!

And... yes! I would have appreciated a default configuration that would have worked 'out of the box' (or at least some warning that things have changed and what to do - which would have saved me a few hours).

I'd like to support the request for a change in the configuration as well.

[Klaus Reichl (klaus-reichl)]

We have the problem, that the nfs-server is externally hosted and NFS entries are published via automounter maps.

Running now 10.10 on an NFS client we have the situation:

* mount uses NFS4 per default
* autmounter maps don't give something like vers=3
=> nfs stuff is mounted nobody/nogroup

The only solution I tested successfully so far is to
manually mount the things with:
   mount ... vers=3 ...

Another solution, I'm thinking about is to replace
/sbin/mount.nfs which is now /sbin/mount.nfs4 really
with an version mount version 3 per default.
But that's ugly!

Is there something I miss here? Can I configure a default for mount?

Thanx for info
Klaus

[Klaus Reichl (klaus-reichl)]

Hi out there,

As there is no activity here, I leave my (working) hack in place. Just posting to help others with the same problem.

Picked up /sbin/mount.nfs from Ubuntu 10.04 installation. This defaults to nfs3 with works around the (I suppose) server side bug with nfs4.

Have fun,
Klaus

[Karsten Suehring (suehring)]

We are also affected.

I have worked around the automounter problem by using auto.net and editing the opt variable to contain "-vers=3"

-- Karsten

[idistech (gary-idistech)]

im getting the same issue....
previously server was 10.04, and clients 10.10.
after upgrade of server to 10.10, both had uid/gid problems.

I have fixed it on one client machine 10.10 by setting the client mount options to include ...nfsvers=3
but on another client 10.10 machine, both patched yesterday, the nfsvers option is not recognised and fails...mounting nfs4 works, but with broken uid/gids...
mount.nfs4 is the same on both machines...
the working/successful client is 10.10 desktop...and the failing version is a 10.10 server..

very strange

[Radu Cristescu (radu.c)]

I only accidentally found this bug using Ubuntu 11.04 pre-installed daily build 20110409 for ARM on a Pandaboard. All my 10.10 i386 machines work fine, but now that I checked, they're actually not fine, so it was just luck that it didn't cause me any trouble. Setting NEED_IDMAPD=yes fixes the problem on the Pandaboard and on my i386 machines. Without that option, "start idmapd" will give me a PID, but when I check with ps there's no rpc.idmapd around.

[exa (examachine)]

This bug persists in Ubuntu 11.04 on amd64. The nfsv4 is absolutely not working, can anybody tell me why this terrible code is the default? Please remove this non-functional kernel server thing from the distribution, it doesn't work as it ought to. If nfsv4 isn't working, then what's the point of this package?

I'll check again about NEED_IDMAPD, it's probably set on my system. I don't think it solves anything as Radu claims.

[exa (examachine)]

BTW, this is likely a 64-bit problem, as that number is -2, right? Maybe it's an exceptionally bad programming error. I guess I won't see this problem on i386.

So, maybe there is a serious 64-bit bug, in this case, why is the nfs-user-server package removed from the distribution, why are the choices being limited despite the fact that this bug has not been solved for over a year? Or are you not testing these things on 64-bit machines?

BTW, the claims that this can be fixed simply by changing the configuration seems to be wrong. The default config in 11.04 already turns on that non-working idmapd.

This bug can be observed on a single machine running ubuntu 11.04 on amd64 currently. I'll write more info as I have it.

[Steve Langasek (vorlon)]

On Wed, Aug 17, 2011 at 06:51:31AM -0000, exa wrote:
> This bug persists in Ubuntu 11.04 on amd64. The nfsv4 is absolutely not
> working, can anybody tell me why this terrible code is the default?

It's not "terrible code". NFSv4 works perfectly well if configured.

> Please remove this non-functional kernel server thing from the
> distribution,

That's not a reasonable request by any measure. It's perfectly functional
for many people - myself included.

> I'll check again about NEED_IDMAPD, it's probably set on my system.

I don't know why you think it's "probably set on your system". It wouldn't
be set by default. If it *is* set on your system and you're experiencing
problems, then you're seeing a different issue than this one.

> BTW, this is likely a 64-bit problem, as that number is -2, right? Maybe
> it's an exceptionally bad programming error. I guess I won't see this
> problem on i386.

Incorrect.

> in this case, why is the nfs-user-server package removed from the
> distribution,

Because that's a dead project with a variety of other deficiencies, and the
Debian package maintainer requested its removal.

> why are the choices being limited despite the fact that this bug has not
> been solved for over a year?

nfs-user-server was removed well before this bug became an issue.

> Or are you not testing these things on 64-bit machines?

This has nothing to do with 32- vs 64-bit systems. (NFSv4 works fine here
on 64-bit installs.)

> BTW, the claims that this can be fixed simply by changing the
> configuration seems to be wrong. The default config in 11.04 already
> turns on that non-working idmapd.

idmapd is turned on by default only on servers, and on clients which specify
nfs4 as a filesystem type in /etc/fstab. Because mount.nfs now negotiates
NFSv4 first if available, this is no longer the correct default behavior; we
should instead start idmapd unconditionally.

So this is definitely a bug - but it's also one for which there is a trivial
workaround. Either edit /etc/default/nfs-common to start idmapd, or if your
server advertises NFSv4 but is not configured to do id mapping correctly,
use nfsvers=3 in your mount options.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[exa (examachine)]

Hi Steve,

On Wed, Aug 17, 2011 at 12:28 PM, Steve Langasek <
<email address hidden>> wrote:

> > BTW, this is likely a 64-bit problem, as that number is -2, right? Maybe
> > it's an exceptionally bad programming error. I guess I won't see this
> > problem on i386.
>
> Incorrect.
>
>
Have you tested it on amd64?

I'll take a look at the config if it can indeed be solved by trivial
configuration. IIRC, on the machine I tested, I did check the proposed flags
and they were correct. I didn't even touch them.

Cheers,

--
Eray Ozkural, PhD candidate. Comp. Sci. Dept., Bilkent University, Ankara
http://groups.yahoo.com/group/ai-philosophy
http://myspace.com/arizanesil http://myspace.com/malfunct

[exa (examachine)]

Dear Steve,

Thank you for your interest in the matter. It's costing me precious time, so all help is greatly appreciated. Many thanks!

I've now checked my nfs-common config, it had the following relevant bits:

# Do you want to start the statd daemon? It is not needed for NFSv4.
NEED_STATD=yes
...
# Do you want to start the idmapd daemon? It is only needed for NFSv4.
NEED_IDMAPD=yes

GSSD is turned off by default, and I hope it has nothing to do with this bug.

For testing purposes I'm using a single NFS4 export line (but it fails with any config, that is not relevant):
/exports *(ro,fsid=0,no_subtree_check,async)

After this point I restarted nfs server, and the mount still failed to produce correct uid/gid values, but then....

So, well, does idmapd run? Yes, I've checked:
examachine@cylon:~$ ps aux | grep idmap
1000 32731 0.0 0.0 13124 1060 pts/3 S+ 17:31 0:00 grep --color=auto idmap
examachine@cylon:~$ sudo start idmapd
start: Job is already running: idmapd
examachine@cylon:~$ ps aux | grep idmap
root 622 0.0 0.0 27336 888 ? Ss 17:35 0:00 rpc.idmapd
1000 2038 0.0 0.0 13128 1060 pts/3 S+ 17:47 0:00 grep --color=auto idmap

However, AFTER the above command, strangely, idmapd suddenly starts giving the right behavior, could it be a permissions or locking problem or something trivial like that? Or starting idmapd only if nfs I believe I'm using the desktop installation.

So, idmapd is running, then what is with the nfs-common config? Oh, well, it does work now, but why didn't it work before? Issuing sudo start idmapd didn't change anything?

Ok, the problem might be with the logic that looks at whether nfs4 entries are present on the fstab. I might want to mount them from the command line, etc, and I shouldn't have to start daemons manually to make that happen.

Slightly at a loss, although happy that my problem is solved by a Deus Ex Machina :)

The point is that, I had to restart idmapd, *even though* the configuration was correct, and the system claimed it to be already running. Those who suffer from this bug, please take note. And please can somebody test this (somewhat unexpected) workaround on a vanilla 11.04 system? Note that I didn't add anything to fstab. That's why it's a little strange. That is to say, there is definitely a bug but I can't pinpoint where exactly.

If you need extra information from my system, please let me know.

Cheers,

[exa (examachine)]

Dear Steve,

Thank you for your interest in the matter. It's cost me precious time, so all help is greatly appreciated. Many thanks!

I've now checked my nfs-common config, it had the following relevant bits:

# Do you want to start the statd daemon? It is not needed for NFSv4.
NEED_STATD=yes
...
# Do you want to start the idmapd daemon? It is only needed for NFSv4.
NEED_IDMAPD=yes

GSSD is turned off by default, and I hope it has nothing to do with this bug.

For testing purposes I'm using a single NFS4 export line (but it fails with any config, that is not relevant):
/exports *(ro,fsid=0,no_subtree_check,async)

After this point I restarted nfs server, and the mount still failed to produce correct uid/gid values, but then....

So, well, does idmapd run? Yes, I've checked:
examachine@cylon:~$ sudo start idmapd
start: Job is already running: idmapd
Was already running?
examachine@cylon:~$ ps aux | grep idmap
root 622 0.0 0.0 27336 888 ? Ss 17:35 0:00 rpc.idmapd
1000 2038 0.0 0.0 13128 1060 pts/3 S+ 17:47 0:00 grep --color=auto idmap

However, *after* the above command, strangely, idmapd suddenly starts giving the right behavior, could it be a permissions or locking problem or something trivial like that? Or starting idmapd only if nfs I believe I'm using the desktop installation.

So, idmapd is running, then what is with the nfs-common config? Oh, well, it does work now, but why didn't it work before? Issuing sudo start idmapd didn't change anything?

Ok, the problem might be with the logic that looks at whether nfs4 entries are present on the fstab. I might want to mount them from the command line, etc, and I shouldn't have to start daemons manually to make that happen.

Slightly at a loss, although happy that my problem is solved by a Deus Ex Machina :)

The point is that, I had to restart idmapd, *even though* the configuration was correct, and the system claimed it to be already running. Those who suffer from this bug, please take note. And please can somebody test this (somewhat unexpected) workaround on a vanilla 11.04 system? Note that I didn't add anything to fstab. That's why it's a little strange. That is to say, there is definitely a bug but I can't pinpoint where exactly.

If you need extra information from my system, please let me know.

Best Regards,

[Steve Langasek (vorlon)]

On Wed, Aug 17, 2011 at 02:53:48PM -0000, exa wrote:
> I've now checked my nfs-common config, it had the following relevant
> bits:

> # Do you want to start the statd daemon? It is not needed for NFSv4.
> NEED_STATD=yes
> ...
> # Do you want to start the idmapd daemon? It is only needed for NFSv4.
> NEED_IDMAPD=yes

Ok.

> GSSD is turned off by default, and I hope it has nothing to do with this
> bug.

Yes, gssd is unrelated unless you are using sec=krb5* mount options.

> For testing purposes I'm using a single NFS4 export line (but it fails with any config, that is not relevant):
> /exports *(ro,fsid=0,no_subtree_check,async)

> After this point I restarted nfs server, and the mount still failed to
> produce correct uid/gid values, but then....

> So, well, does idmapd run? Yes, I've checked:
> examachine@cylon:~$ ps aux | grep idmap
> 1000 32731 0.0 0.0 13124 1060 pts/3 S+ 17:31 0:00 grep --color=auto idmap
> examachine@cylon:~$ sudo start idmapd
> start: Job is already running: idmapd
> examachine@cylon:~$ ps aux | grep idmap
> root 622 0.0 0.0 27336 888 ? Ss 17:35 0:00 rpc.idmapd
> 1000 2038 0.0 0.0 13128 1060 pts/3 S+ 17:47 0:00 grep --color=auto idmap

Is this on the client or server?

The above output doesn't make sense to me. First you show ps output with no
idmapd running; then you show the output of a 'start' command that reports
that it's already running; then it appears in the ps output. None of these
things follow from one another.

It may be that you're encountering a bug in the startup of idmapd at boot,
unrelated to this bug report. There is an SRU awaiting verification for
precisely such an issue: bug #811823.

> However, AFTER the above command, strangely, idmapd suddenly starts
> giving the right behavior, could it be a permissions or locking problem
> or something trivial like that? Or starting idmapd only if nfs I believe
> I'm using the desktop installation.

As mentioned, idmapd will not start up automatically if it doesn't detect
that it's needed. When it declines to start for this reason, the upstart
job will be left in 'stopped' state. So it should be enough to edit the
config file and run 'service idmapd start' once to fix the problem.

> Ok, the problem might be with the logic that looks at whether nfs4
> entries are present on the fstab. I might want to mount them from the
> command line, etc, and I shouldn't have to start daemons manually to
> make that happen.

Yes, that's the bit here that I've confirmed is a bug in nfs-utils.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Launchpad Janitor (janitor)]

This bug was fixed in the package nfs-utils - 1:1.2.4-1ubuntu3

---------------
nfs-utils (1:1.2.4-1ubuntu3) precise; urgency=low

  * debian/nfs-common.defaults, debian/nfs-common.idmapd.upstart: idmapd
    should always be started automatically, because we can no longer assume
    that a mount of type 'nfs' in /etc/fstab is not nfs4. This also lets
    things work by default with nfs4 autofs. LP: #662711.
  * Move /var/lib/nfs/rpc_pipefs to /run/rpc_pipefs. This does not belong
    in /var/lib.
  * Ignore errors from mount if the filesystem is already mounted.
    LP: #811823.
 -- Steve Langasek <email address hidden> Thu, 27 Oct 2011 12:04:58 -0700

[Nathan Grennan (ngrennan)]

Where is the update for Natty? 1:1.2.4-1ubuntu3 looks like an oneiric update.

[Stefan Bader (smb)]

The package actually is for Precise (12.04). The question is how important it is to have the change backported as this can be simply changed in /etc/default/nfs-common. That compared to requiring effort to backport the changes, make sure things work as expected and release the package.

[Richard Laager (rlaager)]

Has anyone gotten this to work with a Solaris/OpenIndiana server? I'm using Precise as the client and haven't had much luck. I've tried setting the idmap domain on both sides to no avail.

[Casey Stone (caseystone)]

Googling my error 'invalid argument' when trying to chown a file on Ubuntu server 12.04.1 32bit NFS shared directory (NFS client) and Open Indiana 151a5 (NFS server) brought me here, but it seemingly is a different bug since adding NEED_IDMAPD=YES does not fix it. Still searching for an answer how to make the NFS shared folder behave normally with regard to permissions/ownership on the Ubuntu machine.