slow NFS4 without "NEED_GSSD=yes"

[Reinhard]

Ubuntu 14.04 NFS4 client has slow access to NFS - share, if standard in /etc/default/nfs-common is left as "NEED_GSSD=".

In syslog message
[406568.806179] RPC: AUTH_GSS upcall timed out.
[406568.806179] Please check user daemon is running.
can be found.

If default is changed to "NEED_GSSD=yes" access speeds up to normal (= good as in 12.04).
I do not use Kerberos, so to set this option to "yes" does not make sense.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Stolle (martin-stolle)]

I had this problem upon upgrade to 13.10

[martin (martin-andersen)]

Another workaround is blacklisting the 'rpcsec_gss_krb5' kernel module (in /etc/modprobe.d/blacklist.conf) -

# Blacklist rpcsec_gss_krb5 kernel module; needed for quick NFSv4 mounts, ref. bug https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1270445
blacklist rpcsec_gss_krb5

But obviously the NFS subsystem should not be attempting to load this at all if the system isn't using Kerberos (i.e, no Kerberos keytabs or krb5.conf files)

[Mr. Jester (mrjester)]

This is occurring on 14.04 as well. Same error, but setting "NEED_GSSD=yes" doesn't remove the delay. Had to blacklist the module.

[Anders Hall (a.hall)]

Can confirm speed-up with blacklisting rpcsec_gss_krb5 on the client side. rmmod also works as a temporary solution.

I also tested the options to turn of security completely on the nfs-server and nfs-client, which is well documented despite numerous variables in man pages. However, those settings appears to confuse the communication between the client (e.g., sec=none,incesure,NEED_GSSD=no, etc) since it still uses rpcsec_gss_krb5. sec=none on the server side seems useless in the same fashion.

[Anders Hall (a.hall)]

Seems like redhat had the same exact issue patched (stated as solved):

http://article.gmane.org/gmane.linux.nfs/60081 (upstream)
https://bugzilla.redhat.com/show_bug.cgi?id=1001934 (redhat)

[Anders Hall (a.hall)]

This must be related or same (older) bug: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/84899

"Gabriel de Perthuis (g2p) wrote on 2012-10-24: #70

So here's a list of the workarounds:

On the client:
# disable reverse lookups in kerberos
echo $'[libdefaults]\n\trdns=false' |sudo tee -a /etc/krb5.conf
# Alternatively, remove mdns, mdns4, mdns6 from nsswitch
/etc/nsswitch.conf
# Or disable GSSAPIAuthentication in ~/.ssh/config or /etc/ssh/ssh_config or with the -o flag
GSSAPIAuthentication=no

On the server:
GSSAPIAuthentication=no in /etc/ssh/sshd_config

Fixes that require coding would be the one at http://bugs.debian.org/409360#40 which seems simple enough.
Paliatives would be a cache of notfound results in avahi or in sshd (so that the 5 seconds Avahi timeout isn't repeated for the four times ssh tries name resolution)."

[Gerhard Burger (burger.ga)]

This comment added for SEO (not easy to find this bug if you're suspecting autofs):

autofs slow mount nfs

[hanz (hansvdvoort)]

Blacklisting the Kerberos module works for me. As expected, also speeds up autofs. Awkward to have to fix this on multiple computers...

[CaCO3 (caco3)]

I can confirm this bug.
Changing /etc/default/nfs-common did not help but blacklisting the module helped.
I am running Kubuntu 14.04

[Rasmus Borup Hansen (rbh-a)]

I found that changing /etc/default/nfs-common DID help on Ubuntu 14.04, but unless you reboot you have to start the gssd service as well.

[Turner (kdrejer)]

I have this issue aswell running Xubuntu 14.04 i've tried all the above solutions, and it hasn't had any positive effects here. Transfering files to my NAS using NFS is still slow and spiky.

[Reinhard (reinhard-fink)]

Since I reported this bug, I did not change the scripts which are installing my workstations (UBUNTU 14.04)
Putting "NEED_GSSD=yes" into "/etc/default/nfs-common" still makes no sense to me,
but it works, so I wonder, why it is not working in Kubuntu/Xubuntu?

Is it possible, that this bug is a problem in communication between server and client?

My Server runs Ubuntu 11.04, with changed
"/etc/default/nfs-kernel-server"
"/etc/default/nfs-common"

this is the server configuration in "/etc/default/nfs-common"
----------------------------------------------------------------
# If you do not set values for the NEED_ options, they will be attempted
# autodetected; this should be sufficient for most people. Valid alternatives
# for the NEED_ options are "yes" and "no".

# Do you want to start the statd daemon? It is not needed for NFSv4.
NEED_STATD=

# Options for rpc.statd.
# Should rpc.statd listen on a specific port? This is especially useful
# when you have a port-based firewall. To use a fixed port, set this
# this variable to a statd argument like: "--port 4000 --outgoing-port 4001".
# For more information, see rpc.statd(8) or http://wiki.debian.org/SecuringNFS
STATDOPTS=

# Do you want to start the idmapd daemon? It is only needed for NFSv4.
NEED_IDMAPD=yes

# Do you want to start the gssd daemon? It is required for Kerberos mounts.
NEED_GSSD=no
----------------------------------------------------------------

this is the server configuration in "/etc/default/nfs-kernel-server"
----------------------------------------------------------------
# Number of servers to start up
# To disable nfsv4 on the server, specify '--no-nfs-version 4' here
RPCNFSDCOUNT=8

# Runtime priority of server (see nice(1))
RPCNFSDPRIORITY=0

# Options for rpc.mountd.
# If you have a port-based firewall, you might want to set up
# a fixed port here using the --port option. For more information,
# see rpc.mountd(8) or http://wiki.debian.org/SecuringNFS
RPCMOUNTDOPTS=--manage-gids

# Do you want to start the svcgssd daemon? It is only required for Kerberos
# exports. Valid alternatives are "yes" and "no"; the default is "no".
NEED_SVCGSSD=no

# Options for rpc.svcgssd.
RPCSVCGSSDOPTS=

# Options for rpc.nfsd.
RPCNFSDOPTS=
----------------------------------------------------------------

[Austin Godber (godber-uberhip)]

I've recently upgraded a few 12.04 machines to the new HWE stack (3.13.0-32-generic) and I am pretty sure they exhibit this bug as well. I've blacklisted the 'rpcsec_gss_krb5' and NFS seems to work "better".

[Francois Noel (kranky-franky)]

I just discovered that disabling IPv6 solves the problem. I'm on Ubuntu 14.04 64bit. I did not apply any other workaround described above.

I used that procedure: http://askubuntu.com/questions/440649/cant-disable-ipv6-in-ubuntu-14-04

[Ali Sait Erkan (ali-sait-erkan)]

Hello everyone, is there any update on this problem? I have the simplest configuration possible and it still seems NFS traffic is very slow. My client and server are both Dell Optiplex 7010 and they are both running Ubuntu 14.04. I tried all of the recommendations listed above.

[Anders Hall (a.hall)]

Hi Ali, which case is yours?

1) the NFS transfer of files between server and client is slow

2) the initial connection to the NFS, from the client, is slow

I have only tested case 2, which is solved with workarounds mentioned in this thread. Case 1 is not a problem on my test servers.

[Ali Sait Erkan (ali-sait-erkan)]

Hi Andrew, thanks for responding. It's the first one (transfer). And after some testing, the delay seems to be exasperated by the number of files, not by the total volume of transferred data (e.g. time for a single 100K is a lot worse that 100 1K file transfers, by much than one would expect).

[Anders Hall (a.hall)]

Hi Ali,

Your problem might be unrelated to this specific bug. Perhaps you need to file another bug?

[latimerio (fomember)]

As a workaround I reverted back to NFS v3 by adding -vers=3 to my /etc/auto.xxx autofs files.
Just because there is a new NFS version it does not mean it is required to use.
In the past I was happy with v3 and I will switch to v4 only when the performance problems have been solved.

[Jiri Hoogeveen (wica128)]

blacklisting rpcsec_gss_krb5 works here to on 14.04.

/etc/default/nfs-common
# Do you want to start the gssd daemon? It is required for Kerberos mounts.
NEED_GSSD=

What is the default? Yes or No?

[mazerj (james-mazer)]

Just for the record -- I just pulled down the published vagrant trusty64 image and it has the same problem. But neither blacklisting rpcsec_gss_krb5 nor changing NEED_GSSD= solves the problem. And I tried all 4 possible combinations of the two.. This sort of makes things unusable in a shared machine environment that depends on autofs! Not even sure where to start on this.

[mazerj (james-mazer)]

Actually, the solution's posted in the forum don't seem to working for
me, so it seems to be more than just a mis-configuration. Have you
managed to get this to work with 14.04LTS?
/jamie

On Thu, Dec 4, 2014 at 6:26 AM, Andre <email address hidden> wrote:
> It is merely a bug than a question. It's just not working in the default
> setup.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1270445
>
> Title:
> slow NFS4 without "NEED_GSSD=yes"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1270445/+subscriptions

--
James Mazer
Department of Neurobiology
Yale School of Medicine
phone: [phone]
fax: [phone]
url: http://jackknife.med.yale.edu

[Andre (ajx)]

I had to blacklist the module rpcsec_gss_krb5 on all Trusty machines to get NFS working at aceptable speeds again. Therefore I added the following line:

blacklist rpcsec_gss_krb5

to the following file:

/etc/modprobe.d/blacklist.conf

After a reboot everything was fine. Other comments here describe that blacklisting IPV6 would help as well.

I see three options:
1) we have all misconfigured our NFS servers (to which the previous Ubuntu releases had a tolerance)
2) there is a bug in the pre-configuration of NFS or other modules in Trusty
3) there is a bug somewhere in the binaries

Given this bug report over at RedHat:
https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1001934
with a link to Linux bugs:
http://article.gmane.org/gmane.linux.nfs/60081
indicates that there is/was a bug in the Linux kernel which was then fixed. Did that fix make it into Trusty's kernel?

[mazerj (james-mazer)]

Strange — that’s exactly what I did on the vagrant image, but it
doesn’t fix the problem.. The symptom is that the first time a fs gets
mounted it takes about 10s (presumably time for the kerberos call to
time out) and then it mounts normally. On my non-vm machines (mostly
mint 17), but at least one trusty system, just setting NEED_GSSD=yes
in /etc/default/nfs-common did the trick, no need to blacklist the
kerberos module.

But neither seems to work in the vagrant vm.. perhaps I need to double
check with a stock trusty vm image not using vagrant..

/jamie

On Fri, Dec 5, 2014 at 4:44 AM, Andre <email address hidden> wrote:
> I had to blacklist the module rpcsec_gss_krb5 on all Trusty machines to
> get NFS working at aceptable speeds again. Therefore I added the
> following line:
>
> blacklist rpcsec_gss_krb5
>
> to the following file:
>
> /etc/modprobe.d/blacklist.conf
>
> After a reboot everything was fine. Other comments here describe that
> blacklisting IPV6 would help as well.
>
> I see three options:
> 1) we have all misconfigured our NFS servers (to which the previous Ubuntu releases had a tolerance)
> 2) there is a bug in the pre-configuration of NFS or other modules in Trusty
> 3) there is a bug somewhere in the binaries
>
> Given this bug report over at RedHat:
> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1001934
> with a link to Linux bugs:
> http://article.gmane.org/gmane.linux.nfs/60081
> indicates that there is/was a bug in the Linux kernel which was then fixed. Did that fix make it into Trusty's kernel?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1270445
>
> Title:
> slow NFS4 without "NEED_GSSD=yes"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1270445/+subscriptions

--
James Mazer
Department of Neurobiology
Yale School of Medicine
phone: [phone]
fax: [phone]
url: http://jackknife.med.yale.edu

[Vytenis Silgalis (vsilgalis)]

So I do have one more workaround.

If you set sec=sys for your mount this should fix the issue.

Fixed this on a couple of boxes.

[Phil Mayhall (phil-tamachain)]

Am running 16.04 Ubuntu (Mate) and was getting 1.2Mb NFS file transfers between 2 directly wired pcs. Adding the blacklist line speeded things up to 102MB/sec so guess this is still an issue.

[Chris Routh (routhinator)]

Blacklisting the gssd module is a terrible solution for those using AD to authenticate users on the machine.

I want to authenticate users while using non-authenticated NFS shares.