NFS mounting hangs instead of returning with permission denied

Bug #214041 reported by Lonnie Olson on 2008-04-08
60
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Debian
Fix Released
Unknown
Fedora
Won't Fix
Low
nfs-utils (Ubuntu)
High
Steve Langasek
Hardy
High
Unassigned
Intrepid
High
Steve Langasek

Bug Description

Binary package hint: nfs-common

Hardy does not properly return Permission denied correctly when attempting to mount NFS shares that are denied. Instead it hangs for 2 minutes as if the error was a connection timeout.

This error is hardy specific as gutsy properly returned Permission denied. Following, you will find output from two different machines: one running Hardy 8.04, the other running Gutsy 7.10. Both are attempting to mount the very same NFS share on a Solaris 10 server. Neither machine actually has permission to mount this share. Both machines *can* mount shares they do have permission for on the same server.

This is critical as it severely impacts the usefulness of autofs.

------------ Hardy test case ------------------
$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=8.04
DISTRIB_CODENAME=hardy
DISTRIB_DESCRIPTION="Ubuntu hardy (development branch)"

$ uname -a
Linux fungus 2.6.24-15-generic #1 SMP Fri Apr 4 03:48:31 UTC 2008 i686 GNU/Linux

$ dpkg -l | grep nfs-common
ii nfs-common 1:1.1.2-2ubuntu2 NFS support files common to client and serve

$ time sudo mount.nfs platinum:/zpool1/vmware tmp
mount.nfs: Connection timed out

real 2m5.071s
user 0m0.000s
sys 0m0.004s

----------------------- Gutsy test case -------------------------
$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=7.10
DISTRIB_CODENAME=gutsy
DISTRIB_DESCRIPTION="Ubuntu 7.10"

$ uname -a
Linux campaigndialer 2.6.22-14-server #1 SMP Tue Dec 18 08:31:40 UTC 2007 i686 GNU/Linux

$ dpkg -l | grep nfs-common
ii nfs-common 1:1.1.1~git-20070709-3ubuntu1 NFS support files common to client and serve

$ time sudo mount.nfs platinum:/zpool1/vmware tmp
mount.nfs: platinum:/zpool1/vmware failed, reason given by server: Permission denied

real 0m0.016s
user 0m0.000s
sys 0m0.010s

mount server:/export /import/ -o
context=system_u:object_r:httpd_sys_content_t:s0,fscontext=system_u:object_r:lib_t:s0

With selinux enforcing this will give you an selinux denial and the actual mount
syscall returns -EACCES. mount.nfs sleeps (increasingly longer up to 10
seconds) and keeps retrying. One permission denied should be enough to cause
mount.nfs to give up.

A bit of an strace is below:
2262 mount("paris.rdu.redhat.com:/export/storage", "/storage", "nfs"..., 0,
"context=\"system_u:object_r:httpd_sys_content_t:s0\",fscontext=\"system_u:object_r:lib_t:s0\",addr=10.11.231.241"...)
= -1 EACCES (Permission denied)
2262 gettimeofday({1206974842, 167191}, NULL) = 0
2262 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
2262 rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8) = 0
2262 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
2262 nanosleep({1, 0}, {1, 0}) = 0
2262 mount("paris.rdu.redhat.com:/export/storage", "/storage", "nfs"..., 0,
"context=\"system_u:object_r:httpd_sys_content_t:s0\",fscontext=\"system_u:object_r:lib_t:s0\",addr=10.11.231.241"...)
= -1 EACCES (Permission denied)
2262 gettimeofday({1206974843, 206725}, NULL) = 0
2262 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
2262 rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8) = 0
2262 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
2262 nanosleep({2, 0}, {2, 0}) = 0
2262 mount("paris.rdu.redhat.com:/export/storage", "/storage", "nfs"..., 0,
"context=\"system_u:object_r:httpd_sys_content_t:s0\",fscontext=\"system_u:object_r:lib_t:s0\",addr=10.11.231.241"...)
= -1 EACCES (Permission denied)

Correction, after 2 minutes of retrying it will give up. Until recently this
will going up immediately.

I speak engrish goodly.

"Until recently this would give up immediately"

Created attachment 299729
patch (untested)

This patch should fix it...

The patch works. With it I get:

[root@dhcp231-25 ~]# mount paris.rdu.redhat.com:/export/storage /storage/ -o
context=system_u:object_r:httpd_sys_content_t:s0,fscontext=system_u:object_r:lib_t:s0
mount.nfs: access denied by server while mounting
paris.rdu.redhat.com:/export/storage

Which is not technically true. The server didn't deny my, the local selinux
policy denied me. But I'm happy with it anyway....

So why was SELunix denying the mount?

In this particular case its because i told it to do something bogus with the
mount options. The SELinux denial from the above line is expected and was
encountered when I was working on doing so SELinux/NFS testing for other
purposes. Historically mount.nfs would kick it back at me instantly but not any
more. This is not by any means common, a much more reasonable failure scenario
(and one that happened under binary mount options) would be:

mount server:/export /import -o context=system_u:object_r:httpd_t:s0

This is invalid but not nearly as strange and contrived as my previous mount
command and the user should get EACCES, but instead, after 2 minutes they get:

mount.nfs: Connection timed out

which does tell the real problem. Even if we let it retry on EACCES for 2
minutes it needs to fail with EACCES. If it pretends it was a timeout problem
it will send the user looking at non permission related problems...

felix68 (felix-dimedis) wrote :

Exactly the same problem over here.

And I also believe, that this bug is critical.

I even changed to xfce4, because I thought gnome was broken. (nautilus accessed a host with one (of five) "permission denied" nfs shares. This resulted in a gnome desktop, where nautilus started 2 minutes later, than the rest. And in a gnome desktop, where every 10 minutes (after the automounter automatically umounted the shares), I experienced 2 minute timeouts, when opening local(!) folders.)

Felix

Fixed in nfs-utils-1.1.2-2

Steve Langasek (vorlon) wrote :

This issue is not reproducible for me. If I try to mount a Linux NFS share that I don't have permission for, I consistently get an immediate "permission denied" returned.

If you're seeing connection timeouts, that looks to me like a firewall issue. Can you provide a network trace of the NFS traffic in the case of this failure?

Changed in nfs-utils:
status: New → Incomplete
felix68 (felix-dimedis) wrote :

Local network.
Second PC with Ubuntu 7.10 works as expected (connected to the same switch).

This is the /etc/exports of the nfs server:

/mega/one *.somedomain.de(ro,async,subtree_check) apple.somedomain.de(ro,async,insecure,subtree_check)
/mega/two *.somedomain.de(ro,async,subtree_check) apple.somedomain.de(ro,async,insecure,subtree_check)
/mega/three *.somedomain.de(rw,async,subtree_check)
/mega/four *.somedomain.de(ro,async,subtree_check)
/home/someone/data wizard.somedomain.de(rw,async,subtree_check)

I am able to (auto)mount all the shares without problem.
Only /home/someone/data (which normally should result (and did before some dist-upgrade one week ago on the client) in a permission denied produces the timeout.

Nevertheless find attached some tcpdump of the client after manually
mount ouzo:/home/someone/data/ /mnt/stick/
mount.nfs: Connection timed out

nfs-common 1:1.1.2-2ubuntu2
nfs-server 1:1.1.1~git-20070709-3ubuntu1

A colleague (who uses Debian Sid) got a similiar error since a few days.
( Debian SID nfs-common 1:1.1.2-2)

Lonnie Olson (fungussn) wrote :

Not a firewall issue. I can fix the timeout simply by granting permission on the NFS server.

Here is a testing scenario.

== Permission Denied ==
Linux NFS server
exportfs localhost:/home/olsonl
/home/olsonl localhost(ro,wdelay,root_squash)

Hardy NFS client
sudo mount s12:/home/olsonl /mnt
mount.nfs: Connection timed out
(see attached packet trace)

Gutsy NFS client
sudo mount s12:/home/olsonl tmp
mount.nfs: s12:/home/olsonl failed, reason given by server: Permission denied

== Same server and clients, granted permission ==
the very same Linux NFS server
exportfs fungus:/home/olsonl
exportfs gutsy:/home/olsonl
/home/olsonl fungus.sosstaffing.com(ro,wdelay,root_squash)
/home/olsonl gutsy.sosstaffing.com(ro,wdelay,root_squash)

Hardy NFS client
sudo mount s12:/home/olsonl /mnt
<succeeded instantly>

Gutsty NFS client
sudo mount s12:/home/olsonl tmp
<succeeded instantly>

This is very clearly not a firewall issue. There is no firewall or NAT at all between these points.

Lonnie Olson (fungussn) wrote :

Here is the capture for the Gutsy permission denied

This could be related to bug #221538

felix68 (felix-dimedis) wrote :

And this is definitely the same bug:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=476128

Sebastien Bacher (seb128) wrote :

confirmed since there is a duplicate and a patch available

Changed in nfs-utils:
importance: Undecided → High
milestone: none → ubuntu-8.04.1
status: Incomplete → Confirmed
Ludwik Trammer (ludwik) wrote :

Is it a way to check if this bug and bug #221538 are manifestations of the same problem (the symptoms are basically the same, but this one happens when the access to the file is not permitted, and the other one when the file is not found. Both started in Gutsy) and maybe merge them into one report?

Anyway, that's a huge problem for institutions who use NFS mounted /home folders (for single sign-on) like High School I'm working for. The desktop machines at first seem to work, but are extremely unstable (something hangs all the time).

Steve Langasek (vorlon) wrote :

Ludwik, thanks for pointing out bug #221538, which I've now marked as a duplicate of this one. I can reproduce the bug now (not sure why I couldn't before!), and the referenced Red Hat bug, https://bugzilla.redhat.com/show_bug.cgi?id=439807, appears to include the fix for this problem.

Changed in nfs-utils:
milestone: ubuntu-8.04.1 → none
importance: Undecided → High
milestone: none → ubuntu-8.04.1
status: New → Confirmed
Steve Langasek (vorlon) wrote :

uploaded to -proposed; SRU team, please review.

Changed in nfs-utils:
status: Confirmed → In Progress
Steve Langasek (vorlon) wrote :

for intrepid, this fix is now available for merge in Debian unstable, in nfs-utils 1:1.1.2-4.

Daniel Kopelove (kopelovd) wrote :

Hi all,

I'm new to the bug-reporting and fixing process. Steve, since the bug exists on other distributions as well, will the ubuntu team roll a new nfs-common package with a fix, or is it up to sys admins to patch nfs-utils ourselves?

Thanks,
-Dan

Steve Langasek (vorlon) wrote :

Daniel,

I'm not sure what you mean by "other distributions" - Ubuntu 8.04 is the only version of Ubuntu that should be affected by this issue since it's the only version which includes a Linux kernel 2.6.23 or later, and the Ubuntu developers aren't responsible for fixes to other Linux distributions?

The bug is on track to be fixed in an update to Ubuntu 8.04.

Martin Pitt (pitti) wrote :

Accepted into -proposed, please test and give feedback here

Changed in nfs-utils:
milestone: ubuntu-8.04.1 → none
status: In Progress → Fix Committed
Ludwik Trammer (ludwik) wrote :

Sorry for asking a dump question, but how do you access "-proposed"?

Ludwik Trammer [2008-05-08 9:42 -0000]:
> Sorry for asking a dump question, but how do you access "-proposed"?

You can enable it in System -> Administration -> Software Sources ->
Updates.

Steve Langasek (vorlon) on 2008-05-08
Changed in nfs-utils:
milestone: none → ubuntu-8.04.1
Daniel Kopelove (kopelovd) wrote :

update has fixed the issue.

Thank you!

Martin Pitt (pitti) wrote :

Copied to hardy-updates.

Changed in nfs-utils:
assignee: nobody → vorlon
status: Fix Committed → Fix Released

Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Martin Pitt (pitti) wrote :

Should be fixed in Intrepid soon, in order to not loose fixes from Hardy SRUs.

Changed in nfs-utils:
milestone: none → intrepid-alpha-2
Martin Pitt (pitti) wrote :

Copied hardy-updates to intrepid.

Changed in nfs-utils:
milestone: intrepid-alpha-2 → none
status: Confirmed → Fix Released
MarianoAbsatz (el-baby) wrote :

Hi,

I understand that this has been fixed in nfs-comon 1:1.1.2-4, however, the version in hardy-updates is 1:1.1.2-2ubuntu2.1

I think I have the same problem and I am already using 1:1.1.2-2ubuntu2.1 and symptoms are still here...

FWIW, I don't see any nfs package in hardy-proposed right now (2008-06-26).

Any hint appreciated.

Rocketero (rocketero2006) wrote :

At 2008-08-24 I report I have the same issue.
I have installed the following packages: samba, smbfs, nfs-common (1.1.1.2-2ubuntu2.1), and still getting: "mount error 13 = Permission denied" when I issue the command: mount -t cifs (or smbfs) //hostname/sharename /home/ubuntu/folder -o username=xxx,password=xxx

Any help is appreciated.

I have a livecd for knoppix and I dont have this problem but I like ubuntu best

Steve Langasek (vorlon) wrote :

Rocketero,

The bug report you've followed up to is about NFS mounts, not CIFS mounts; your problem is unrelated to this bug report. Please file a new report against the samba package, or alternatively, you might try asking for help in the ubuntu forums since I'm sure that CIFS in Ubuntu 8.04 is generally working as expected (having extensively tested it myself).

Michael Shadle (mshadle) wrote :

I am having an issue I do not know if this is related, but I have a server that uses NFS lightly, it has just broken now after an upgrade to intrepid.

I have 6 client machines to a FreeBSD NFS server running NFS v3 over TCP.

After an hour or two, NFS just starts hanging and cannot recover. I can umount -l /home, but it will never let me remount. Can't find anything in the client or server logs that is relevant as to why.

My dpkg list is identical on all 6 servers, all the configs I compared are identical too (well besides for server specific info) so I don't know what could be the reason why only 1 of the 6 seems to be having the issue. I downgraded the kernel from linux-image-2.6.27-10-server to linux-image-2.6.27-7-server, still doesn't matter.

I'll pay someone to figure this out ASAP. This is wrecking my hosting business right now... mike503 AT gmail.com

Stefan Pappalardo (sjuk) wrote :

I have the same Problem in Intrepid. Mounting with "sudo mount -a" is all ok.
But on boot mount of nfs shares hangs for a while, and gives up.

This message is a reminder that Fedora 9 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 9. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora
'version' of '9'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version prior to Fedora 9's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we may not be able to fix it before Fedora 9 is end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora please change the 'version' of this
bug to the applicable version. If you are unable to change the version,
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Fedora 9 changed to end-of-life (EOL) status on 2009-07-10. Fedora 9 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Changed in fedora:
status: Fix Committed → Won't Fix
Changed in fedora:
importance: Unknown → Low
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.