nfsver=3,udp not working anymore

[Jörg Hänsel]

Package: nfs-kernel-server
Architecture: amd64
Version: 1:2.6.1-1~exp1ubuntu1

nfsversion 3 with proto udp seems not to be supported by the nfs-kernel-server anymore.

In Ubuntu 18.04 (nfs-kernel-server 1:1.3.4-2.1ubuntu5) it was still possible. I could mount with either tcp or udp:

1. tcp
mount -o nfsvers=3 192.168.0.1:/srv/nfs/testshare /mnt

mount|grep mnt:

192.168.0.1:/srv/nfs/testshare on /mnt type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.1,mountvers=3,mountport=59808,mountproto=udp,local_lock=none,addr=192.168.0.1)

2. udp
mount -o nfsvers=3,udp 192.168.0.1:/srv/nfs/testshare /mnt

mount|grep mnt:

192.168.0.1:/srv/nfs/testshare on /mnt type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.0.1,mountvers=3,mountport=59808,mountproto=udp,local_lock=none,addr=192.168.0.1)

Now in Ubuntu 22.04 I get the message:

mount.nfs: an incorrect mount option was specified

I tried to enable upd in /etc/nfs.conf in section [nfsd]:

[nfsd]
# debug=0
# threads=8
# host=
# port=0
# grace-time=90
# lease-time=90
# udp=n
udp=y
# tcp=y
# vers2=n
# vers3=y

But there was no effect.

nfs3 with udp is needed by several busybox mount binaries in initrds of Knoppix and other live distributions used via PXE boot.

Best regards
Jörg

[Andreas Hasenack (ahasenack)]

Taking a look at this, most curious:

# mount localhost:/storage /mnt -o vers=3,proto=udp
mount.nfs: an incorrect mount option was specified
root@j-nfs-dep8:~# mount localhost:/storage /mnt -v -o vers=3,proto=udp
mount.nfs: timeout set for Tue Mar 8 14:59:11 2022
mount.nfs: trying text-based options 'vers=3,proto=udp,addr=127.0.0.1'
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: trying 127.0.0.1 prog 100003 vers 3 prot UDP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 127.0.0.1 prog 100005 vers 3 prot UDP port 58825
mount.nfs: mount(2): Invalid argument
mount.nfs: an incorrect mount option was specified

# rpcinfo -p
   program vers proto port service
    100000 4 tcp 111 portmapper
    100000 3 tcp 111 portmapper
    100000 2 tcp 111 portmapper
    100000 4 udp 111 portmapper
    100000 3 udp 111 portmapper
    100000 2 udp 111 portmapper
    100024 1 udp 44661 status
    100024 1 tcp 40383 status
    100005 1 udp 45269 mountd
    100005 1 tcp 58873 mountd
    100005 2 udp 44744 mountd
    100005 2 tcp 41675 mountd
    100005 3 udp 58825 mountd
    100005 3 tcp 60591 mountd
    100003 3 tcp 2049 nfs
    100003 4 tcp 2049 nfs
    100227 3 tcp 2049
    100003 3 udp 2049 nfs
    100227 3 udp 2049
    100021 1 udp 33771 nlockmgr
    100021 3 udp 33771 nlockmgr
    100021 4 udp 33771 nlockmgr
    100021 1 tcp 34605 nlockmgr
    100021 3 tcp 34605 nlockmgr
    100021 4 tcp 34605 nlockmgr

[Andreas Hasenack (ahasenack)]

Using focal as a server, jammy still cannot mount an export using udp, but the other way around works: focal client can mount nfs share from jammy using udp. So this seems a client problem.

[Andreas Hasenack (ahasenack)]

I restored nfs-utils 1.x in the jammy client, and tried to mount again. It still fails. I don't think this is an nfs-utils problem:

ubuntu@j-nfs-dep8:~$ dpkg -l|grep nfs
ii libnfsidmap2:amd64 0.25-6build1 amd64 NFS idmapping library
ii nfs-common 1:1.3.4-6ubuntu1 amd64 NFS support files common to client and server
ubuntu@j-nfs-dep8:~$ sudo -i
root@j-nfs-dep8:~# mount f1:/storage /mnt -o udp -v
mount.nfs: timeout set for Tue Mar 8 17:54:59 2022
mount.nfs: trying text-based options 'udp,vers=4.2,addr=192.168.122.197,clientaddr=192.168.122.61'
mount.nfs: mount(2): Invalid argument
mount.nfs: trying text-based options 'udp,vers=4.1,addr=192.168.122.197,clientaddr=192.168.122.61'
mount.nfs: mount(2): Invalid argument
mount.nfs: trying text-based options 'udp,vers=4.0,addr=192.168.122.197,clientaddr=192.168.122.61'
mount.nfs: mount(2): Invalid argument
mount.nfs: trying text-based options 'udp,addr=192.168.122.197'
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: trying 192.168.122.197 prog 100003 vers 3 prot UDP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.122.197 prog 100005 vers 3 prot UDP port 51873
mount.nfs: mount(2): Invalid argument
mount.nfs: an incorrect mount option was specified

[Andreas Hasenack (ahasenack)]

Found this in /boot/config-$(uname -r):

CONFIG_NFS_DISABLE_UDP_SUPPORT=y

[Andreas Hasenack (ahasenack)]

This was first disabled on groovy, the first ubuntu release after focal:

root@g1:~# grep NFS /boot/config-5.8.0-59-generic |grep UDP
CONFIG_NFS_DISABLE_UDP_SUPPORT=y

The upstream help text next to that kernel option says:
"""
Choose Y here to disable the use of NFS over UDP. NFS over UDP
on modern networks (1Gb+) can lead to data corruption caused by
fragmentation during high loads.
"""

I'll leave it to the kernel team to comment further.

[Jörg Hänsel (joerg-haensel)]

I just updated linux-image-5.15.0-18-generic to today's new version linux-image-5.15.0-22-generic.

Now with enabling "udp=y" in /etc/nfs.conf and the new kernel version I can mount
the nfs-share on the jammy server from the koppix pxe initdrd (busybox mount) with nfs version 3 and udp.

Only mounting jammy server from jammy client is still not possible with nfs3+udp.

The diff of /boot/config-5.15.0-18-generic and /boot/config-5.15.0-22-generic shows no hints of NFS related changes in the kernel config.

[Andreas Hasenack (ahasenack)]

I added this to the Jammy 22.04 release notes at https://discourse.ubuntu.com/t/jammy-jellyfish-release-notes/24668

If it's decided to flip this option before release, please remove that release notes remark.

[Andreas Hasenack (ahasenack)]

Are you sure it's using udp? Did you confirm with /proc/mounts or mount output? I didn't get it to work, not even with a focal server, and the kernel option disabling UDP sounds very adamant.

[Jörg Hänsel (joerg-haensel)]

I am currently very confused. I restored the snapshots of my test VMs from yesterday and now I can mount the NFS-shares on the jammy server on all clients (Knoppix, Ubuntu 18.04) with nfs3+udp after putting "udp=y" in /etc/nfs.conf, also with the last kernel version 5.15.0-18.

Yesterday in all my tests the udp=y param was ignored, only the vers=? params had an effect after restarting the nfs-kernel-server.
My only explanation for now is a problem with the internal network between my VirtualBox Test VMs. Sorry, I know this sounds very strange.

But despite of the kernel config option CONFIG_NFS_DISABLE_UDP_SUPPORT=y I can mount the NFS-shares on the jammy server with nfs3+udp from every test client VM. /proc/mounts shows: vers=3 and proto=udp

Only on the server itself I cannot mount with nfs3+udp.

mount -o vers=3,udp 192.168.0.1:/srv/nfs/testshare /mnt

leads to
mount.nfs: an incorrect mount option was specified

but
mount -o vers=3,tcp 192.168.0.1:/srv/nfs/testshare /mnt

is working.

So conclusion for me now. The /etc/nfs.conf params are working correctly and my nfs3+udp problem is solved.

But nfs3+udp should not be possible to be enabled with respect to the kernel param.
This disabled NFS3 support seems only to affect the client part of the kernel-module.

[Andreas Hasenack (ahasenack)]

> This disabled NFS3 support seems only to affect the client part of the kernel-module.

NFSv3 is fine, it's udp that is disabled in the kernel (I think you understood that, and above is just a typo, but clarifying nonetheless).

Clients running a kernel with CONFIG_NFS_DISABLE_UDP_SUPPORT=y should not be able to mount NFS exports using udp, that's my understanding. I believe it can still *export* (aka, server) nfs via udp.

[Jörg Hänsel (joerg-haensel)]

yes, this was a typo... of course I meant disabled UDP NFS mounting.
And regarding the 2nd point we have the same understanding.

So since "old" clients are still able to mount the NFS shares via nfs3 over udp on jammy servers, I would not flip the kernel config CONFIG_NFS_DISABLE_UDP_SUPPORT back to "n" and leave it as it is:
CONFIG_NFS_DISABLE_UDP_SUPPORT=y

thank you very much for clarifying!

[jferguson (jferg977)]

I've encountered same problem after 22.04 upgrade. I want to mount shares from SPARCstation 10 based SunOS 4.1.4 system on the 22.04 system but cannot. It does work in the other direction, ie. I can mount 22.04 shares on the Sun.

If you could confirm my understanding that without recompiling the Kernel to remove the nfs-udp load block, I will not be able to do anything to restore the sunos nfs udp export loading on the 22.04 system, it would save me a lot of time messing with this.

I understand from what you've written above that providing nfs udp service opens a security risk.

[Jörg Hänsel (spiderbaby)]

yes, I confirm. You cannot mount nfs3 shares over udp served by your SPARCstation with 22.04 as client running the standard kernel due to the option CONFIG_NFS_DISABLE_UDP_SUPPORT=y.

I don't know what would be necessary to add nfs3-udp to the kernel. Maybe changing CONFIG_NFS_DISABLE_UDP_SUPPORT=n and recompiling will work, but there could also be options necessary to be changed, too.

[jferguson (jferg977)]

Thanks much,

[Zeke Farwell (ezekielf)]

I've run into this issue when using NFS mounts via Vagrant which uses NFSv3 and UDP by default. Relevant issues:
https://github.com/hashicorp/vagrant/issues/12758
https://github.com/hashicorp/vagrant/issues/13078
Vagrant can be forced to use TCP to work around the issue, but UDP is preferred due to being faster than TCP for this particular use case.
https://developer.hashicorp.com/vagrant/docs/synced-folders/nfs#nfs-synced-folder-options
Looks like if I want to keep using UDP I'll need to stick with 20.04 for now, but it would be nice if it weren't disabled in 22.04.

[Chris (onyxdrumz)]

> Looks like if I want to keep using UDP I'll need to stick with 20.04 for now, but it would be nice if it weren't disabled in 22.04.

Yea, same here. I've ended up here trying to update to 22.04 but not that big of a need, just trying to be proactive.