mount.nfs: fix version negotiation laddering with parameters '-t nfs4' or '-o vers=4'

[Shengxue Pan]

[Impact]

Calling mount.nfs with parameters '-t nfs4' or '-o vers=4' will default to attempting to connect with version 4.0 only, and if the server does not support 4.0, the mount will fail.

The expected behaviour is first default to negotiating version 4.2, and if that fails, then 4.1, and then 4.0, before raising an error.

This laddering approach was specified and agreed upon in:

commit f980298853d9b12a222421342418732f65883c30
Author: Benjamin Coddington <email address hidden>
Date: Mon Dec 8 15:51:07 2014 -0500
Subject: mount.nfs: configurable minor version defaults
Link: https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=f980298853d9b12a222421342418732f65883c30

first introduced in nfs-utils in 2014, in version 1.3.2-rc6, and is included in the version offered in focal.

This behaviour works when you don't specify a version at all, or even specify nfs at all to the mount command.

A graphical explanation of the laddering approach is:

    ------------------
    | nfsmount.conf |-----------------------------------
    | Defaultvers= | arg option | attempts: |
    |---------------------------------------------------|
    | 4.2 | not set | v4.2 v4.1 v4.0 v3 |
    | 4.2 | v4 | v4.2 v4.1 v4.0 |
    | 4.1 | not set | v4.1 v4.0 v3 |
    | 4.1 | v4 | v4.1 v4.0 |
    | 4 | not set | v4.0 v3 |
    | 4 | v4 | v4.0 |
    | 3 | not set | v3 |
    | any set | v4.2 | v4.2 |
    | any set | v4.1 | v4.1 |
    | any set | v4 | v4.0 |
    | any set | v3 | v3 |
    | not set | not set | v4.2 v4.1 v4.0 v3 |
    -----------------------------------------------------

This should also apply to using the command line arguments '-t nfs4' or '-o vers=4' matching existing and current upstream designs.

[Testcase]

Create two VMs, one jammy and one focal.

The jammy VM will be the server.

Server VM:
$ sudo hostnamectl set-hostname jammy-nfs-server
$ sudo apt update && sudo apt upgrade -y
$ sudo apt install nfs-kernel-server
$ sudo mkdir /export
$ sudo mkdir /export/users
$ sudo mkdir /home/users
$ sudo vi /etc/fstab # add the following line:
/home/users /export/users none bind 0 0
$ sudo mount -a
$ sudo vi /etc/exports # add the following lines:
/export 192.168.122.0/24(rw,fsid=0,no_subtree_check,sync)
/export/users 192.168.122.0/24(rw,nohide,insecure,no_subtree_check,sync)
$ sudo vi /etc/nfs.conf # Under [nfsd], uncomment / change the following lines:
vers2=n
vers3=n
#vers4=y
vers4.0=n
vers4.1=y
vers4.2=y
$ sudo systemctl restart nfs-server.service
$ sudo cat /proc/fs/nfsd/versions # confirm nfs 4.0 is not allowed
-2 -3 +4 -4.0 +4.1 +4.2

Focal VM:
$ sudo hostnamectl set-hostname focal-nfs-client
$ sudo apt update && sudo apt upgrade -y
$ sudo apt install nfs-common

And then try to mount using '-o vers=4':

$ sudo mount -o vers=4 -vvv jammy-nfs-server:/ /mnt

You will see:

$ sudo mount -o vers=4 -vvv jammy-nfs-server:/ /mnt
mount.nfs: timeout set for Fri Jan 19 00:44:42 2024
mount.nfs: trying text-based options 'vers=4,addr=192.168.122.13,clientaddr=192.168.122.106'
mount.nfs: mount(2): Protocol not supported
mount.nfs: Protocol not supported

Test packages are available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf377082-test

If you install the test package, you will see:

$ sudo mount -o vers=4 -vvv jammy-nfs-server:/ /mnt
mount.nfs: timeout set for Fri Jan 19 00:46:09 2024
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.122.13,clientaddr=192.168.122.106'
$ ll /mnt
total 12
drwxr-xr-x 3 root root 4096 Jan 17 22:30 ./
drwxr-xr-x 19 root root 4096 Jan 19 00:41 ../
drwxr-xr-x 2 root root 4096 Jan 17 22:30 users/

If you disable v4.2 on the jammy nfs server VM, and try and reconnect, we see
the laddering to 4.1:

$ sudo mount -o vers=4 -vvv jammy-nfs-server:/ /mnt
mount.nfs: timeout set for Fri Jan 19 00:53:43 2024
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.122.13,clientaddr=192.168.122.106'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'vers=4.1,addr=192.168.122.13,clientaddr=192.168.122.106'
:~$ ll /mnt
total 12
drwxr-xr-x 3 root root 4096 Jan 17 22:30 ./
drwxr-xr-x 19 root root 4096 Jan 19 00:41 ../
drwxr-xr-x 2 root root 4096 Jan 17 22:30 users/

[Where problems could occur]

We are changing the default protocol negotiation code for mount.nfs, so specifying '-t nfs4' or '-o vers=4' will no longer default to 4.0, instead, it will now attempt 4.2, 4.1, 4.0 in a laddering fashion. This may cause a different NFS version to be selected on mount, if the fstab or shell scripts system administers use '-t nfs4' or '-o vers=4' instead of specifying a specific version.

e.g. 4.2 may be selected instead of 4.0 used previously. This could have an impact on workloads if there are any regressions in switching to 4.2 from 4.0 etc. Network traffic may be different as newer protocols are used.

The dependency commit "mount.nfs: improve version negotiation when vers=4 is specified." also fixed a few bugs, with the most impactful being that in some cases mount.nfs would downgrade from v4 to v3, even when v4 was explicity asked for, if a particular namespace was available in v3, but not in v4, e.g. by fsid=0.

If a regression were to occur, users could have trouble mounting NFS v4.x shares as a client.

NFS server has no changes.

[Other info]

This was fixed in nfs-utils 2.1.2-rc4 by the following commits:

commit d406648690fa0fdf5333d7d54cf8210dab7f4d9c
Author: NeilBrown <email address hidden>
Date: Thu Jun 1 09:42:16 2017 -0400
Subject: mount.nfs: improve version negotiation when vers=4 is specified.
Link: https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=d406648690fa0fdf5333d7d54cf8210dab7f4d9c

commit 62a4d95854e5cda4b772fa132cbd16c4429412c8
Author: Steve Dickson <email address hidden>
Date: Tue Jun 13 12:00:39 2017 -0400
Subject: mount.nfs: Use default minor version when -t nfs4 is specified
Link: https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=62a4d95854e5cda4b772fa132cbd16c4429412c8

commit 90790d3129cf6f5fe095cf01e37d2d0a89d8dbec
Author: Steve Dickson <email address hidden>
Date: Mon Jun 19 11:19:55 2017 -0400
Subject: mount.nfs: Use default minor version when -o v4 is specified
Link: https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=90790d3129cf6f5fe095cf01e37d2d0a89d8dbec

These are found in Jammy and later.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matthew Ruffell (mruffell)]

Attached is a debdiff that solves this issue.

[Heitor Alves de Siqueira (halves)]

Hi Matthew,

thank you for the detailed description! Patches look good, and seem to be clean cherry-picks on top of Focal.
I've tested your changes and they seem to be working correctly, autopkgtest looks good as well.

I cleaned up some trailing whitespace on your changelog entry, but otherwise left it untouched. Sponsored for Focal, thank you!

[Timo Aaltonen (tjaalton)]

Hello Shengxue, or anyone else affected,

Accepted nfs-utils into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nfs-utils/1:1.3.4-2.5ubuntu3.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Matthew Ruffell (mruffell)]

Performing verification for Focal.

I set up a Jammy VM to be the nfs server, using the instructions in the SRU template.

$ sudo cat /proc/fs/nfsd/versions
-2 -3 +4 -4.0 +4.1 +4.2

4.0 is not allowed, but 4.1 and 4.2 are.

I then started a fresh Focal VM, and installed nfs-common 1.3.4-2.5ubuntu3.5 from -updates.

I then tried to mount the NFS share:

$ sudo mount -o vers=4 -vvv jammy-nfs-server:/ /mnt
mount.nfs: timeout set for Mon Feb 26 01:21:35 2024
mount.nfs: trying text-based options 'vers=4,addr=192.168.122.17,clientaddr=192.168.122.19'
mount.nfs: mount(2): Protocol not supported
mount.nfs: Protocol not supported

as the bug report describes, when we specify -o vers=4 on the mount command line, it only tries 4.0 and no other version.

I then enabled -proposed and installed nfs-common 1.3.4-2.5ubuntu3.6.

Trying the mount again:

$ sudo mount -o vers=4 -vvv jammy-nfs-server:/ /mnt
mount.nfs: timeout set for Mon Feb 26 01:27:47 2024
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.122.17,clientaddr=192.168.122.19'
$ ll /mnt
total 12
drwxr-xr-x 3 root root 4096 Feb 26 01:10 ./
drwxr-xr-x 19 root root 4096 Feb 26 01:13 ../
drwxr-xr-x 2 root root 4096 Feb 26 01:10 users/

This time it correctly tries version 4.2, and since 4.2 is allowed, the mount succeeds.

To test the laddering, I went to the nfs-server and then edited /etc/nfs.conf and changed:

vers4.2=n

and restarted nfs-server.service.

Back on the Focal VM, we now see:

$ sudo mount -o vers=4 -vvv jammy-nfs-server:/ /mnt
mount.nfs: timeout set for Mon Feb 26 01:30:40 2024
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.122.17,clientaddr=192.168.122.19'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'vers=4.1,addr=192.168.122.17,clientaddr=192.168.122.19'
$ ll /mnt
total 12
drwxr-xr-x 3 root root 4096 Feb 26 01:10 ./
drwxr-xr-x 19 root root 4096 Feb 26 01:13 ../
drwxr-xr-x 2 root root 4096 Feb 26 01:10 users/

The laddering works, and we try version 4.2, then 4.1 which succeeds.

Lets also turn off 4.1 on the server, and enable 4.0:

vers4.0=y
vers4.1=n

now on the client, we see:

$ sudo mount -o vers=4 -vvv jammy-nfs-server:/ /mnt
mount.nfs: timeout set for Mon Feb 26 01:32:53 2024
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.122.17,clientaddr=192.168.122.19'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'vers=4.1,addr=192.168.122.17,clientaddr=192.168.122.19'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'vers=4.0,addr=192.168.122.17,clientaddr=192.168.122.19'
$ ll /mnt
total 12
drwxr-xr-x 3 root root 4096 Feb 26 01:10 ./
drwxr-xr-x 19 root root 4096 Feb 26 01:13 ../
drwxr-xr-x 2 root root 4096 Feb 26 01:10 users/

We laddered all the way down to 4.0 and connected, since we just allowed it.

This works great. The package in -proposed fixes the issue. Happy to mark verified for Focal.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nfs-utils - 1:1.3.4-2.5ubuntu3.6

---------------
nfs-utils (1:1.3.4-2.5ubuntu3.6) focal; urgency=medium

  * Fix default protocol version negotiation when NFSv4 is specified
    on the command line with '-t nfs4' or '-o vers=4', such that it
    actually follows the previously agreed-upon laddering approach,
    of trying 4.2, then 4.1, then 4.0 before raising an error.
    These patches also fix an issue where asking for a v4 mount could
    result in a v3 mount if the namespace was available in v3, but
    not in v4, such as with fsid=0. (LP: #2049262)
    - d/p/lp-2049262-1-mount.nfs-improve-version-negotiation-when-vers-4-i.patch
    - d/p/lp-2049262-2-mount.nfs-Use-default-minor-version-when-t-nfs4-is.patch
    - d/p/lp-2049262-3-mount.nfs-Use-default-minor-version-when-o-v4-is-sp.patch

 -- Matthew Ruffell <email address hidden> Thu, 18 Jan 2024 22:29:59 +1300

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for nfs-utils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Andreas Hasenack (ahasenack)]

A regression on this update for focal was reported in https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2056549