mount.nfs: sloppy options processing is broken on 5.15 HWE kernel

[Matthew Ruffell]

[Impact]

There was a new mount API introduced to the nfs subsystem in the 5.6 kernel release, that changed how parameters should be passed from userspace mount.nfs to the kernel.

This broke processing of the sloppy '-s' and '-o' options, where if you have an invalid mount option, or the ordering of mount options was incorrect, the mount would fail instead of going through.

The fix is to move the sloppy options to the front of the options list, and then passing the rest of the options to the kernel, so the sloppy operator can be parsed correctly.

[Testcase]

Start two Focal VMs, one for the server, and one for the client.

On the Server:
1) sudo apt update
2) sudo apt install nfs-kernel-server
3) sudo mkdir -p /mnt/nfs_share
4) sudo chown -R nobody:nogroup /mnt/nfs_share/
5) sudo chmod 777 /mnt/nfs_share/
6) echo "/mnt/nfs_share 192.168.122.0/24(rw,sync,no_subtree_check)" | sudo tee /etc/exports
7) sudo exportfs -a
8) sudo systemctl restart nfs-kernel-server
9) echo "Testfile" | tee /mnt/nfs_share/testfile

On the client:
1) sudo apt update
2) sudo apt install nfs-common
3) sudo mkdir -p /mnt/nfs_clientshare
4) sudo mount -t nfs 192.168.122.114:/mnt/nfs_share /mnt/nfs_clientshare -s -o invalid,intr

On the client with 5.4 kernel:

$ uname -rv
5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023
$ sudo mount -t nfs 192.168.122.114:/mnt/nfs_share /mnt/nfs_clientshare -s -o invalid,intr
$ cd /mnt/nfs_clientshare/
/mnt/nfs_clientshare$ ll
total 12
drwxrwxrwx 2 nobody nogroup 4096 Jun 29 01:13 ./
drwxr-xr-x 3 root root 4096 Jun 29 01:11 ../
-rw-rw-r-- 1 ubuntu ubuntu 9 Jun 29 01:13 testfile
/mnt/nfs_clientshare$ cat testfile
Testfile

On the client with the 5.15 HWE kernel:

$ uname -rv
5.15.0-76-generic #83~20.04.1-Ubuntu SMP Wed Jun 21 20:23:31 UTC 2023
$ sudo mount -t nfs 192.168.122.114:/mnt/nfs_share /mnt/nfs_clientshare -s -o invalid,intr
mount.nfs: an incorrect mount option was specified

There are test packages in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf352441-test

If you install them, when using the HWE kernel, you will see:

$ uname -rv
5.15.0-76-generic #83~20.04.1-Ubuntu SMP Wed Jun 21 20:23:31 UTC 2023
$ sudo mount -t nfs 192.168.122.114:/mnt/nfs_share /mnt/nfs_clientshare -s -o invalid,intr
$ cd /mnt/nfs_clientshare/
/mnt/nfs_clientshare$ ll
total 12
drwxrwxrwx 2 nobody nogroup 4096 Jun 29 01:13 ./
drwxr-xr-x 3 root root 4096 Jun 29 01:11 ../
-rw-rw-r-- 1 ubuntu ubuntu 9 Jun 29 01:13 testfile
/mnt/nfs_clientshare$ cat testfile
Testfile

[Where problems could occur]

We are changing how options parameters are parsed. If a regression were to occur, user's nfs mount commands might fail, which has the possibility of breaking nfs-clients. If a system is set to mount a share on boot, a share might not be available without manual intervention.

Users should not have to modify their mount commands at all. But if a regression were to occur, sysadmins may need to change to the ordering of their options.

This does not change nfs-servers, only nfs-clients.

[Other Info]

This issue was fixed in 2.6.1 of nfs-utils with the following commits:

commit 92b664ef4f25f1bd920bea4db435345629543353
From: Jianhong Yin <[e-mail]>
Date: Thu, 10 Jun 2021 13:27:29 -0400
Subject: mount.nfs: insert 'sloppy' at beginning of the options
Link: https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=patch;h=92b664ef4f25f1bd920bea4db435345629543353

commit 4dd8d833c9350d42528ada0fd65aee41b712f41d
From: Steve Dickson <email address hidden>
Date: Tue, 20 Jul 2021 17:14:04 -0400
Subject: mount.nfs: Fix the sloppy option processing
Link: https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=patch;h=4dd8d833c9350d42528ada0fd65aee41b712f41d

[Matthew Ruffell (mruffell)]

Attached is a debdiff for focal that solves this issue.

[Heitor Alves de Siqueira (halves)]

Thank you for the debdiff, Matthew! I've noticed that the upstream patches seem to apply directly onto nfs-utils-1.3.4 we have in Focal (with offset changes), but you've marked the Origin tag with 'backport'. Could you please confirm whether you had to do any changes to the upstream patches for them to apply?

[Matthew Ruffell (mruffell)]

Hi Heitor, it is a clean cherry pick. I always use "backport" in the origin field. Maybe I should start using "upstream" for cherry picks going forward and keep "backport" for things that actually needed modifying.

[Heitor Alves de Siqueira (halves)]

Thanks for the confirmation, Matthew! That's the only change I did, debdiff looks great otherwise.
Sponsored for Focal, thank you!

[Timo Aaltonen (tjaalton)]

Hello Matthew, or anyone else affected,

Accepted nfs-utils into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nfs-utils/1:1.3.4-2.5ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Matthew Ruffell (mruffell)]

Performing verification for Focal.

I started two VMs, both running Focal. One will be the server, the other the client.

On the server:

ubuntu@focal-server:~$ sudo apt update
ubuntu@focal-server:~$ sudo apt install nfs-kernel-server
...
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libtirpc-common all 1.2.5-1ubuntu0.1 [7712 B]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libtirpc3 amd64 1.2.5-1ubuntu0.1 [77.9 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 rpcbind amd64 1.2.5-8 [42.8 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 keyutils amd64 1.6-6ubuntu1.1 [44.8 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libnfsidmap2 amd64 0.25-5.1ubuntu1 [27.9 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nfs-common amd64 1:1.3.4-2.5ubuntu3.4 [204 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nfs-kernel-server amd64 1:1.3.4-2.5ubuntu3.4 [98.9 kB]
...
ubuntu@focal-server:~$ sudo mkdir -p /mnt/nfs_share
ubuntu@focal-server:~$ sudo chown -R nobody:nogroup /mnt/nfs_share/
ubuntu@focal-server:~$ sudo chmod 777 /mnt/nfs_share/
ubuntu@focal-server:~$ echo "/mnt/nfs_share 192.168.122.0/24(rw,sync,no_subtree_check)" | sudo tee /etc/exports
/mnt/nfs_share 192.168.122.0/24(rw,sync,no_subtree_check)
ubuntu@focal-server:~$ sudo exportfs -a
ubuntu@focal-server:~$ sudo systemctl restart nfs-kernel-server
ubuntu@focal-server:~$ echo "Testfile" | tee /mnt/nfs_share/testfile
Testfile
ubuntu@focal-server:~$ ip addr
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:fa:b4:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.142/24 brd 192.168.122.255 scope global dynamic enp1s0

On the client, using nfs-common 1.3.4-2.5ubuntu3.4 from -updates:

ubuntu@focal-client:~$ sudo apt update
ubuntu@focal-client:~$ sudo apt install nfs-common
...
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libtirpc-common all 1.2.5-1ubuntu0.1 [7712 B]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libtirpc3 amd64 1.2.5-1ubuntu0.1 [77.9 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 rpcbind amd64 1.2.5-8 [42.8 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 keyutils amd64 1.6-6ubuntu1.1 [44.8 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libnfsidmap2 amd64 0.25-5.1ubuntu1 [27.9 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nfs-common amd64 1:1.3.4-2.5ubuntu3.4 [204 kB]
...
ubuntu@focal-client:~$ sudo mkdir -p /mnt/nfs_clientshare

On the 5.4 kernel:

ubuntu@focal-client:~$ uname -rv
5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023
ubuntu@focal-client:~$ sudo mount -t nfs 192.168.122.142:/mnt/nfs_share /mnt/nfs_clientshare -s -o invalid,intr
ubuntu@focal-client:~$ cd /mnt/nfs_clientshare/
ubuntu@focal-client:/mnt/nfs_clientshare$ ll
total 12
drwxrwxrwx 2 nobody nogroup 4096 Aug 20 08:26 ./
drwxr-xr-x 3 root root 4096 Aug 20 08:27 ../
-rw-rw-r-- 1 ubuntu ubuntu 9 Aug 20 08:26 testfile
ubuntu@focal-client:/mnt/nfs_clientshare$ cat testfile
Testfile

...

[Launchpad Janitor (janitor)]

This bug was fixed in the package nfs-utils - 1:1.3.4-2.5ubuntu3.5

---------------
nfs-utils (1:1.3.4-2.5ubuntu3.5) focal; urgency=medium

  * Fix sloppy argument processing with the '-s' and '-o' command line
    options to ensure options are passed to the kernel at the very
    beginning of the list. This fixes compatibility with the 5.15
    HWE kernel. (LP: #2025302)
    - d/p/lp-2025302-1-mount.nfs-insert-sloppy-at-beginning-of-the-options.patch
    - d/p/lp-2025302-2-mount.nfs-Fix-the-sloppy-option-processing.patch

 -- Matthew Ruffell <email address hidden> Thu, 29 Jun 2023 14:29:20 +1200

[Robie Basak (racb)]

The verification of the Stable Release Update for nfs-utils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.