rpi nfsroot vers=4 not supported anymore?

[Seth Bromberger]

I'm running `5.13.0-1011-raspi #13-Ubuntu SMP PREEMPT Fri Nov 19 18:40:23 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux` and have successfully set up netbooting on my RPi 4B (4GB). When I do not specify a `vers=` in cmdline.txt, NFS version 3 is used. I'd like to move to version 4, but unlike the setup described in https://forums.raspberrypi.com/viewtopic.php?p=1751741#p1751741, I cannot get `vers=4` (or any sub-version to work): the console logs `invalid value for vers.` for whatever I try.

Current working cmdline.txt:
```
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.3.101:/srv/pinet/root,tcp ro ip=dhcp rootwait fixrtc splash
```

Nonworking cmdline.txt:
```
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.3.101:/srv/pinet/root,tcp,vers=4 ro ip=dhcp rootwait fixrtc splash
```

[Chris Guiver (guiverc)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command only once, as it will automatically gather debugging information, in a terminal:

apport-collect 1954716

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

Bug reporting is mostly about finding & fixing problems thus preventing future users from hitting the same bug.

I suspect a Support site would be more appropriate, eg. https://answers.launchpad.net/ubuntu. You can also find help with your problem in the support forum of your local Ubuntu community http://loco.ubuntu.com/ or asking at https://askubuntu.com or https://ubuntuforums.org, or for more support options please look at https://discourse.ubuntu.com/t/community-support/709

[Seth Bromberger (sbromberger)]

> apport-collect 1954716

I'll run this but in order to do this I have to boot into a working system, which means that you won't see the error.

[Seth Bromberger (sbromberger)]

Actually, I can't run this. These are headless isolated devices, and it looks like this utility tries to launch a browser to authenticate to launchpad. How do I get you the information you need?

[Juerg Haefliger (juergh)]

NFS root mounts are handled by nfsmount which is part of the klibc-utils package. nfsmount only accepts integers for the 'vers' argument. And only values 2 and 3 are valid:

$ ./usr/bin/nfsmount -o vers=4.1
4.1: invalid value for vers
$ ./usr/bin/nfsmount -o vers=4
./usr/bin/nfsmount: bad NFS version '4'
$ ./usr/bin/nfsmount -o vers=3
./usr/bin/nfsmount: need a path

The kernel does support NFSv4:
./lib/modules/5.13.0-1011-raspi/kernel/fs/nfs/nfsv4.ko

[Juerg Haefliger (juergh)]

The title says 'rpi nfsroot vers=4 not supported anymore?' which implies that it used to work in the past. Has it?

[Juerg Haefliger (juergh)]

Spoken too soon. There's a kernel config option that we might be missing (CONFIG_ROOT_NFS), which seems to pull in code that parses nfsroot kernel commandline options.

[Juerg Haefliger (juergh)]

More data points. The raspi OS kernel has ROOT_NFS enabled which presumably handles the 'vers=<foo>' parameters. In our case that option is disabled and initrd scripts try to parse the kernel parameters and fall on their faces.

[Juerg Haefliger (juergh)]

Upstream Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=409272

[Haro (h4r0)]

I just switched from raspberry pi os arm64 which supports netbooting from nfs vers=4.2

With ubuntu (server 21.10) console spams:
Begin: Retrying nfs mount ... 4.2: invalid value for vers

I tried rebuilding the kernel with:
CONFIG_NFS_V4 y
CONFIG_NFS_V4_1 y
CONFIG_NFS_V4_2 y
CONFIG_NFS_COMMON y
CONFIG_ROOT_NFS y

But that alone does not fix the issue.

[Haro (h4r0)]

Using the workaround in the debian bug report works.

You don't even need to recompile the kernel.

Just add mount.nfs via initram hook from nfs-common

[Juerg Haefliger (juergh)]

As mentioned, NFS mounts are handled by nfsmount provided by the klibc-utils package which currently doesn't support NFSv4:

$ /usr/lib/klibc/bin/nfsmount -o vers=4
/usr/lib/klibc/bin/nfsmount: bad NFS version '4'

Reassigning the bug to klibc-utils.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stephen Winnall (winnall)]

The upstream bug is in Debian 11. The current version of RaspiOS, which seems to support booting from NFSv4, is a Debian 11 derivative. That means that RaspiOS has fixed it for Debian 11. Is it possible to copy the RaspiOS solution?

[Juerg Haefliger (juergh)]

Debian 11 is also broken if an intird is being used but Raspberry Pi OS is *not* using an initrd hence not using nfsmount.

[Juerg Haefliger (juergh)]

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=409272#14 for a work-around which replaces nfsmount in the initrd with /sbin/mount.nfs. But that probably pulls in dependencies and blows up the size of the initrd.

[Stephen Winnall (winnall)]

Thanks, I’ll look at that.

I note that this issue is now a confirmed bug. What does that mean in practical terms? Will it be dealt with this week, this month, this year…?

I need to be able to boot from NFSv4. If RaspiOS can do that it might make sense for me to move after 15 years from Ubuntu to RaspiOS. I don’t want to, but I’ve already wasted months on this issue.

[Juerg Haefliger (juergh)]

It simply means that the ticket has been acknowledged as a bug/issue, nothing more. As you might have noticed it's been reported in Debian in 2007, so it doesn't seem to be that important. I know that doesn't help you but the fact that there's a work-around makes it even less important. I'll take a look, no guarantees.

As for reporting issues, it might help to jump in an IRC channel [1] and harass people directly, then it's harder for them to ignore you ;-)

[1] https://wiki.ubuntu.com/IRC/ChannelList

[Stephen Winnall (winnall)]

Thanks Juerg, you’ve been very helpful.

Presumably there is no quick fix for this because of the klibc/glibc thing, so it’s an architectural rather than a programming issue. And it needs to be fixed in Debian rather than in Ubuntu.

I’d offer to help but I arrived here after drilling down from a completely different realm and it’s outside my comfort zone.

I want to evaluate the work-around against using RaspiOS before deciding how to proceed.

Is there any way of crowd-funding bug fixing?

[Juerg Haefliger (juergh)]

Please do test the work-around. If indeed NFSv4 solves your overlayfs problem then that would be another reason for fixing nfsmount.

We can carry Ubuntu specific fixes if for some reason Debian doesn't want them but the preferred way is obviously to fix it in Debian so that it tickles down naturally to Ubuntu (and other Debian based distros).

I'm looking at it atm but NFS is not my strong suit :-)

[Stephen Winnall (winnall)]

I will test the work-around, but I won’t get round to it before the end of the week.

> On 9 Aug 2022, at 13:55, Juerg Haefliger <email address hidden> wrote:
>
> Please do test the work-around. If indeed NFSv4 solves your overlayfs
> problem then that would be another reason for fixing nfsmount.
>
> We can carry Ubuntu specific fixes if for some reason Debian doesn't
> want them but the preferred way is obviously to fix it in Debian so that
> it tickles down naturally to Ubuntu (and other Debian based distros).
>
> I'm looking at it atm but NFS is not my strong suit :-)
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1983620).
> https://bugs.launchpad.net/bugs/1954716
>
> Title:
> rpi nfsroot vers=4 not supported anymore?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1954716/+subscriptions
>

[Thorsten Glaser (mirabilos)]

Have you considered informing klibc upstream? (Granted, save for hpa that’s basically the Debian maintainers, but they don’t necessarily get Launchpad bugreports either.)

[Stephen Winnall (winnall)]

Good point: since this is the first bug I have ever actively pursued, this is one of possibly many avenues I should follow.

> On 9 Aug 2022, at 19:30, Thorsten Glaser <email address hidden> wrote:
>
> Have you considered informing klibc upstream? (Granted, save for hpa
> that’s basically the Debian maintainers, but they don’t necessarily get
> Launchpad bugreports either.)
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1983620).
> https://bugs.launchpad.net/bugs/1954716
>
> Title:
> rpi nfsroot vers=4 not supported anymore?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1954716/+subscriptions
>

[Juerg Haefliger (juergh)]

Upstream is aware since 2007 [1] :-) But apparently lacking NFSv4 support is not painful enough to get some traction. I'm currently digging through the code.

[1] https://lists.zytor.com/archives/klibc/2007-February/002062.html

[Stephen Winnall (winnall)]

Done it! I've managed to boot my RPi4 with Ubuntu Jammy off an NFSv4 share using the workaround from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=409272#14 .

I created the file in the attachment and compiled it with klcc. If anyone else wants to use it they'll have to edit the file, put their own share address in, and recompile.

I then copied the file to /usr/lib/klibc/bin/nfsmount, replacing the version installed with Ubuntu.

After that I ran update-initramfs -u (ignoring the errors - update-initramfs seems blissfully unaware of network booting with NFS of any flavour).

After ensuring that /boot/initrd.img did in fact contain my version of nfsmount (lsinitramfs -l /boot/initrd.img) I rebooted the RPi. It came up so cleanly that I thought I must have made a mistake :-)

But df -hT showed that the root was mounted with NFSv4. Now for the next problem... :-)

[Stephen Winnall (winnall)]

Whilst trying to understand the problem I looked at the source of nfsmount in klibc. Nfsmount has a switch statement in which it checks the value of nfsvers and only proceeds if it is 2 or 3. Given that my version of nfsmount, compiled with klcc, can indeed mount using NFSv4 I suspect that in libc's version the switch statement only needs to be extended with the cases 4, 4.0, 4.1 and 4.2. It's not entirely trivial, but it's hard to understand why it hasn't been done sometime during the last 15 years.

I may take a closer look when I've fixed my next bug.

[Stephen Winnall (winnall)]

And being able to boot from NFSv4 has also made overlayroot work. I just need to go back and ensure that it still doesn’t work with NFSv3, just in case it was fixed by something else...

> On 9 Aug 2022, at 13:47, Juerg Haefliger <email address hidden> wrote:
>
> Please do test the work-around. If indeed NFSv4 solves your overlayfs
> problem then that would be another reason for fixing nfsmount.
>
> We can carry Ubuntu specific fixes if for some reason Debian doesn't
> want them but the preferred way is obviously to fix it in Debian so that
> it tickles down naturally to Ubuntu (and other Debian based distros).
>
> I'm looking at it atm but NFS is not my strong suit :-)
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1983620).
> https://bugs.launchpad.net/bugs/1954716
>
> Title:
> rpi nfsroot vers=4 not supported anymore?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1954716/+subscriptions
>