Longhorn rwx nfs mount issue - Protocol not supported

[Yassine JEANNETTE]

Hello,

I'm experiencing issues with longhorn rwx feature on k3s cluster since upgrade to kernel 5.15.0-1046-raspi

The symptom are exactly the same as the one describe on longhorn KB (https://longhorn.io/kb/troubleshooting-rwx-volume-fails-to-attached-caused-by-protocol-not-supported/)

My version deployed on my rasberry pi

root@my-pi:~$ uname -a
Linux my-pi 5.15.0-1046-raspi #49-Ubuntu SMP PREEMPT Thu Jan 18 12:45:41 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Mount request (with options)
root@my-pi:~$ sudo mount -t nfs -o vers=4.2,noresvport,softerr,timeo=600,retrans=5 XX.XX.XX.XX:/pvc-UID /mnt/nfs -vvv
mount.nfs: timeout set for Tue Feb 20 11:00:38 2024
mount.nfs: trying text-based options 'vers=4.2,noresvport,softerr,timeo=600,retrans=5,proto=tcp,addr=XX.XX.XX.XX,clientaddr=YY.YY.YY.YY'
mount.nfs: mount(2): Protocol not supported
mount.nfs: Protocol not supported

Mount request (without options)
root@my-pi:~$mount -t nfs XX.XX.XX.XX:/pvc-UID /mnt/nfs -vvv
mount.nfs: timeout set for Tue Feb 20 11:02:29 2024
mount.nfs: trying text-based options 'vers=4.2,addr=XX.XX.XX.XX,clientaddr=YY.YY.YY.YY'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'vers=4,minorversion=1,addr=XX.XX.XX.XX,clientaddr=YY.YY.YY.YY'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'vers=4,addr=XX.XX.XX.XX,clientaddr=YY.YY.YY.YY'
mount.nfs: mount(2): Protocol not supported
Created symlink /run/systemd/system/remote-fs.target.wants/rpc-statd.service → /lib/systemd/system/rpc-statd.service.
mount.nfs: trying text-based options 'addr=XX.XX.XX.XX'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: Protocol not supported

On another node (not using RPI) I had same issue with kernel linux-image-5.15.0-94-generic.Downgrading to kernel linux-image-5.15.0-92-generic solved the issue.

[Juerg Haefliger (juergh)]

Should be fixed in 5.15.137.

ommit 431a5010bce29809e68111c83e31bfd06d15a7d3
Author: Olga Kornievskaia <email address hidden>
Date: Mon Oct 9 10:59:01 2023 -0400

    NFSv4.1: fixup use EXCHGID4_FLAG_USE_PNFS_DS for DS server

    commit 379e4adfddd6a2f95a4f2029b8ddcbacf92b21f9 upstream.

    This patches fixes commit 51d674a5e488 "NFSv4.1: use
    EXCHGID4_FLAG_USE_PNFS_DS for DS server", purpose of that
    commit was to mark EXCHANGE_ID to the DS with the appropriate
    flag.

    However, connection to MDS can return both EXCHGID4_FLAG_USE_PNFS_DS
    and EXCHGID4_FLAG_USE_PNFS_MDS set but previous patch would only
    remember the USE_PNFS_DS and for the 2nd EXCHANGE_ID send that
    to the MDS.

    Instead, just mark the pnfs path exclusively.

    Fixes: 51d674a5e488 ("NFSv4.1: use EXCHGID4_FLAG_USE_PNFS_DS for DS server")
    Signed-off-by: Olga Kornievskaia <email address hidden>
    Signed-off-by: Anna Schumaker <email address hidden>
    Signed-off-by: Greg Kroah-Hartman <email address hidden>

[Juerg Haefliger (juergh)]

Which is in Ubuntu-5.15.0-100.110 from jammy-proposed. The raspi kernel that is based on this version will be prepared this week:
https://bugs.launchpad.net/kernel-sru-workflow/+bug/2052084

[Linus Östberg (lostb-sr)]

I have observed the same issue on our production Kubernetes clusters on amd64. We've been forced to hold our kernel image at 5.15.0-92. Both 5.15.0-94 and 5.15.0-97 give us the same error as above when attempting to mount NFS (RWX) volumes from Longhorn 1.5.3.

[Linus Östberg (lostb-sr)]

linux-image-generic 5.15.0.100.97 fixed the issue for us (AMD64).