NFSv4 shares with option `-vers=4.0` are not mounted: mount(nfs): no hosts available

[Paul Menzel]

[Impact]

 * Specifying NFS v4 works, but using subversion fails.
   To make it worse v4.2 is considered not v4 at all which is not what
   users expect or need. If then your server does not support v4 the
   mount fails.

 * This is fixed upstream for a while, backport the patch from there to at
   lest the latest LTS (older ones are unlikely to suddenly need version
   4.x).

[Test Case]

 * This is a bit tricky you need a NFS server that is "less tolerant" liek
   the one reported in comment #5. If you have such a server set up
   configure your exports following [1] (or similar)
   I usually place a file called "testfile" in that export to check if it
   exists, feel free to do similarly.
 * Then on the client side install autofs and follow [2]
   The summary of those steps is:
   - you need the autofs4 kernel module which is in linux-modules-extra
     so install that if you don't have it.
   - configure your autofs.master to use a nfs mount for example
     $ echo "/nfs/ /etc/auto.nfs" | sudo tee -a /etc/auto.master
     $ sudo mkdir -p /nfs/testmount
     $ echo "testmount -fstype=nfs,rw,retry=0
       <SERVERIP>:/export/users/testmount" | sudo tee -a /etc/auto.nfs
     $ ll /nfs/testmount/testfile
       -rw-r--r-- 1 root root 0 Mar 1 10:43 /nfs/testmount/testfile

     Without the fix it will fail to mount and "/nfs/testmount" will be an
     empty unmoutned directory.

     With the fix you will see the mount working
     (from mount output)
     192.168.122.55:/export/users/testmount on /nfs/testmount type nfs4
     (rw,relatime,vers=4.2,rsize=65536,wsize=65536,namlen=255,hard,
     proto=tcp,timeo=600,retrans=2,sec=sys,
     clientaddr=192.168.122.226,local_lock=none,addr=192.168.122.55)

[1]: https://help.ubuntu.com/community/SettingUpNFSHowTo
[2]: https://wiki.ubuntuusers.de/Autofs/

[Regression Potential]

 * The change is minimal and just changes from a full string match to a
   prefix match. In theory that could make configurations which today
   specify 4.2 (and silently fail doing so as this is broken) then
   "suddenly" really use v4 which it currently does not always do.
   v4.2 currently is mounted as v4 and after the change will be mounted as
   v4.2 as specified - but that is the fix and not so much a regression.

[Other Info]

 * n/a

---

With Ubuntu 18.10 and autofs 5.1.2-4ubuntu1, mounting NFSv4 shares does not work. Directly mounting them with `mount.nfs` works.

```
Feb 28 18:08:30 nfsclient automount[22978]: expire_proc: exp_proc = 131147807650128 path /home_mariux
Feb 28 18:08:30 nfsclient automount[22978]: expire_cleanup: got thid 131147807650128 path /home_mariux stat 0
Feb 28 18:08:30 nfsclient automount[22978]: expire_cleanup: sigchld: exp 131147807650128 finished, switching from 2 to 1
Feb 28 18:08:30 nfsclient automount[22978]: st_ready: st_ready(): state = 2 path /home_mariux
Feb 28 18:08:32 nfsclient automount[22978]: handle_packet: type = 3
Feb 28 18:08:32 nfsclient automount[22978]: handle_packet_missing_indirect: token 46, name joey, request pid 28647
Feb 28 18:08:32 nfsclient automount[22978]: attempting to mount entry /home_mariux/joey
Feb 28 18:08:32 nfsclient automount[22978]: lookup_mount: lookup(file): looking up joey
Feb 28 18:08:32 nfsclient automount[22978]: lookup_mount: lookup(file): joey -> claptrap:/amd/claptrap/2/home/edv/joey
Feb 28 18:08:32 nfsclient automount[22978]: parse_mount: parse(sun): expanded entry: claptrap:/amd/claptrap/2/home/edv/joey
Feb 28 18:08:32 nfsclient automount[22978]: parse_mount: parse(sun): gathered options: nosuid
Feb 28 18:08:32 nfsclient automount[22978]: parse_mount: parse(sun): dequote("claptrap:/amd/claptrap/2/home/edv/joey") -> claptrap:/amd/claptrap/2/home/edv/bucz
Feb 28 18:08:32 nfsclient automount[22978]: parse_mount: parse(sun): core of entry: options=nosuid, loc=claptrap:/amd/claptrap/2/home/edv/joey
Feb 28 18:08:32 nfsclient automount[22978]: sun_mount: parse(sun): mounting root /home_mariux, mountpoint joey, what claptrap:/amd/claptrap/2/home/edv/joey, f
Feb 28 18:08:32 nfsclient automount[22978]: mount_mount: mount(nfs): root=/home_mariux name=joey what=claptrap:/amd/claptrap/2/home/edv/joey, fstype=nfs, opti
Feb 28 18:08:32 nfsclient automount[22978]: mount_mount: mount(nfs): nfs options="nosuid", nobind=0, nosymlink=0, ro=0
Feb 28 18:08:32 nfsclient automount[22978]: get_nfs_info: called with host claptrap(141.14.16.132) proto 6 version 0x30
Feb 28 18:08:32 nfsclient automount[22978]: get_nfs_info: called with host claptrap(141.14.16.132) proto 17 version 0x30
Feb 28 18:08:32 nfsclient automount[22978]: mount(nfs): no hosts available
Feb 28 18:08:32 nfsclient automount[22978]: dev_ioctl_send_fail: token = 46
Feb 28 18:08:32 nfsclient automount[22978]: failed to mount /home_mariux/joey
Feb 28 18:08:32 nfsclient automount[22978]: st_readmap: state 1 path /home_mariux
Feb 28 18:08:32 nfsclient automount[22978]: handle_packet: type = 3
Feb 28 18:08:32 nfsclient automount[22978]: handle_packet_missing_indirect: token 47, name joey, request pid 28647
Feb 28 18:08:32 nfsclient automount[22978]: dev_ioctl_send_fail: token = 47
Feb 28 18:08:32 nfsclient automount[22978]: re-reading map for /home_mariux
Feb 28 18:08:32 nfsclient automount[22978]: lookup_nss_read_map: reading map file /etc/automount/auto.home
Feb 28 18:08:32 nfsclient automount[22978]: do_init: parse(sun): init gathered global options: nosuid
Feb 28 18:08:32 nfsclient automount[22978]: st_ready: st_ready(): state = 4 path /home_mariux
```

[Christian Ehrhardt  (paelzer)]

I was following https://help.ubuntu.com/community/SettingUpNFSHowTo in two KVM machines using 18.10
and then https://wiki.ubuntuusers.de/Autofs/
(note you need linux-generic to get autofs4 module)

In exports I have
/export/users/testmount 192.168.122.0/24(rw,insecure,no_subtree_check,async)
$ sudo mkdir -p /export/users/testmount
$ sudo touch /export/users/testmount/testfile

After that in my case I can mount it like:
 sudo mount -v -t nfs 192.168.122.55:/export/users/testmount /mnt

Now trying that via automount:
$ echo "/nfs/ /etc/auto.nfs" | sudo tee -a /etc/auto.master
$ sudo mkdir -p /nfs/testmount
$ echo "testmount -fstype=nfs,rw,retry=0 192.168.122.55:/export/users/testmount" | sudo tee -a /etc/auto.nfs
$ ll /nfs/testmount/testfile
-rw-r--r-- 1 root root 0 Mar 1 10:43 /nfs/testmount/testfile

The mount works just fine and it is v4 (from mount)
192.168.122.55:/export/users/testmount on /nfs/testmount type nfs4 (rw,relatime,vers=4.2,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.226,local_lock=none,addr=192.168.122.55)

It must be something to your local configuration that is different.
I have to ask you to try setting up two clean VMs as well and then modifying the config I used to step by step to match yours.
That should either reveal which change broke your case.
Or you'll end up with a step by step howto that developers can follow to further debug your case.

[Christian Ehrhardt  (paelzer)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Since it seems likely to me that this is a local configuration problem, rather than a bug in Ubuntu, I'm marking this bug as Incomplete.

If indeed this is a local configuration problem, you can find pointers to get help for this sort of problem here: http://www.ubuntu.com/support/community

Or if you believe that this is really a bug, then you may find it helpful to read "How to report bugs effectively" http://www.chiark.greenend.org.uk/~sgtatham/bugs.html. We'd be grateful if you would then provide a more complete description of the problem, explain why you believe this is a bug in Ubuntu rather than a problem specific to your system, and then change the bug status back to New.

[Paul Menzel (paulmenzel)]

Thank you for trying to reproduce the issue, and sorry for missing out information. In this case, due to NFS 4.2 issues, we have specified the exact NFS version.

     joey -vers=4.0 claptrap:/amd/claptrap/2/home/edv/joey

Bisecting the fix in autofs 5.1.4, results in the commit aa1f4321 (autofs-5.1.3 - handle additional nfs versions in mount_nfs.c).

commit aa1f432180f3878c303088def8d647f1bd50b10b (refs/bisect/fixed)
Author: Ian Kent <email address hidden>
Date: Thu Oct 19 08:48:44 2017 +0800

    autofs-5.1.3 - handle additional nfs versions in mount_nfs.c

    Since NFSv4 can now have a subversion it needs to be allowed for.

    It's enough to check for an options string starting with "vers=4" or
    "nfsvers=4" because it's used only to set a flag used for special
    casing the availibility probe for any NFS version 4 version.

    Signed-off-by: Ian Kent <email address hidden>

diff --git a/CHANGELOG b/CHANGELOG
index 2da2cdc..3a1e4a6 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -35,6 +35,7 @@ xx/xx/2017 autofs-5.1.4
 - make map source reference message debug only.
 - improve description of mount_nfs_default_protocol.
 - the port option should not behave like nobind option.
+- handle additional nfs versions in mount_nfs.c.

 24/05/2017 autofs-5.1.3
 =======================
diff --git a/modules/mount_nfs.c b/modules/mount_nfs.c
index 5245d96..d3ebd92 100644
--- a/modules/mount_nfs.c
+++ b/modules/mount_nfs.c
@@ -147,8 +147,9 @@ int mount_mount(struct autofs_point *ap, const char *root, const char *name, int
                        } else if (_strncmp("use-weight-only", cp, o_len) == 0) {
                                flags |= MOUNT_FLAG_USE_WEIGHT_ONLY;
                        } else {
- if (_strncmp("vers=4", cp, o_len) == 0 ||
- _strncmp("nfsvers=4", cp, o_len) == 0)
+ /* Is any version of NFSv4 in the options */
+ if (_strncmp("vers=4", cp, 6) == 0 ||
+ _strncmp("nfsvers=4", cp, 9) == 0)
                                        vers = NFS4_VERS_MASK | TCP_SUPPORTED;
                                else if (_strncmp("vers=3", cp, o_len) == 0 ||
                                         _strncmp("nfsvers=3", cp, o_len) == 0) {

Reading the commit message and looking at the change, it makes sense, that this fixes the issue. Could you please apply this to the package?

[Christian Ehrhardt  (paelzer)]

Thanks for bisecting to the change you need already - that is great!

But it seems we need something more to recreate and verify the fix eventually.

Per your last comment I have set my autofs config to:
  testmount -vers=4.0 -fstype=nfs,rw,retry=0 192.168.122.55:/export/users/testmount
Which is using your -vers=4.0 string that you have shown

At least after an autofs reload it still worked.
Then I restarted the system to make sure it only knows about the new vers=4.0 config.
But it still works and mounts it as 4.0

Here from mount output:
192.168.122.55:/export/users/testmount on /nfs/testmount type nfs4 (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.226,local_lock=none,addr=192.168.122.55)

That is using autofs in Cosmic as you reported:
autofs 5.1.2-4ubuntu1
nfs-common 1:1.3.4-2.2ubuntu3

Do you happen to know what we still miss to trigger it for my repro?

[Paul Menzel (paulmenzel)]

Our NFS server is not a Ubuntu system. It’s basically an LFS based set-up [1]. The autofs client installed on our system has no trouble connecting to the server, so some minor detail seems to be different, that it doesn’t work with the Ubuntu client out of the box.

[1]: https://github.molgen.mpg.de/mariux64

[Paul Menzel (paulmenzel)]

Our NFS server does not advertise/support NFSv3 for example.

[Christian Ehrhardt  (paelzer)]

Hmm, ok sounds ok even thou it makes the SRU process harder :-/
And the change seems safe and small enough that I don't want to be blocking it just by "process imprfection" :-)

But I'd need to rely on you to test from PPAs (for Disco) and then Cosmic and Bionic for the SRU.
Are you willing and able to do that - because otherwise the SRU will get stuck being non verifiable - so I wanted to check in advance if you are ok with that?

[Christian Ehrhardt  (paelzer)]

I have prepared a PPA [1] which if confirmed working fine could slip into Ubutnu Disco just before we hit the harder freezes (since it is just a small fix it can still go in right now).

@Paul - please test the PPA on Disco (19.10) and let me know if it works fine.

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1818121-autofs-nfs4.x

[Christian Ehrhardt  (paelzer)]

Opened also an MP to get a check on the packaging changes [1].
But they are minimal so I hope those will be fine as-is.

[1]: https://code.launchpad.net/~paelzer/ubuntu/+source/autofs/+git/autofs/+merge/364082

[Paul Menzel (paulmenzel)]

Ubuntu Disco is 19.04, right? I’ll try to test this properly today.

[Christian Ehrhardt  (paelzer)]

Yes 19.04 == Disco, sorry that I confused you by writing 19.10 in comment #8.
The current PPA is for Disco.

[Paul Menzel (paulmenzel)]

@Paul - please test the PPA on Disco (19.04) [1] and let me know if it works fine.

I just tested this. With autofs 5.1.2-4ubuntu1 it does *not* work, with autofs 5.1.2-4ubuntu2~ppa1 it does work.

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1818121-autofs-nfs4.x

[Christian Ehrhardt  (paelzer)]

Thanks, then I only wait until a teammate had time to review the MP.

[Launchpad Janitor (janitor)]

This bug was fixed in the package autofs - 5.1.2-4ubuntu2

---------------
autofs (5.1.2-4ubuntu2) disco; urgency=medium

  * d/p/autofs-5.1.3-handle-additional-nfs-versions-in-mount.patch:
    fix usage of NFS 4.x subversions (LP: #1818121)

 -- Christian Ehrhardt <email address hidden> Thu, 07 Mar 2019 13:03:35 +0100

[Christian Ehrhardt  (paelzer)]

Ok, that worked well in Disco.

The PPA [1] now contains the builds for a fix for Bionic/Cosmic as well.
Two MPs for that are up (linked to the bug here already)

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1818121-autofs-nfs4.x

[Christian Ehrhardt  (paelzer)]

All is pre-approved we are now entering the SRU [1] process and I have uploaded the changes to Bionic-/Cosmic-unapproved.

@Paul - once accepted by the SRU Team I'd hope you can again do the quick checks with your already set up case on 18.04 and 18.10 then.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates

[Paul Menzel (paulmenzel)]

I checked, that your packages fix the problem on both Bionic and Cosmic.

[Brian Murray (brian-murray)]

Hello Paul, or anyone else affected,

Accepted autofs into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autofs/5.1.2-4ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Paul, or anyone else affected,

Accepted autofs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autofs/5.1.2-1ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Christian Ehrhardt  (paelzer)]

@Paul - you were so active before - thanks for checking the PPAs and all that.
But I still have no NFS server uncoorperative enough to trigger this; Could you for the (final) SRU verification as asked on the comments #18 and #19 confirm that this works in your setup with the changes that are in proposed [1].

[1]: https://wiki.ubuntu.com/Testing/EnableProposed

[Paul Menzel (paulmenzel)]

On 25.03.19 07:53, Christian Ehrhardt wrote:
> @Paul - you were so active before - thanks for checking the PPAs and
> all that. But I still have no NFS server uncoorperative enough to
> trigger this; Could you for the (final) SRU verification as asked on
> the comments #18 and #19 confirm that this works in your setup with
> the changes that are in proposed [1].

Aren’t they identical to the packages in your PPA? Anyway, I can test
the proposed packages.

[Paul Menzel (paulmenzel)]

It worked with bionic-proposed.

[Paul Menzel (paulmenzel)]

On a ppc64le system I verified, that the version in cosmic-proposed
fixes this issue.

A big thanks to everyone, fixing this so quickly.

[Christian Ehrhardt  (paelzer)]

Thank you Paul, updating the tags accordingly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package autofs - 5.1.2-4ubuntu1.1

---------------
autofs (5.1.2-4ubuntu1.1) cosmic; urgency=medium

  * d/p/autofs-5.1.3-handle-additional-nfs-versions-in-mount.patch:
    fix usage of NFS 4.x subversions (LP: #1818121)

 -- Christian Ehrhardt <email address hidden> Thu, 07 Mar 2019 13:03:35 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for autofs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package autofs - 5.1.2-1ubuntu3.1

---------------
autofs (5.1.2-1ubuntu3.1) bionic; urgency=medium

  * d/p/autofs-5.1.3-handle-additional-nfs-versions-in-mount.patch:
    fix usage of NFS 4.x subversions (LP: #1818121)

 -- Christian Ehrhardt <email address hidden> Thu, 07 Mar 2019 13:03:35 +0100