nfs-utils/nfs-kernel-server (rpc.svcgssd) ignored /etc/nfs.conf settings

[Marcel]

[Impact]
The "principal" option for svcgssd in /etc/nfs.conf is being ignored by the service. This breaks NFS setups that use kerberos and need a specific principal name.

Another issue is that some svcgssd options are missing from the nfs.conf(5) manpage, notably verbosity, rpc-verbosity and idmap-verbosity, that on its own probably wouldn't warrant an SRU, but can go together with this one. I don't think a new bug is needed, since they are about svcgssd too.

Finally, when using the "principal" option, it would be helpful if the error message actually used the given principal name, instead of a generic "nfs/<your.host>@<YOUR.REALM>" one. Again, on its own it does not warrant an SRU, but can go together with this one.

The test case will cover all three issues.

[Test Plan]

Create a jammy vm. For example, with lxd:

$ lxc launch ubuntu:jammy j-nfs-1977745 --vm

Then login on the sytstem:

$ lxc shell j-nfs-1977745

Install a kerberos kdc, utilities, and nfs server packages:

# apt update && apt install -y krb5-kdc krb5-admin-server nfs-kernel-server

Above, when prompted for the realm and KDC/ADMIN servers, use SRU for the realm, and "localhost" for the kdc and admin servers.

Now create the realm. Choose whatever password you like, it won't be needed again:

# krb5_newrealm

Create a service principal for the nfs server. To test this bug, we will create a principal with a non-standard name:

# kadmin.local -q "addprinc -randkey someservice/somehost@SRU"

Extract it to the system keytab:
# kadmin.local -q "ktadd someservice/somehost@SRU"

Stop nfs services, just to be sure they are not running for the subsequent tests:

# systemctl stop nfs-utils.service nfs-server.service

Now the tests begin.

a) In the released version, the nfs.conf(5) manpage only lists the "principal" option under "svcgssd":
"""
svcgssd
 Recognized values: principal.
 See rpc.svcgssd(8) for details.
"""

The fixed version will add to that list verbosity, rpc-verbosity, and idmap-vervosity:
"""
svcgssd
   Recognized values: principal, verbosity, rpc-verbosity, idmap-verbosity.
"""

b) Set the specific principal we created for svcgssd in /etc/nfs.conf:

# nfsconf --set svcgssd principal someservice/somehost@SRU

Confirm it's there:
# grep principal /etc/nfs.conf -B 1
[svcgssd]
principal = someservice/somehost@SRU

Run the svcgssd binary, and confirm it fails and complains about missing credentials:

# /usr/sbin/rpc.svcgssd -f
ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - No key table entry found for @SRU
unable to obtain root (machine) credentials
do you have a keytab entry for nfs/<your.host>@<YOUR.REALM> in /etc/krb5.keytab?

In the fixed version, it will start normally:

# /usr/sbin/rpc.svcgssd -f
(no further output)

c) In the error case from (b), note that it suggests to check for a generic principal name in the keytab, without mentioning the specific principal we set in nfs.conf.

For the fixed version, let's change the principal name to one we really don't have a keytab for, and see what the error message suggests this time:

# nfsconf --set svcgssd principal anotherservice/anotherhost@SRU

This time the error specifically mentions anotherservice/anotherhost@SRU instead of the generic nfs/<your.host>@<YOUR.REALM>:

# /usr/sbin/rpc.svcgssd -f
ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - No key table entry found for anotherservice/anotherhost@SRU
unable to obtain root (machine) credentials
do you have a keytab entry for anotherservice/anotherhost@SRU in/etc/krb5.keytab?

[Where problems could occur]

An NFS server is actually comprised of multiple services, specially when kerberos is involved. Restarting them in the correct order is the goal of the packaging, but there are some cases where this doesn't yet work correctly, like bug #1971935 shows.

[Other Info]
Not at this time.

[Original Description]

Tested on:
 Ubuntu 22.04 (x86_64)
 Package: nfs-kernel-server 1:2.6.1-1ubuntu1

Set config options in /etc/nfs.conf are ignored by rpc.svcgssd (required for krb5 NFSv4).

I was trying to set the principal name like:

[svcgssd]
<email address hidden>

but rpc.svcgssd refused to start. When specified on command line (using the -p option) things started working.

After having a look at the code (nfs-utils-2.6.1/utils/gssd/svcgssd.c), the problem seems to be

        /* We don't need the config anymore */
        conf_cleanup();

which is called too early. So at the point where gssd_acquire_cred() is called the variable "principal" does no longer contain the data read from the config file.

Moving conf_cleanup() to the end of the code helps.

I also tried to get into contact with the nfs-utils developer themselves - but I hope someone at Ubuntu has a better way to contact them.

[Andreas Hasenack (ahasenack)]

Was your upstream contact attempt via the mailing list? Do you happen to have a link, or the subject of the message?

[Marcel (linux-ng)]

Send some patches to <email address hidden>:

Subject(s):
[PATCH 1/3] cifs-utils/svcgssd: Fix use-after-free bug (config variables)
[PATCH 2/3] cifs-utils/svcgssd: Display principal if set
[PATCH 3/3] cifs-utils/svcgssd: Add (undocumented) config options to man page

(Update: just noticed, that I added "cifs-utils" to the description - that's wrong of course: "nfs-utils" would be correct)

s. https://lore.kernel.org/linux-nfs/

First patch is the one fixing the problem itself.
Patch 2 and 3 improve documentation/logging.

BTW: using gssproxy as a workaround helped.

[Marcel (linux-ng)]

Also sending the patches here.

Easy way to test:

1. Edit /etc/nfs.conf:

[svcgssd]
<email address hidden>

2. Try to run
  /usr/sbin/rpc.svcgssd -f (.deb version)
  This will fail even if you set a correct principal name
  that's present in /etc/krb5.keytab

3. Unpack nfs-utils sources and apply patch 02-nfs-utils-log-principal.patch.
   (This only adds an output of the principal set in the config)

   Then run:
   LD_LIBRARY_PATH=./support/nfsidmap/.libs ./utils/gssd/.libs/svcgssd -f

   The output shows some random principal string

4. Apply patch 01-nfs-utils-fix-conf.patch (patch will be in next comment, only one patch per comment) and compile again

   Run above test again:
   LD_LIBRARY_PATH=./support/nfsidmap/.libs ./utils/gssd/.libs/svcgssd -f

   This time the principal set by you should be shown.

As a side note: Patch #3 is only about adding some undocumented options.

[Marcel (linux-ng)]

The attached file is the real patch for the problem.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "nfs-utils/svcgssd: Display principal if set" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Marcel (linux-ng)]

Patvhes got merged in nfs-utils:

http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commit;h=2eabb25d5a43e48e769a0db29956e9f5dc5b5913
http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commit;h=284d249e0fe58443dafc96fa8be51a2cef4541a0
http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commit;h=f541550358f136e9a6d1fd131e83d17e6269dae4

[Andreas Hasenack (ahasenack)]

I'll include these in the merge from debian I'm working on at the moment.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nfs-utils - 1:2.6.1-2ubuntu1

---------------
nfs-utils (1:2.6.1-2ubuntu1) kinetic; urgency=medium

  * Merge with Debian unstable (LP: #1974233). Remaining changes:
    - d/control: don't provide libnfsidmap2 in libnfsidmap1. This
      package contains not only plugins, but an actual shared library,
      with a different soname.
    - Don't install the regex module, as it's built by
      src:libnfsidmap-regex which is in Universe (MIR: #1960824)
      + d/control: don't conflict/break/etc with libnfsidmap-regex
      + d/libnfsidmap1.install: don't install regex.so
      + d/not-installed: mark files we knowingly don't include in the
        packaging
      + d/p/remove-regex-from-docs.patch: remove the regex section from
        the idmapd.conf(5) manpage, as we are not building that plugin in
        this package
    - Update README file:
      + d/README.Ubuntu: new /etc/nfs.conf config structure
      + d/libnfsidmap1.docs, d/nfs-common.docs: install README.Ubuntu
    - d/nfs-common.postrm: also purge /etc/nfs.conf.d/local.conf
    - d/nfs-common.dirs: we also own /etc/nfs.conf.d
    - New apport hook (LP #1961058):
      + d/source.apport: apport hook for nfs-utils
      + d/control: build-depend dh-apport
      + d/rules: build with apport, and install the hook in the
        nfs-common package which is installed on both client and servers
    - Add more DEP8 tests (LP #1960828):
      + d/t/{control,kerberos-mount,util}: test NFSv4 krb5p mounts
      + d/t/{control, v3-moun}t: specific NFSv3 mount test
  * Dropped:
    - d/nfsconvert.py: add short "u" option for mountd's no-udp
      [Included in 1:2.6.1-2]
    - d/NEWS: explain some of the major changes in 2.6.x
      [Obsoleted by Debian's update to the per-package NEWS files]
    - d/nfs-*.bug-script: update to also include /etc/nfs.conf and
      /etc/nfs.conf.d/*.conf
      [Included in 1:2.6.1-2]
  * Added changes:
    - New binary package libnfsidmap-regex (LP: #1974067):
      + d/control: new package
      + d/libnfsidmap-regex.install: install the plugin file
      + d/not-installed: remove the plugin from the not-installed list
      + d/p/remove-regex-from-docs.patch: deleted
      + d/p/ubuntu-idmapd-manpage-update-regex-other-package.patch:
        note that the regex plugin is in another package
    - rpc.svcgssd fixes and improvements (LP: #1977745):
      + d/p/svcgssd-fix-use-after-free.patch: fix use-after-free which was
        preventing svcgssd options set in /etc/nfs.conf from being used
      + d/p/svcgssd-display-principal-if-set.patch: improve logging,
        showing the expected principal name if it was set in the config
      + d/p/svcgssd-document-missing-options.patch: add missing options to
        the svcgssd manpage
      + d/p/nfs-conf-manpage-missing-svcgssd-options.patch: also
        document the missing svcgssd options to the nfs.conf(5) manpage
    - d/README.Ubuntu: updated with the content of the previous d/NEWS
      file
    - d/rules: re-add hardening option lost from the src:libnfsidmap to
      src:nfs-utils transition (LP: #1980095)

 -- Andreas Hasenack <email address hidden> Tue, 28 Jun 2022 10:59...

[Robie Basak (racb)]

Hello Marcel, or anyone else affected,

Accepted nfs-utils into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nfs-utils/1:2.6.1-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Jammy verification

Reproducing the bug:
root@j-nfs-1977745:~# apt-cache policy nfs-common
nfs-common:
  Installed: 1:2.6.1-1ubuntu1
  Candidate: 1:2.6.1-1ubuntu1
  Version table:
 *** 1:2.6.1-1ubuntu1 500
        500 http://br.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

root@j-nfs-1977745:~# grep principal /etc/nfs.conf -B 1
[svcgssd]
principal = someservice/somehost@SRU

root@j-nfs-1977745:~# /usr/sbin/rpc.svcgssd -f
ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - No key table entry found for @SRU
unable to obtain root (machine) credentials
do you have a keytab entry for nfs/<your.host>@<YOUR.REALM> in /etc/krb5.keytab?

It fails to start, and mentions a generic keytab entry in the error message.

Now with the fixed package from jammy-proposed:

root@j-nfs-1977745:~# apt-cache policy nfs-common
nfs-common:
  Installed: 1:2.6.1-1ubuntu1.1
  Candidate: 1:2.6.1-1ubuntu1.1
  Version table:
 *** 1:2.6.1-1ubuntu1.1 500
        500 http://br.archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages

The service starts without errors:
root@j-nfs-1977745:~# /usr/sbin/rpc.svcgssd -f
(nothing in the output)

Updating the principal name and trying again, this time it fails to start (as it should), and instead of a generic keytab entry name, it mentions the one it was expecting to find:

root@j-nfs-1977745:~nfsconf --set svcgssd principal anotherservice/anotherhost@SRURU

root@j-nfs-1977745:~# /usr/sbin/rpc.svcgssd -f
ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - No key table entry found for anotherservice/anotherhost@SRU
unable to obtain root (machine) credentials
do you have a keytab entry for anotherservice/anotherhost@SRU in/etc/krb5.keytab?

As for the manpages:

nfs.conf(5) mentions the extra options:
       svcgssd
              Recognized values: principal, verbosity, rpc-verbosity, idmap-verbosity.

Jammy verification succeeded.

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for nfs-utils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nfs-utils - 1:2.6.1-1ubuntu1.1

---------------
nfs-utils (1:2.6.1-1ubuntu1.1) jammy; urgency=medium

  * rpc.svcgssd fixes and improvements (LP: #1977745):
    - d/p/svcgssd-fix-use-after-free.patch: fix use-after-free which was
      preventing svcgssd options set in /etc/nfs.conf from being used
    - d/p/svcgssd-display-principal-if-set.patch: improve logging,
      showing the expected principal name if it was set in the config
    - d/p/svcgssd-document-missing-options.patch: add missing options to
      the svcgssd manpage
    - d/p/nfs-conf-manpage-missing-svcgssd-options.patch: also
      document the missing svcgssd options to the nfs.conf(5) manpage

 -- Andreas Hasenack <email address hidden> Wed, 14 Sep 2022 14:34:00 -0300

[Andreas Hasenack (ahasenack)]

Phasing[1] stopped this SRU because it detected that blkmapd was crashing on startup after the update was applied. That crash is due to another bug[2], which I was finally able to reproduce and apply a fix, and it will be SRUed soon.

1. https://people.canonical.com/~ubuntu-archive/phased-updates.html
2. https://bugs.launchpad.net/debian/+source/nfs-utils/+bug/1979885