nfs-utils/nfs-kernel-server (rpc.svcgssd) ignored /etc/nfs.conf settings

Bug #1977745 reported by Marcel
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
nfs-utils (Ubuntu)
Status tracked in Kinetic
Jammy
In Progress
Low
Andreas Hasenack
Kinetic
Fix Released
Low
Andreas Hasenack

Bug Description

[Impact]
The "principal" option for svcgssd in /etc/nfs.conf is being ignored by the service. This breaks NFS setups that use kerberos and need a specific principal name.

Another issue is that some svcgssd options are missing from the nfs.conf(5) manpage, notably verbosity, rpc-verbosity and idmap-verbosity, that on its own probably wouldn't warrant an SRU, but can go together with this one. I don't think a new bug is needed, since they are about svcgssd too.

Finally, when using the "principal" option, it would be helpful if the error message actually used the given principal name, instead of a generic "nfs/<your.host>@<YOUR.REALM>" one. Again, on its own it does not warrant an SRU, but can go together with this one.

The test case will cover all three issues.

[Test Plan]

Create a jammy vm. For example, with lxd:

$ lxc launch ubuntu:jammy j-nfs-1977745 --vm

Then login on the sytstem:

$ lxc shell j-nfs-1977745

Install a kerberos kdc, utilities, and nfs server packages:

# apt update && apt install -y krb5-kdc krb5-admin-server nfs-kernel-server

Above, when prompted for the realm and KDC/ADMIN servers, use SRU for the realm, and "localhost" for the kdc and admin servers.

Now create the realm. Choose whatever password you like, it won't be needed again:

# krb5_newrealm

Create a service principal for the nfs server. To test this bug, we will create a principal with a non-standard name:

# kadmin.local -q "addprinc -randkey someservice/somehost@SRU"

Extract it to the system keytab:
# kadmin.local -q "ktadd someservice/somehost@SRU"

Stop nfs services, just to be sure they are not running for the subsequent tests:

# systemctl stop nfs-utils.service nfs-server.service

Now the tests begin.

a) In the released version, the nfs.conf(5) manpage only lists the "principal" option under "svcgssd":
"""
svcgssd
 Recognized values: principal.
 See rpc.svcgssd(8) for details.
"""

The fixed version will add to that list verbosity, rpc-verbosity, and idmap-vervosity:
"""
svcgssd
   Recognized values: principal, verbosity, rpc-verbosity, idmap-verbosity.
"""

b) Set the specific principal we created for svcgssd in /etc/nfs.conf:

# nfsconf --set svcgssd principal someservice/somehost@SRU

Confirm it's there:
# grep principal /etc/nfs.conf -B 1
[svcgssd]
principal = someservice/somehost@SRU

Run the svcgssd binary, and confirm it fails and complains about missing credentials:

# /usr/sbin/rpc.svcgssd -f
ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - No key table entry found for @SRU
unable to obtain root (machine) credentials
do you have a keytab entry for nfs/<your.host>@<YOUR.REALM> in /etc/krb5.keytab?

In the fixed version, it will start normally:

# /usr/sbin/rpc.svcgssd -f
(no further output)

c) In the error case from (b), note that it suggests to check for a generic principal name in the keytab, without mentioning the specific principal we set in nfs.conf.

For the fixed version, let's change the principal name to one we really don't have a keytab for, and see what the error message suggests this time:

# nfsconf --set svcgssd principal anotherservice/anotherhost@SRU

This time the error specifically mentions anotherservice/anotherhost@SRU instead of the generic nfs/<your.host>@<YOUR.REALM>:

# /usr/sbin/rpc.svcgssd -f
ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - No key table entry found for anotherservice/anotherhost@SRU
unable to obtain root (machine) credentials
do you have a keytab entry for anotherservice/anotherhost@SRU in/etc/krb5.keytab?

[Where problems could occur]

An NFS server is actually comprised of multiple services, specially when kerberos is involved. Restarting them in the correct order is the goal of the packaging, but there are some cases where this doesn't yet work correctly, like bug #1971935 shows.

[Other Info]
Not at this time.

[Original Description]

Tested on:
 Ubuntu 22.04 (x86_64)
 Package: nfs-kernel-server 1:2.6.1-1ubuntu1

Set config options in /etc/nfs.conf are ignored by rpc.svcgssd (required for krb5 NFSv4).

I was trying to set the principal name like:

[svcgssd]
<email address hidden>

but rpc.svcgssd refused to start. When specified on command line (using the -p option) things started working.

After having a look at the code (nfs-utils-2.6.1/utils/gssd/svcgssd.c), the problem seems to be

        /* We don't need the config anymore */
        conf_cleanup();

which is called too early. So at the point where gssd_acquire_cred() is called the variable "principal" does no longer contain the data read from the config file.

Moving conf_cleanup() to the end of the code helps.

I also tried to get into contact with the nfs-utils developer themselves - but I hope someone at Ubuntu has a better way to contact them.

Related branches

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Was your upstream contact attempt via the mailing list? Do you happen to have a link, or the subject of the message?

Revision history for this message
Marcel (linux-ng) wrote (last edit ):

Send some patches to <email address hidden>:

Subject(s):
[PATCH 1/3] cifs-utils/svcgssd: Fix use-after-free bug (config variables)
[PATCH 2/3] cifs-utils/svcgssd: Display principal if set
[PATCH 3/3] cifs-utils/svcgssd: Add (undocumented) config options to man page

(Update: just noticed, that I added "cifs-utils" to the description - that's wrong of course: "nfs-utils" would be correct)

s. https://lore.kernel.org/linux-nfs/

First patch is the one fixing the problem itself.
Patch 2 and 3 improve documentation/logging.

BTW: using gssproxy as a workaround helped.

Revision history for this message
Marcel (linux-ng) wrote :

Also sending the patches here.

Easy way to test:

1. Edit /etc/nfs.conf:

[svcgssd]
<email address hidden>

2. Try to run
  /usr/sbin/rpc.svcgssd -f (.deb version)
  This will fail even if you set a correct principal name
  that's present in /etc/krb5.keytab

3. Unpack nfs-utils sources and apply patch 02-nfs-utils-log-principal.patch.
   (This only adds an output of the principal set in the config)

   Then run:
   LD_LIBRARY_PATH=./support/nfsidmap/.libs ./utils/gssd/.libs/svcgssd -f

   The output shows some random principal string

4. Apply patch 01-nfs-utils-fix-conf.patch (patch will be in next comment, only one patch per comment) and compile again

   Run above test again:
   LD_LIBRARY_PATH=./support/nfsidmap/.libs ./utils/gssd/.libs/svcgssd -f

   This time the principal set by you should be shown.

As a side note: Patch #3 is only about adding some undocumented options.

Revision history for this message
Marcel (linux-ng) wrote :

The attached file is the real patch for the problem.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "nfs-utils/svcgssd: Display principal if set" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Revision history for this message
Marcel (linux-ng) wrote :
Changed in nfs-utils (Ubuntu):
status: New → Incomplete
status: Incomplete → In Progress
assignee: nobody → Andreas Hasenack (ahasenack)
tags: added: server-todo
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I'll include these in the merge from debian I'm working on at the moment.

Changed in nfs-utils (Ubuntu Jammy):
assignee: nobody → Andreas Hasenack (ahasenack)
status: New → Triaged
Changed in nfs-utils (Ubuntu Kinetic):
importance: Undecided → Low
Changed in nfs-utils (Ubuntu Jammy):
importance: Undecided → Low
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.1 KiB)

This bug was fixed in the package nfs-utils - 1:2.6.1-2ubuntu1

---------------
nfs-utils (1:2.6.1-2ubuntu1) kinetic; urgency=medium

  * Merge with Debian unstable (LP: #1974233). Remaining changes:
    - d/control: don't provide libnfsidmap2 in libnfsidmap1. This
      package contains not only plugins, but an actual shared library,
      with a different soname.
    - Don't install the regex module, as it's built by
      src:libnfsidmap-regex which is in Universe (MIR: #1960824)
      + d/control: don't conflict/break/etc with libnfsidmap-regex
      + d/libnfsidmap1.install: don't install regex.so
      + d/not-installed: mark files we knowingly don't include in the
        packaging
      + d/p/remove-regex-from-docs.patch: remove the regex section from
        the idmapd.conf(5) manpage, as we are not building that plugin in
        this package
    - Update README file:
      + d/README.Ubuntu: new /etc/nfs.conf config structure
      + d/libnfsidmap1.docs, d/nfs-common.docs: install README.Ubuntu
    - d/nfs-common.postrm: also purge /etc/nfs.conf.d/local.conf
    - d/nfs-common.dirs: we also own /etc/nfs.conf.d
    - New apport hook (LP #1961058):
      + d/source.apport: apport hook for nfs-utils
      + d/control: build-depend dh-apport
      + d/rules: build with apport, and install the hook in the
        nfs-common package which is installed on both client and servers
    - Add more DEP8 tests (LP #1960828):
      + d/t/{control,kerberos-mount,util}: test NFSv4 krb5p mounts
      + d/t/{control, v3-moun}t: specific NFSv3 mount test
  * Dropped:
    - d/nfsconvert.py: add short "u" option for mountd's no-udp
      [Included in 1:2.6.1-2]
    - d/NEWS: explain some of the major changes in 2.6.x
      [Obsoleted by Debian's update to the per-package NEWS files]
    - d/nfs-*.bug-script: update to also include /etc/nfs.conf and
      /etc/nfs.conf.d/*.conf
      [Included in 1:2.6.1-2]
  * Added changes:
    - New binary package libnfsidmap-regex (LP: #1974067):
      + d/control: new package
      + d/libnfsidmap-regex.install: install the plugin file
      + d/not-installed: remove the plugin from the not-installed list
      + d/p/remove-regex-from-docs.patch: deleted
      + d/p/ubuntu-idmapd-manpage-update-regex-other-package.patch:
        note that the regex plugin is in another package
    - rpc.svcgssd fixes and improvements (LP: #1977745):
      + d/p/svcgssd-fix-use-after-free.patch: fix use-after-free which was
        preventing svcgssd options set in /etc/nfs.conf from being used
      + d/p/svcgssd-display-principal-if-set.patch: improve logging,
        showing the expected principal name if it was set in the config
      + d/p/svcgssd-document-missing-options.patch: add missing options to
        the svcgssd manpage
      + d/p/nfs-conf-manpage-missing-svcgssd-options.patch: also
        document the missing svcgssd options to the nfs.conf(5) manpage
    - d/README.Ubuntu: updated with the content of the previous d/NEWS
      file
    - d/rules: re-add hardening option lost from the src:libnfsidmap to
      src:nfs-utils transition (LP: #1980095)

 -- Andreas Hasenack <email address hidden> Tue, 28 Jun 2022 10:59...

Read more...

Changed in nfs-utils (Ubuntu Kinetic):
status: In Progress → Fix Released
Changed in nfs-utils (Ubuntu Jammy):
status: Triaged → In Progress
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers