Make NFS client/server work under systemd

Bug #1312976 reported by RussianNeuroMancer on 2014-04-26
122
This bug affects 32 people
Affects Status Importance Assigned to Milestone
apparmor (Debian)
Fix Released
Unknown
apparmor (Ubuntu)
Undecided
Martin Pitt
nfs-utils (Ubuntu)
High
Martin Pitt
rpcbind (Debian)
Fix Released
Unknown
rpcbind (Ubuntu)
Undecided
Martin Pitt

Bug Description

nfs-utils can not be used with systemd due to missing systemd unit or init.d script.

tags: added: systemd-boot
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in nfs-utils (Ubuntu):
status: New → Confirmed

I had an attempt to bring back the init.d script in https://launchpad.net/ubuntu/+source/nfs-utils/1:1.2.8-6ubuntu2 but Steve Langasek didn't like this much. He'll soon split the init.d scripts in Debian as well, then we can bring them back into Ubuntu and fix booting under systemd.

summary: - nfs-utils can not be installed with systemd
+ nfs-utils needs systemd unit or init.d script
Changed in nfs-utils (Ubuntu):
status: Confirmed → Triaged
Steve Langasek (vorlon) on 2014-06-02
Changed in nfs-utils (Ubuntu):
importance: Undecided → High
assignee: nobody → Steve Langasek (vorlon)
Andy Whitcroft (apw) wrote :

Note there are three units needed.

invoke-rc.d: Unit statd.service is masked
invoke-rc.d: Unit gssd.service is masked
invoke-rc.d: Unit idmapd.service is masked

Tom H (tomh0665) wrote :

I'd posted this to ubuntu-devel-discuss but I was just looking at systemd-boot bugs and thought that I'd post it here too, with apologies for the duplication. I hope that it helps whoever puts together the real thing.

I installed the upstream nfs-utils units and adapted them with "/etc/" overrides; and I created rpcbind units.

# diff /etc/systemd/system/nfs-config.service
/lib/systemd/system/nfs-config.service
7c7
< ExecStart=/lib/systemd/scripts/nfs-utils_env.sh
---
> ExecStart=/usr/lib/systemd/scripts/nfs-utils_env.sh

# diff /etc/systemd/system/rpc-statd-notify.service
/lib/systemd/system/rpc-statd-notify.service
19c19
< ExecStart=-/sbin/sm-notify -d $SMNOTIFYARGS
---
> ExecStart=-/usr/sbin/sm-notify -d $SMNOTIFYARGS

# diff /etc/systemd/system/rpc-statd.service
/lib/systemd/system/rpc-statd.service
16,17c16,17
< PIDFile=/run/rpc.statd.pid
< ExecStart=/sbin/rpc.statd --no-notify $STATDARGS
---
> PIDFile=/var/run/rpc.statd.pid
> ExecStart=/usr/sbin/rpc.statd --no-notify $STATDARGS

# cat /lib/systemd/scripts/nfs-utils_env.sh
#!/bin/sh

NFS_CONFIG=/run/sysconfig/nfs-utils

if [ -e $NFS_CONFIG ] ; then
    rm -f $NFS_CONFIG
fi

if [ -r /etc/default/nfs-common ] ; then
    . /etc/default/nfs-common
fi

if [ -r /etc/default/nfs-kernel-server ] ; then
    . /etc/default/nfs-kernel-server
fi

RPCNFSDARGStmp="$RPCNFSDCOUNT $RPCNFSDOPTS"

mkdir -p /run/sysconfig

echo "GSSDARGS=\"\"" >> $NFS_CONFIG
echo "RPCIDMAPDARGS=\"$RPCIDMAPDARGS\"" >> $NFS_CONFIG
echo "SMNOTIFYARGS=\"\"" >> $NFS_CONFIG
echo "STATDARGS=\"$STATDOPTS\"" >> $NFS_CONFIG

echo "RPCMOUNTDARGS=\"$RPCMOUNTDOPTS\"" >> $NFS_CONFIG
echo "RPCNFSDARGS=\"$RPCNFSDARGStmp\"" >> $NFS_CONFIG
echo "SVCGSSDARGS=\"$RPCSVCGSSDOPTS\"" >> $NFS_CONFIG

[Note on RPCIDMAPDARGS: I set it to "-p /var/lib/nfs/rpc_pipefs" in
"/etc/default/nfs-common" because nfs-idmapd.service was failing
without it. The path to rpc_pipefs was defaulting to "/run/rpc_pipefs"
even though the man page says that the default is
"/var/lib/nfs/rpc_pipefs".]

# cat /lib/systemd/system/rpcbind.service
[Unit]
Description=RPC Bind
After=network.target

[Service]
Type=forking
ExecStart=/sbin/rpcbind
Restart=always

[Install]
WantedBy=rpcbind.target

[Note on rpcbind.service: I'm not sure whether this is the right unit
or whether it should have "Before=rpcbind.target multi-user.target"
and "Requires=rpcbind.target" in the "Unit" section and
"WantedBy=multi-user.target" in the "Install" section.]

# cat /lib/systemd/system/rpcbind.socket
[Unit]
Description=RPC Bind Socket

[Socket]
ListenStream=/run/rpcbind.sock

[Install]
WantedBy=sockets.target

Gabriel Mazetto (brodock) wrote :

Can't we have a solution to this, please?

My nfs-common doesn't upgrade if I'm using systemd instead of sysvinit and upstart:

Configurando nfs-common (1:1.2.8-9ubuntu1.1) ...
Failed to issue method call: Unit statd.service failed to load: No such file or directory.
invoke-rc.d: initscript statd, action "restart" failed.
dpkg: erro ao processar o pacote nfs-common (--configure):
 sub-processo script post-installation instalado retornou estado de saída de erro 6
Erros foram encontrados durante o processamento de:
 nfs-common
E: Sub-process /usr/bin/dpkg returned an error code (1)

Martin Pitt (pitti) on 2015-01-08
Changed in nfs-utils (Ubuntu):
milestone: none → ubuntu-15.02
Martin Pitt (pitti) wrote :

I've recently seen some upstream activity on the native systemd jobs (like on http://www.spinics.net/lists/linux-nfs/msg48981.html), maybe we should just use these?

Martin Pitt (pitti) on 2015-03-02
Changed in nfs-utils (Ubuntu):
milestone: ubuntu-15.02 → ubuntu-15.03
Martin Pitt (pitti) wrote :

Taking over from Steve as discussed on IRC, he doesn't find enough time for this.

These are the steps to set up a VM to test both server and client:

# set up exports
sudo apt-get install -y nfs-kernel-server
echo -e '/home *(rw,no_subtree_check)\n/var/log/ *(ro,subtree_check)' | sudo tee -a /etc/exports
sudo service nfs-kernel-server start

# set up client mounts
sudo mkdir /mnt/nfs_home /mnt/nfs_log
cat << EOF | sudo tee -a /etc/fstab
localhost:/home /mnt/nfs_home nfs defaults,nofail 0 0
localhost:/var/log /mnt/nfs_log nfs defaults,nofail 0 0
EOF
sudo reboot

# some simple tests to see that it worked
ls -l /mnt/nfs_log/wtmp /mnt/nfs_home/ubuntu/.bash*
echo hello > /mnt/nfs_home/ubuntu/world.txt
cat ~/world.txt # should give "hello"
sudo touch /mnt/nfs_log/pwned # should fail with "read-only fs"

This works fine under upstart, but as expected fails once you install systemd-sysv:

$ sudo systemctl status nfs-kernel-server
● nfs-kernel-server.service - LSB: Kernel NFS server support
   Loaded: loaded (/etc/init.d/nfs-kernel-server)
   Active: active (exited) since Mon 2015-03-02 14:42:41 CET; 1min 9s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 507 ExecStart=/etc/init.d/nfs-kernel-server start (code=exited, status=0/SUCCESS)

Mar 02 14:42:41 autopkgtest systemd[1]: Starting LSB: Kernel NFS server support...
Mar 02 14:42:41 autopkgtest nfs-kernel-server[507]: * Exporting directories for NFS kernel daemon...
Mar 02 14:42:41 autopkgtest nfs-kernel-server[507]: ...done.
Mar 02 14:42:41 autopkgtest nfs-kernel-server[507]: * Starting NFS kernel daemon
Mar 02 14:42:41 autopkgtest nfs-kernel-server[507]: * Not starting: portmapper is not running
Mar 02 14:42:41 autopkgtest systemd[1]: Started LSB: Kernel NFS server support.

and the client mounts hang:

$ ps aux|grep nfs
root 608 0.0 0.0 21544 1696 ? S 14:42 0:00 /bin/mount localhost:/var/log /mnt/nfs_log -n -t nfs -o defaults
root 610 0.0 0.0 21544 1688 ? S 14:42 0:00 /bin/mount localhost:/home /mnt/nfs_home -n -t nfs -o defaults
root 611 0.0 0.1 40192 3052 ? S 14:42 0:00 /sbin/mount.nfs localhost:/var/log /mnt/nfs_log -n -o rw
root 612 0.0 0.1 40192 3180 ? S 14:42 0:00 /sbin/mount.nfs localhost:/home /mnt/nfs_home -n -o rw

Changed in nfs-utils (Ubuntu):
assignee: Steve Langasek (vorlon) → Martin Pitt (pitti)
status: Triaged → In Progress
description: updated
Martin Pitt (pitti) wrote :

As we are quite a few upstream versions behind, I went through all systemd related upstream commits. There were a ton for adding the systemd/ directory with the units, which we should just start with as a whole and adjust to the ubuntu specifics/what the upstart jobs do: http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=tree;f=systemd

Aside from those, there were just 4 commits to utils/statd/start-statd, but it seems easier to just take the whole current file: http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=utils/statd/start-statd

Martin Pitt (pitti) wrote :

There is an ordering cycle in rpcbind which needs to be fixed first. This is because /etc/init.d/rpcbind depends on $network which in Ubuntu is implemented to actually wait for ifupdown/NM/etc. But that needs dbus, which in turn needs $remote_fs, which needs rpcbind.

Changed in rpcbind (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
status: New → In Progress
Martin Pitt (pitti) on 2015-03-03
Changed in rpcbind (Ubuntu):
status: In Progress → Fix Committed
Martin Pitt (pitti) wrote :

Finally there are two dependency cycles which we need to fix:

Found ordering cycle on basic.target/start
Found dependency on sysinit.target/start
Found dependency on apparmor.service/start
Found dependency on remote-fs.target/start
Found dependency on remote-fs-pre.target/start
Found dependency on nfs-client.target/start
Found dependency on rpc-statd-notify.service/start
Found dependency on nfs-server.service/start
Found dependency on rpc-svcgssd.service/start
Found dependency on basic.target/start
Breaking ordering cycle by deleting job apparmor.service/start
Job apparmor.service/start deleted to break ordering cycle starting with basic.target/start

Found ordering cycle on basic.target/start
Found dependency on sysinit.target/start
Found dependency on console-setup.service/start
Found dependency on remote-fs.target/start
Found dependency on remote-fs-pre.target/start
Found dependency on nfs-client.target/start
Found dependency on rpc-statd-notify.service/start
Found dependency on nfs-server.service/start
Found dependency on rpc-svcgssd.service/start
Found dependency on basic.target/start
Breaking ordering cycle by deleting job console-setup.service/start
Job console-setup.service/start deleted to break ordering cycle starting with basic.target/start

This is because these two don't have native systemd units, and their init.d script depends on $remote_fs. But remote-fs.target requires quite a lot (networking, dbus, perhaps NetworkManager, etc.), and we do want to protect those with apparmor. Also, apparmor only uses files in /, so $local_fs is sufficient even for the hypothetical "/usr on NFS" case.

That's less true for console-setup: This does use files from /usr; however, as we've never officially supported /usr on NFS, and nobody ever tested that under upstart, I'm very much inclined to just fix its init.d dependency to $local_fs, too.

Changed in apparmor (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
status: New → In Progress
Changed in console-setup (Ubuntu):
status: New → In Progress
assignee: nobody → Martin Pitt (pitti)
Changed in apparmor (Ubuntu):
status: In Progress → Fix Committed
Martin Pitt (pitti) on 2015-03-03
Changed in console-setup (Ubuntu):
status: In Progress → Fix Committed
summary: - nfs-utils needs systemd unit or init.d script
+ Make NFS client/server work under systemd
Changed in console-setup (Ubuntu):
status: Fix Committed → In Progress
Martin Pitt (pitti) wrote :

First test package available in https://launchpad.net/~pitti/+archive/ubuntu/systemd
I added the current upstream units as a big patch, and then added separate patches for our changes, so that it's easier to see what we changed; these also become easier to upstream that way.

TODO:
 - debian/nfs-common.gssd.upstart does some elaborate NEED_GSSD detection. The systemd unit just does ConditionPathExists=/etc/krb5.keytab which looks simpler; but I don't know enough about krb to say whether it's equivalent/sufficient, or whether we need the complicated parsing? Steve?

 - $NEED_STATD isn't taken into account. At the moment, rpc.statd is started unconditionally.

 - unmounting the NFS mounts hangs on shutdown

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rpcbind - 0.2.1-6ubuntu2

---------------
rpcbind (0.2.1-6ubuntu2) vivid; urgency=medium

  * Add systemd unit. Patch by Simon McVittie. (Closes: #748074, LP: #1312976)
 -- Martin Pitt <email address hidden> Tue, 03 Mar 2015 08:32:38 +0100

Changed in rpcbind (Ubuntu):
status: Fix Committed → Fix Released
Changed in rpcbind (Debian):
status: Unknown → New
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apparmor - 2.9.1-0ubuntu5

---------------
apparmor (2.9.1-0ubuntu5) vivid; urgency=medium

  * debian/apparmor.init: Replace unnecessary $remote_fs dependency with
    $local_fs. This is sufficient as during boot we don't use anything from
    /usr. It's also necessary to avoid dependency cycles when using NFS (as
    its dependencies should be covered by AppArmor). (LP: #1312976)
 -- Martin Pitt <email address hidden> Tue, 03 Mar 2015 08:54:33 +0100

Changed in apparmor (Ubuntu):
status: Fix Committed → Fix Released
Martin Pitt (pitti) wrote :

pitti2 uploaded to PPA now. This fixes the umount hang, and drops the manual modprobing. The modules get loaded automatically without that too. Neither upstream nor the Fedora package do any extra modprobing.

Changed in console-setup (Ubuntu):
milestone: none → ubuntu-15.03
Martin Pitt (pitti) wrote :

nfs-utils - 1:1.2.8-9ubuntu2pitti3 uploaded to PPA. This merely cleans up the changelog a bit and adds a proper "Forwarded:" reference to an upstreamable patch.

I now tested this on both a single machine as well as two separate VMs (one server, one client), with some /mnt/* mounts as well as /home on NFS. Works fine now.

The remaining TODO is

- $NEED_STATD isn't taken into account. At the moment, rpc.statd is started unconditionally.

But that's an optimization and thus we could do this in a followup upload.

Martin Pitt (pitti) on 2015-03-03
Changed in nfs-utils (Ubuntu):
status: In Progress → Fix Committed
Martin Pitt (pitti) on 2015-03-03
no longer affects: console-setup (Ubuntu)
Martin Pitt (pitti) on 2015-03-03
Changed in console-setup (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
status: New → In Progress
Martin Pitt (pitti) on 2015-03-05
no longer affects: console-setup (Ubuntu)
Martin Pitt (pitti) wrote :

I split out the remaining issues into bug 1428486 and bug 1428487. I'm also going to create an autopkgtest.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nfs-utils - 1:1.2.8-9ubuntu3

---------------
nfs-utils (1:1.2.8-9ubuntu3) vivid; urgency=medium

  * Add 00git-start-statd-systemd.patch: Latest start-statd script from
    1.3.2 to start rpc-statd.service under systemd.
  * Add 23-systemd-pipefs_in_run.patch: systemd: Mount rpc_pipefs in /run instead of
    /var/lib/nfs/, like in the upstart units.
  * Add 24-systemd-daemon-paths.patch: Adjust program paths in systemd units.
  * debian/nfs-kernel-server.links: Add nfs-kernel-server.service alias
    symlink, to match SysV init script.
  * Add 25-systemd-server-before-client.patch: Order NFS server before client,
    to make mounting NFS shares from localhost work reliably.
  * Add debian/nfs-utils_env.sh: Translate our /etc/default files into runtime
    configuration for nfs-config.service.
  * debian/nfs-{common,kernel-server}.install: Install systemd units.
  * debian/rules: Enable/start systemd units. (LP: #1312976)
 -- Martin Pitt <email address hidden> Thu, 05 Mar 2015 07:46:41 +0100

Changed in nfs-utils (Ubuntu):
status: Fix Committed → Fix Released
Changed in apparmor (Debian):
status: Unknown → New
Changed in apparmor (Debian):
status: New → Fix Committed
Changed in apparmor (Debian):
status: Fix Committed → Fix Released
Changed in rpcbind (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.