nfs-kernel-server does not start because of dependency failure

Bug #1590799 reported by Paul Hewlett on 2016-06-09
134
This bug affects 24 people
Affects Status Importance Assigned to Milestone
nfs-utils (Ubuntu)
Medium
Rafael David Tinoco
Trusty
Undecided
Unassigned
Xenial
Medium
Rafael David Tinoco
Yakkety
Medium
Rafael David Tinoco
Zesty
Medium
Rafael David Tinoco

Bug Description

[Impact]

 * nfs-mountd doesn't get started because of a race condition happening when rpcbind.socket is not specified as a needed service for it to start.
 * nfs-server using rpcbind.target instead of using rpcbind.socket. Target should not be used (Comment #24)

[Test Case]

 * Install nfs-kernel-server inside a xenial lxc guest and restart it until nfs-mountd doesn't start complaining on rpc error.
 * Comment #25

[Regression Potential]

 * Cons: Systemd dependencies could brake for nfs-server and nfs-mountd.
 * Pros: Patches have been accepted upstream (and tested).

[Other Info]

# Original Bug Description

Immediately after boot:

root@feynmann:~# systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Jun 09 14:35:47 feynmann systemd[1]: Dependency failed for NFS server and services.
Jun 09 14:35:47 feynmann systemd[1]: nfs-server.service: Job nfs-server.service/start failed

root@feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2016-06-09 14:35:47 BST; 7min ago
  Process: 1321 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=1/FAILURE)

Jun 09 14:35:47 feynmann systemd[1]: Starting NFS Mount Daemon...
Jun 09 14:35:47 feynmann rpc.mountd[1321]: mountd: could not create listeners
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Control process exited, code=exited
Jun 09 14:35:47 feynmann systemd[1]: Failed to start NFS Mount Daemon.
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Unit entered failed state.
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.

root@feynmann:~# systemctl list-dependencies nfs-kernel-server
nfs-kernel-server.service
● ├─auth-rpcgss-module.service
● ├─nfs-config.service
● ├─nfs-idmapd.service
● ├─nfs-mountd.service
● ├─proc-fs-nfsd.mount
● ├─rpc-svcgssd.service
● ├─system.slice
● ├─network.target
● └─rpcbind.target
● └─rpcbind.service

root@feynmann:~# systemctl list-dependencies nfs-mountd.service
nfs-mountd.service
● ├─nfs-config.service
● ├─nfs-server.service
● ├─proc-fs-nfsd.mount
● └─system.slice
root@feynmann:~#

root@feynmann:~# lsb_release -rd
Description: Ubuntu 16.04 LTS
Release: 16.04

root@feynmann:~# apt-cache policy nfs-kernel-server
nfs-kernel-server:
  Installed: 1:1.2.8-9ubuntu12
  Candidate: 1:1.2.8-9ubuntu12
  Version table:
 *** 1:1.2.8-9ubuntu12 500
        500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status

Additional comments:

    1. There seems to be a circular dependency between nfs-mountd and nfs-kernel-server
    2. I can get it working by changing the AFter,Requires in /lib/ssystemd/system/nfs-{mountd|server}.service files. I have managed to get nfs-kernel-server to start but not nfs-mountd.
    3. /usr/lib/systemd/scripts/nfs-utils_env.sh references /etc/sysconfig/nfs which is Centos/RedHat location of this file. Also /etc/default/nfs does not exist. (possibly unrelated to this bug)
    4. A file "/lib/systemd/system/-.slice" exists. this file prevents execution of 'ls *' or 'grep xxx *' commands in that directory. I am unsure whether this is intended by the systemd developers but it is unfriendly when investigating this bug.

Attempted solution:

    1. Edit /lib/systemd/system/nfs-server.service (original lines are commented out:

[Unit]
Description=NFS server and services
DefaultDependencies=no
Requires=network.target proc-fs-nfsd.mount rpcbind.target
# Requires=nfs-mountd.service
Wants=nfs-idmapd.service

After=local-fs.target
#After=network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
After=network.target proc-fs-nfsd.mount rpcbind.target
After=nfs-idmapd.service rpc-statd.service
#Before=rpc-statd-notify.service
Before=nfs-mountd.service rpc-statd-notify.service
...

followed by a systemctl daemon-reload and a reboot.

This results in nfs-kernel-server starting correctly but nfs-mountd not so. However starting nfs-mountd manually after reboot is successful:

root@feynmann:~# systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2016-06-09 15:07:23 BST; 1min 25s ago
  Process: 1391 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
  Process: 1384 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 1391 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/nfs-server.service

Jun 09 15:07:23 feynmann systemd[1]: Starting NFS server and services...
Jun 09 15:07:23 feynmann systemd[1]: Started NFS server and services.

root@feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: inactive (dead)

root@feynmann:~# systemctl start nfs-mountd.service

root@feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: active (running) since Thu 2016-06-09 15:09:02 BST; 3s ago
  Process: 2044 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
 Main PID: 2046 (rpc.mountd)
    Tasks: 1
   Memory: 904.0K
      CPU: 12ms
   CGroup: /system.slice/nfs-mountd.service
           └─2046 /usr/sbin/rpc.mountd --manage-gids

Jun 09 15:09:02 feynmann systemd[1]: Starting NFS Mount Daemon...
Jun 09 15:09:02 feynmann rpc.mountd[2046]: Version 1.2.8 starting
Jun 09 15:09:02 feynmann systemd[1]: Started NFS Mount Daemon.

Enabling nfs-mountd.service (systemctl enable nfs-mountd.service) has no effect in this case.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: nfs-kernel-server 1:1.2.8-9ubuntu12 [modified: lib/systemd/system/nfs-server.service]
ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Thu Jun 9 14:38:58 2016
InstallationDate: Installed on 2016-06-08 (1 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
ProcEnviron:
 SHELL=/bin/bash
 TERM=linux
 PATH=(custom, no user)
 LANG=en_GB.UTF-8
 LANGUAGE=en_GB:en
SourcePackage: nfs-utils
UpgradeStatus: No upgrade log present (probably fresh install)

Paul Hewlett (phewlett76) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in nfs-utils (Ubuntu):
status: New → Confirmed
Changed in nfs-utils (Ubuntu):
importance: Undecided → High
CDSRV TechSupport (cdsrv) wrote :

the workaround described earlier here does not seem to work.

starting with a clean install, a simple export, and no changes in /etc/default/nfs-*

the first error message in the journal was " rpc.nfsd: invalid option -- 'v' "

this worked:
> systemctl add-wants multi-user.target rpcbind.service
> systemctl restart nfs-kernel-server

then add the restart to /etc/rc.local and the system will reboot with nfs services running - otherwise it will not start, instead showing "Dependency failed for NFS server and services" in the journal

CDSRV TechSupport (cdsrv) wrote :

system updated and problem returned in 4.4.0-31

fixed by removing $RPCNFSDARGS from line 29 in

/lib/systemd/system/nfs-server.service

replacing with debug flag, this works through reboot without any restart required in rc.local

ExecStart=/usr/sbin/rpc.nfsd -d

RPCNFSDARGS was not defined in /etc/default/nfs-kernel-server
and it wasn't read from there when set - defining it as "-d" did not enable the option or allow the service to start.

Karl-Heinz Fischbach (c38sgzkz) wrote :

Hi,
this seems still an issue in ubuntu 16.04

ii nfs-common 1:1.2.8-9ubuntu12 amd64 NFS support files common to client and server
ii nfs-kernel-server 1:1.2.8-9ubuntu12 amd64 support for NFS kernel server

systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Aug 29 07:26:17 f42252s3 systemd[1]: Dependency failed for NFS server and services.
Aug 29 07:26:17 f42252s3 systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.

Any help to solve it?

Malcolm Scott (malcscott) wrote :

It appears that nfs-mountd.service is missing dependencies:

Wants=rpcbind.service
After=rpcbind.service

(as hinted above in comment 3, but we need After= too).

Paul Hewlett (phewlett76) wrote :

I lost the system disk on my server and re-installed from scratch. I did a apt update/upgrade before installing nfs-kernel-server and now everything seems OK.

root@server:~# systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2016-09-03 11:59:27 BST; 5min ago
  Process: 3324 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
  Process: 3317 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 3324 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/nfs-server.service

Sep 03 11:59:27 server systemd[1]: Starting NFS server and services...
Sep 03 11:59:27 server systemd[1]: Started NFS server and services.

root@server:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: active (running) since Sat 2016-09-03 11:59:27 BST; 3min 5s ago
  Process: 3306 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
 Main PID: 3315 (rpc.mountd)
    Tasks: 1
   Memory: 3.3M
      CPU: 8ms
   CGroup: /system.slice/nfs-mountd.service
           └─3315 /usr/sbin/rpc.mountd --manage-gids

Sep 03 11:59:27 server systemd[1]: Starting NFS Mount Daemon...
Sep 03 11:59:27 server rpc.mountd[3315]: Version 1.2.8 starting
Sep 03 11:59:27 server systemd[1]: Started NFS Mount Daemon.

root@server:~# lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04

and I can access the share from my laptop successfully.

It appears that 16.04.1 has fixed this bug

Thomas Runge (thomas-runge) wrote :

I've recently upgraded from Ubuntu Server 14.04 to 16.04.1 and I am experiencing this issue right now.

I tried to purge and reinstall nfs-kernel-server, but the issue came back at once. I was only able to work around this issue by adding

systemctl restart nfs-kernel-server

in /etc/rc.local.

So, from my perspective the issue might be gone for a clean install of Ubuntu 16.04.1, but not for systems having been upgraded from an older release.

linuxball (linuxball) wrote :

I confirm that this bug also affects my Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-51-lowlatency i686) system (with all updates as of today, even those from xenial-proposed).

However, in my case the failure (rpc.mountd[<number>]: mountd: could not create listeners) occured only in about 50% of all boot processes and the system booted without this failure elsewise.

Using journalctl I could narrow the error down: It is a race condition: The error occurs when rpcbind is not yet ready when rpc.mountd is started and it does not occur when rpcbind is already ready at the time rpc.mountd starts.

So Malcolm Scott gives the correct solution (add dependency and order information to systemd) in comment #6. I created the snippet /etc/systemd/system/nfs-mountd.service.d/rpcbind-dep.conf:

[Unit]
Wants=rpcbind.service
After=rpcbind.service

and updated the initramfs (update-initramfs -u) and now the bug has gone. For a clean bug fix the above two lines should be added to /lib/systemd/system/nfs-mountd.service by the maintainers of the nfs-kernel-server package.

linuxball (linuxball) wrote :

Addendum: Link to Radhat's "Storage Administration Guide" which states that "Because RPC-based services rely on rpcbind to make all connections with incoming client requests, rpcbind must be available BEFORE any of these services start."

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/s2-nfs-methodology-portmap.html

Why has noone updated the package by adding the following to /lib/systemd/system/nfs.mountd.service:
Wants=rpcbind.service
After=rpcbind.service

I confirmed that this solves the issue.

So why isn't the package updated yet?

dave (docjackly) wrote :

Same here! Running 16.04.1 (with all packages updated) and nfs still fails to start after reboot and returns the familiar dependency error message.

I've the workaround suggest by Thomas Runge with great success. So thanks for that Thomas!

Strangely, this bug is open for more than 7 months now... when is it going to be fixed in the core?

CDSRV TechSupport (cdsrv) wrote :

we had some hacks running so the service starts when the system comes up, and a recent update to the latest 16.04 (4.4.0-57-generic #78) broke them.

current workarounds, placed in /etc/rc.local :

##
systemctl add-wants multi-user.target rpcbind.service
service nfs-kernel-server restart
systemctl restart rpcbind
##

---------------------------------------------------

# run the commands manually to enable the services, it may be necessary to issue 'systemctl daemon-reload' after adding the target.

# reboot to check that the service starts. some of these may not be needed every startup but are there to recover the config after an upgrade breaks the workaround.

# check
service nfs-kernel-service status

# for debugging:

# set debug flags
rpcdebug -m rpc -s all
rpcdebug -m nfsd -s all

# tail the logfile
tail -f /var/log/kern.log

# clear debug flags:
rpcdebug -m rpc -c all
rpcdebug -m nfsd -c all

Changed in nfs-utils (Ubuntu):
assignee: nobody → Rafael David Tinoco (inaddy)
status: Confirmed → In Progress
Rafael David Tinoco (inaddy) wrote :

Okay, sorry for the delay. It took me sometime to review all SysV, Upstart scripts and Systemd unit files. What "linuxball" said is correct, we're missing rpcbind dependency for nfs-mountd.service.

# nfs-server.service

[Unit]
...
Requires= network.target proc-fs-nfsd.mount rpcbind.target
Requires= nfs-mountd.service
Wants= nfs-idmapd.service

After= local-fs.target
After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
After= nfs-idmapd.service rpc-statd.service
Before= rpc-statd-notify.service

Wants=auth-rpcgss-module.service
After=rpc-gssd.service rpc-svcgssd.service

Before=remote-fs-pre.target

Wants=nfs-config.service
After=nfs-config.service

As it stands, "rpcbind.target is being put as a dependency and to run before "nfs-server.service" but not before "nfs-mountd.service":

[Unit]
...
Requires=proc-fs-nfsd.mount
After=proc-fs-nfsd.mount
After=network.target local-fs.target
BindsTo=nfs-server.service

Wants=nfs-config.service
After=nfs-config.service

Since the services run in parallel there is an window of opportunity for "nfs-mountd.service" to be started before "rpcbind.target".

Only problem is that rpcbind.target is for synchronization with SysV scripts:

rpcbind.target
The portmapper/rpcbind pulls in this target and orders itself before it, to indicate its availability. systemd automatically adds dependencies of type After= for this target unit to all SysV init script service units with an LSB header referring to the "$portmap" facility.

Right now we have:

system/nfs-server.service:Requires= ... rpcbind.target
system/nfs-server.service:After= ... rpcbind.target ...
system/rpc-statd.service:Requires= ... rpcbind.target
system/rpc-statd.service:After= ... rpcbind.target

So the correct fix would be:

Wants= ... rpcbind.target
After= ... rpcbind.target

For all NFS daemons depending on rpcbind to be fully operational, since the systemd generators would take also care of SysV script dependencies (allegedly if rpcbind is started through SysV generator).

The thing is that, this upstream discussion has taken place:

https://patchwork.kernel.org/patch/8976701/

I think I should move on and "cherry-pick" this change as well. Upstream "Linux NFS" changed the dependency from "target" to "service" - which makes perfectly sense based on the explanation from Neil Brown.

I'm providing a PPA soon and will ask for feedback and, at the same time, I'll attach this small fix as a debdiff proposal for sponsors team to upload it to -propose if your feedback is good. I'd like to ask you to reproduce the issue and to make sure issue is gone with the package I'll provide.

Sorry for the amount of time this bug was opened without a fix. I think it never gained much traction because the workaround as simple so people had stick with the workaround you guys provided.
Jeroen, if you want to understand how Ubuntu development works, specifically for fixes you can read this Wiki: https://wiki.ubuntu.com/StableReleaseUpdates.

Will get back to this today.

Rafael David Tinoco (inaddy) wrote :

Just one thing that I'm trying to figure out:

nfs-mountd.service already has:

[Unit]
...
BindsTo=nfs-server.service

Which means, it "requires" nfs-server.service AND it is stopped as soon nfs-server.service is stopped. Judging by this upstream commit: http://paste.ubuntu.com/23940610/

"""
So while I think it is valid to use rpcbind.target for ordering
(before/after) it shouldn't be used for dependencies (Wants/Requires).
The rpcbind.target file included in systemd does not "Require" the
actual service, so requiring rpcbind.target itself is pointless.
"""

This race condition might be happening because of the ".target" choice for rpcbind dependency - on nfs-server - instead of the "lack" of dependency in nfs-mountd. My only doubt is that developer preferred to depend on "rpcbind.socket" instead of "rpcbind.service". I'll go ahead and give it a try by testing ".socket", based on upstream commits, and ask for you to test it.

Thank you
Rafael

Rafael David Tinoco (inaddy) wrote :

Could anyone able to reproduce this issue check if the following PPA:

https://launchpad.net/~inaddy/+archive/ubuntu/lp1590799

Solves the problem ?

Thank you
Rafael

Changed in nfs-utils (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Rafael David Tinoco (inaddy)
Rafael David Tinoco (inaddy) wrote :

I'm attaching the debdiffs just "in case". After I get confirmation that the packages from PPA solves the issue I'll go on and ask for sponsorship.

Changed in nfs-utils (Ubuntu Xenial):
importance: Undecided → High
Malcolm Scott (malcscott) wrote :

Thanks, Rafael -- I'll put together an environment in which to test the packages (I'd rather not reboot my live NFS servers).

However, I think nfs-mountd.service still needs tweaking. rpc.mountd (if using NFSv2 or v3) needs to communicate with rpcbind on startup to register services. Currently nfs-mountd.service has

   BindsTo=nfs-server.service

but as far as I can tell this does NOT imply an ordering (it's Requires= plus the additional behaviour of stopping when another service stops; I don't think it covers After=).

So, I suspect you may also need to add to nfs-mountd.service:

   After=nfs-server.service

Malcolm Scott (malcscott) wrote :

Furthermore, arguably every RPC service (rpc-statd.service, nfs-mountd.service, etc.) should explicitly have

   BindsTo=rpcbind.service
   After=rpcbind.service

because they literally are bound to rpcbind. If rpcbind is stopped/restarted, every RPC service must also be stopped/restarted so as to rebind (as stated in rpcbind(8)). Depending on rpcbind.socket seems wrong when rpcbind must itself be running for any RPC service to start serving requests.

Or perhaps just put that in nfs-server.service and have everything transitively bind to rpcbind via that unit.

I had this issue when running under an LXD/LXC container. If you need I
can deploy it out and test. Just let me know.

On Mon, Feb 6, 2017 at 1:00 PM, Rafael David Tinoco <
<email address hidden>> wrote:

> ** Patch added: "yakkety_nfs-utils_1.2.8-9.2ubuntu2.debdiff"
> https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/
> 1590799/+attachment/4814234/+files/yakkety_nfs-utils_1.2.8-
> 9.2ubuntu2.debdiff
>
> ** Changed in: nfs-utils (Ubuntu Xenial)
> Importance: Undecided => High
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1590799
>
> Title:
> nfs-kernel-server does not start because of dependency failure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/
> 1590799/+subscriptions
>

Rafael David Tinoco (inaddy) wrote :

Malcolm,

Your first comment makes sense: I really don't know if order will be guaranteed without After= for when we're using BindsTo :\. I think it might be safer for us to use it since it will cause no harm having it (i'll have to include this upstream somehow so we don't break our future releases).

NOW, I DO agree with your statement on rpcbind. I thought it was weird creating dependency on the socket itself. I think the idea was to make services startup "faster", the developer "assumed" the socket being ready would make that happen. For rpcbind it doesn't make much sense since it will take care of all port orchestration for the rest of services and we have to have the service up and running, not only its socket.

I'll make those changes and will update PPA. For now you can keep with this version - since it will, at least, remove the ".target" logic. I'll do it tonight.

Tks Malcom, please be sure to reproduce it before checking if it works (so we can "verify" it later, when the SRU is done).

o/

The attachment "xenial_nfs-utils_1.2.8-9ubuntu13.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Rafael David Tinoco (inaddy) wrote :
Download full text (3.6 KiB)

Summarizing

From systemd.special we have:

sockets.target:
A special target unit that sets up all socket units (see systemd.socket(5) for details) that shall be active after boot. Services that can be socket-activated shall add Wants= dependencies to this unit for their socket unit during installation. This is best configured via a WantedBy=sockets.target in the socket unit's "[Install]" section.

Lets see who is starting rpcbind.socket:

[Install]
WantedBy=sockets.target

Unit file will be linked as "wanted" by "sockets.target".

$ systemctl list-dependencies sockets.target
sockets.target
● ├─acpid.socket
● ├─apport-forward.socket
● ├─dbus.socket
● ├─dm-event.socket
● ├─rpcbind.socket
... <many others>

$ systemctl --reverse list-dependencies sockets.target
sockets.target
● └─basic.target
● └─multi-user.target
● └─graphical.target

basic.target starts the sockets.target.
sockets.target start rpcbind.socket.
rpcbind.target doesn't start rpcbind.service.

Lets who starts rpcbind.service:

[Install]
Also=rpcbind.socket

rpcbind.{socket,service} are enabled/disabled together (not started).

$ systemctl list-dependencies rpcbind.service
rpcbind.service
● ├─rpcbind.socket
● ├─system.slice
● ├─remote-fs-pre.target
● └─rpcbind.target
    └─...

rpcbind.service depends on rpcbind.target (not with my patch) AND rpcbind.socket (started by basic.target with the sockets.target). Lets see rpcbind.target (its a special target made by a generator, its unit file doesn't contain anything):

rpcbind.target:
The portmapper/rpcbind pulls in this target and orders itself before it, to indicate its availability. systemd automatically adds dependencies of type After= for this target unit to all SysV init script service units with an LSB header referring to the "$portmap" facility.

So, systemd puts an After=rpcbind.target in all SysV scripts with LSB header containing "$portmap" as facility (insserv style):

$ systemctl status rpcbind.target
● rpcbind.target - RPC Port Mapper
   Loaded: loaded (/etc/insserv.conf.d/rpcbind; static; vendor preset: enabled)
  Drop-In: /run/systemd/generator/rpcbind.target.d
           └─50-hard-dependency-rpcbind-$portmap.conf

/etc/insserv.conf.d/rpcbind contains "rpcbind".
/run/systemd/generator/rpcbind.service.d/50-rpcbind-\$portmap.conf contains:

[Unit]
Wants=rpcbind.target
Before=rpcbind.target

So rpcbind.target doesn't start "rpcbind.service" and any "rpcbind.target" usage seems broken to me. rpcbind was ever started by nfs-{kernel-server,mountd}.service (since they were relying on the rpcbind.target). rpcbind (the portmap) was being activated by a connection to the rpc socket.

But NOT the NETWORK socket, the LOCAL socket:

[Socket]
ListenStream=/run/rpcbind.sock

You can test this by stopping the "rpcbind.service" and trying:

$ rpcinfo -T tcp 127.0.0.1
rpcinfo: can't contact rpcbind: RPC: Remote system error - Connection refused

$ NETPATH=/run/rpcbind.sock rpcinfo
   program version netid address service owner
    100000 4 tcp6 ::.0.111 portmapper superuser
    100000 3 tcp6 ::.0.111 portmapper superuser
    100000 4 udp6 ::.0.11...

Read more...

Rafael David Tinoco (inaddy) wrote :

Spyderdyne,

Tks for letting me know about the container situation. I this it is easier to reproduce it there. With this tip I was able to reproduce here at will. I first started getting this in a Xenial machine:

$ systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Feb 07 01:46:49 nfsstartxen systemd[1]: Dependency failed for NFS server and services.
Feb 07 01:46:49 nfsstartxen systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.

AND (after changing rpcbind.target to rpcbind.socket in the unit files):

$ systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/etc/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2017-02-07 01:55:39 UTC; 39s ago
  Process: 244 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=1/FAILURE)

Feb 07 01:55:39 nfsstartxen systemd[1]: Starting NFS Mount Daemon...
Feb 07 01:55:39 nfsstartxen systemd[1]: nfs-mountd.service: Control process exited, code=exited status=1
Feb 07 01:55:39 nfsstartxen systemd[1]: Failed to start NFS Mount Daemon.
Feb 07 01:55:39 nfsstartxen systemd[1]: nfs-mountd.service: Unit entered failed state.
Feb 07 01:55:39 nfsstartxen systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.

This is good because I expected the error to continue after my last comment - since enabling the socket is EXACTLY what I showed we had before.

Malcolm,

You were right, for the "nfs-mountd.service" I HAD TO have:

Wants=nfs-config.service rpcbind.service
After=nfs-config.service rpcbind.service

because:

BindsTo=nfs-server.service in "nfs-mountd.service"

Wasn't enough to guarantee the order if "nfs-server.service" contains:

Wants=rpcbind.service
After= rpc-statd.service

Looks like we have to force the order and I'll have to report this upstream pointing out this bug, together with this fix.

Will get back here tomorrow.

Rafael David Tinoco (inaddy) wrote :

Just submitted a new fix into:

https://launchpad.net/~inaddy/+archive/ubuntu/lp1590799

It will be ready soon.
It fixes the issue for me (since I was able to reproduce it).

Based on the SRU guidelines: https://wiki.ubuntu.com/StableReleaseUpdates

I'll first check with upstream possibility to get this fixed there and backport it appropriately.

Marat Khalili (mkh-t) wrote :

Was just hit by this bug on two production servers. Is it possible to apply a fix manually? For some reason I don't see a diff in https://launchpad.net/~inaddy/+archive/ubuntu/lp1590799 .

I have replaced rpcbind.target with rpcbind.service in Requires and After lines of /etc/systemd/system/multi-user.target.wants/nfs-server.service , will this be sufficient? I don't have any other nfs-*.service files, including nfs-mountd.service, in my /etc directory.

Rafael David Tinoco (inaddy) wrote :

Marat: http://paste.ubuntu.com/23948501/ this is what you want.

Rafael David Tinoco (inaddy) wrote :

Using ".socket" instead of ".service" is okay. Seems that rpcbind is accessed from local socket, controlled by systemd. This is what I'll propose upstream:

http://paste.ubuntu.com/23949146/

Rafael David Tinoco (inaddy) wrote :

Proposed upstream fix:

http://marc.info/?l=linux-nfs&m=148649501611915&w=2

Waiting for upstream a bit before moving on with Xenial and Yakkety proposals.

Marat Khalili (mkh-t) wrote :

Rafael David Tinoco (inaddy): thank you very much for the diff! Have applied it to live nfs-mountd.service, nfs-server.service, and rpc-statd.service in /lib/systemd/system, will report result of the next reboot here if it happens before official fix arrives for Xenial.

Rafael David Tinoco (inaddy) wrote :

Commit got into master branch upstream:

commit 907426b00bdcd69d9a56ac1870990e8ae8c6fe9f
Author: Rafael David Tinoco <email address hidden>
Date: Wed Feb 15 10:26:55 2017 -0500

    systemd: Fix nfs-mountd dependency on rpcbind

    Following commit 91da135f - it replaced "rpcbind.target" by "rpcbind.socket" in
    some unit files - "rpcbind.socket" should also be added to "nfs-mountd.service"
    as a dependency to avoid race conditions.

    Usually "rpcbind.socket" is either started as a "sockets.target" dependency, or
    as a dependency for "nfs-server.service", when unit files include it in
    "BindsTo" or "After". Unfortunately there is a possilibility to have
    "nfs-mountd.service" started when the rpcbind socket is not yet created:

    systemd[1]: Starting NFS Mount Daemon...
    systemd[1]: nfs-mountd.service: Control process exited, code=exited status=1
    systemd[1]: Failed to start NFS Mount Daemon.
    systemd[1]: nfs-mountd.service: Unit entered failed state.
    systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.

    Nowadays "nfs-mountd.service" uses "BindTo" directive to "nfs-server.service".
    That, per se, doesn't guarantee ordering for NFS server to start rpcbind and for
    nfs-mountd to depend on it.

    https://bugs.launchpad.net/bugs/1590799

    Reviewed-by: NeilBrown <email address hidden>
    Signed-off-by: Rafael David Tinoco <email address hidden>
    Signed-off-by: Steve Dickson <email address hidden>

Now I can do the SRU and provide the fix proposal.

tags: added: sts sts-sponsor sts-sru
description: updated
Changed in nfs-utils (Ubuntu):
importance: High → Medium
Changed in nfs-utils (Ubuntu Xenial):
importance: High → Medium
tags: removed: sts-sru
tags: added: sts-sru
Rafael David Tinoco (inaddy) wrote :

Waiting for sponsorship.

Changed in nfs-utils (Ubuntu Yakkety):
status: New → In Progress
assignee: nobody → Rafael David Tinoco (inaddy)
importance: Undecided → Medium
tags: added: sts-sru-needed
removed: sts-sru
Marat Khalili (mkh-t) wrote :

Can confirm that it booted successfully (once) with patch live-applied in a system that earlier experienced this bug.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nfs-utils - 1:1.2.8-9.2ubuntu2

---------------
nfs-utils (1:1.2.8-9.2ubuntu2) zesty; urgency=medium

  * Fixing nfs-mountd dependency on rpcbind (race condition) (LP: #1590799)
    by adding "rpcbind.socket" to "nfs-mountd.service" as a dependency
    to avoid race conditions:
    - Add systemd-Fix-nfs-mountd-dependency-on-rpcbind.patch
    - Add systemd-unit-files-fix-up-dependencies-on-rpcbind.patch

 -- Rafael David Tinoco <email address hidden> Fri, 17 Mar 2017 12:19:53 +0100

Changed in nfs-utils (Ubuntu Zesty):
status: In Progress → Fix Released
Louis Bouchard (louis) wrote :

Adding a Trusty task & marking it invalid so we know that it doesn't concern Trusty (no systemd there)

Changed in nfs-utils (Ubuntu Trusty):
status: New → Invalid
Louis Bouchard (louis) wrote :

Updated debdiff with version change ( .13 collided with a prevously uploaded pkg)

tags: removed: sts-sponsor

Hello Paul, or anyone else affected,

Accepted nfs-utils into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nfs-utils/1:1.2.8-9.2ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nfs-utils (Ubuntu Yakkety):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in nfs-utils (Ubuntu Xenial):
status: In Progress → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Paul, or anyone else affected,

Accepted nfs-utils into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nfs-utils/1:1.2.8-9ubuntu12.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Paul Hewlett (phewlett76) wrote :

Hello Brian

I followed the links you gave for testing this (my first time)

Repeating the commands I originally did in the bug report after applying the proposed packaged shows:

 root@coulomb:~# dmesg | grep nfs
[ 3.131507] systemd[1]: nfs-blkmap.service: Job nfs-blkmap.service/start failed with result 'dependency'.
[ 3.189403] Installing knfsd (copyright (C) 1996 <email address hidden>).
[ 5.115023] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
root@coulomb:~# systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor prese
   Active: active (exited) since Sun 2017-04-02 08:58:33 BST; 4min 38s ago
  Process: 929 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/
  Process: 928 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS
 Main PID: 929 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

Apr 02 08:58:33 coulomb systemd[1]: Starting NFS server and services...
Apr 02 08:58:33 coulomb systemd[1]: Started NFS server and services.
root@coulomb:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset
   Active: active (running) since Sun 2017-04-02 08:58:33 BST; 4min 55s ago
  Process: 921 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, statu
 Main PID: 927 (rpc.mountd)
   CGroup: /system.slice/nfs-mountd.service
           └─927 /usr/sbin/rpc.mountd --manage-gids

Apr 02 08:58:33 coulomb systemd[1]: Starting NFS Mount Daemon...
Apr 02 08:58:33 coulomb rpc.mountd[927]: Version 1.2.8 starting
Apr 02 08:58:33 coulomb systemd[1]: Started NFS Mount Daemon.
root@coulomb:~# systemctl list-dependencies nfs-kernel-server
nfs-kernel-server.service
● ├─auth-rpcgss-module.service
● ├─nfs-config.service
● ├─nfs-idmapd.service
● ├─nfs-mountd.service
● ├─proc-fs-nfsd.mount
● ├─rpc-svcgssd.service
● ├─rpcbind.socket
● ├─system.slice
● └─network.target
root@coulomb:~# systemctl list-dependencies nfs-mountd.service
nfs-mountd.service
● ├─nfs-config.service
● ├─nfs-server.service
● ├─proc-fs-nfsd.mount
● └─system.slice
root@coulomb:~# lsb_release -rd
Description: Ubuntu 16.04.2 LTS
Release: 16.04
root@coulomb:~# apt-cache policy nfs-kernel-server
nfs-kernel-server:
  Installed: 1:1.2.8-9ubuntu12.1
  Candidate: 1:1.2.8-9ubuntu12.1
  Version table:
 *** 1:1.2.8-9ubuntu12.1 400
        400 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.2.8-9ubuntu12 500
        500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

which shows the bug as fixed.

This is for Xenial.

Paul Hewlett (phewlett76) wrote :

I just noticed that the nfs.blkmap service now hows a failed dependency.

Paul Hewlett (phewlett76) wrote :

I have set tag to verification-done. Bit worried abouth the nfs.blkmap failed dependency shown in dmesg output but I assume that this is a separate issue.

Rafael David Tinoco (inaddy) wrote :

I have tested both and found no apparent race condition. Since I'm the second one, I will flag this as verification-done. Paul, the nfs.blkmap is not in even in the same systemd unit files we are working on.

nfs.blkmap is declared for me in:

nfs-blkmap.service:Requisite=nfs-blkmap.target
nfs-blkmap.service:After=nfs-blkmap.target
nfs-blkmap.service:PIDFile=/var/run/blkmapd.pid
nfs-blkmap.service:ExecStart=/usr/sbin/blkmapd $BLKMAPDARGS

nfs-blkmap.target:Description= PNFS blkmaping enablement.
nfs-blkmap.target:# If this target is enabled, then blkmapd will be started

nfs-client.target:Wants=nfs-blkmap.service
nfs-client.target:After=nfs-blkmap.service

being responsible for blkmapd daemon (for pNFS).

blkmapd can be (re)started through nfs-utils.service if blkmapd.target is enabled. It can also be started by nfs-client.target, if blkmapd.target is enabled.

[ 3.131507] systemd[1]: nfs-blkmap.service: Job nfs-blkmap.service/start failed with result 'dependency'.

This looks like a leftover from some other tasks/cmds you made. If you think its needed, this should be handled in another bug (specially because its unrelated to the race itself AND this bug is already in verification phase).

I'm marking this as verification done.

tags: added: verification-done
removed: verification-needed
Johan Hagman (joaxel) on 2017-04-10
Changed in nfs-utils (Ubuntu Xenial):
status: Fix Committed → Fix Released
Rafael David Tinoco (inaddy) wrote :

Johan why are you changing status for this bug ? It has not been released yet. The status is changed automatically when the package moves from -proposed to -updates.

Changed in nfs-utils (Ubuntu Xenial):
status: Fix Released → Fix Committed
Johan Hagman (joaxel) wrote :

I'm sorry, my mistake,clicked on the wrong page.

Robie Basak (racb) wrote :

I see confirmation of 1:1.2.8-9ubuntu12.1 from xenial-proposed, but I don't see the version from yakkety-proposed confirmed anywhere. So I'll release Xenial but defer releasing Yakkety. Please can someone test the package from yakkety-proposed and confirm what you tested and the package version you tested in a comment?

tags: added: verification-done-xenial verification-needed
removed: verification-done
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nfs-utils - 1:1.2.8-9ubuntu12.1

---------------
nfs-utils (1:1.2.8-9ubuntu12.1) xenial; urgency=medium

  * Fixing nfs-mountd dependency on rpcbind (race condition) (LP: #1590799)
    by adding "rpcbind.socket" to "nfs-mountd.service" as a dependency
    to avoid race conditions:
    - Add systemd-Fix-nfs-mountd-dependency-on-rpcbind.patch
    - Add systemd-unit-files-fix-up-dependencies-on-rpcbind.patch

 -- Rafael David Tinoco <email address hidden> Wed, 29 Mar 2017 10:50:52 +0200

Changed in nfs-utils (Ubuntu Xenial):
status: Fix Committed → Fix Released
Rafael David Tinoco (inaddy) wrote :

Hello Robie,

From comment #48: "I have tested both and found no apparent race condition". Sorry for not making myself clear, both here means Yakkety and Xenial. I'm not facing the startup race condition in Xenial (as I was) AND I never faced in Yakkety (likely because the difference of services being started and intermittent behavior for this race). So my verification was to guarantee that the services would come up appropriately on both for a reasonable amount of times (several reboots).

Thank you for publishing it.

Best,
Rafael

Kuba (jakub-nowacki) wrote :

I'm still seeing this issue while the package is installed.

~$ apt-cache policy nfs-kernel-server
nfs-kernel-server:
  Installed: 1:1.2.8-9ubuntu12.1
  Candidate: 1:1.2.8-9ubuntu12.1
  Version table:
 *** 1:1.2.8-9ubuntu12.1 500
        500 http://nl3.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.2.8-9ubuntu12 500
        500 http://nl3.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"

~$ cat /lib/systemd/system/nfs-mountd.service
[Unit]
Description=NFS Mount Daemon
DefaultDependencies=no
Requires=proc-fs-nfsd.mount
After=proc-fs-nfsd.mount
After=network.target local-fs.target
After=rpcbind.socket
BindsTo=nfs-server.service

Wants=nfs-config.service
After=nfs-config.service

[Service]
EnvironmentFile=-/run/sysconfig/nfs-utils
Type=forking
ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS

CDSRV TechSupport (cdsrv) wrote :

after updating (a copy of) a working system to the latest release (1:1.2.8-9ubuntu12.1), nfs-kernel-service was in a broken state with the same errors.

further testing is needed.

As part of a recent change in the Stable Release Update verification policy we would like to inform that for a bug to be considered verified for a given release a verification-done-$RELEASE tag needs to be added to the bug where $RELEASE is the name of the series the package that was tested (e.g. verification-done-xenial). Please note that the global 'verification-done' tag can no longer be used for this purpose.

Thank you!

Thanks!

On Tue, Jun 27, 2017 at 11:24 AM, Łukasz Zemczak <<email address hidden>
> wrote:

> As part of a recent change in the Stable Release Update verification
> policy we would like to inform that for a bug to be considered verified
> for a given release a verification-done-$RELEASE tag needs to be added
> to the bug where $RELEASE is the name of the series the package that was
> tested (e.g. verification-done-xenial). Please note that the global
> 'verification-done' tag can no longer be used for this purpose.
>
> Thank you!
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1590799
>
> Title:
> nfs-kernel-server does not start because of dependency failure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/
> 1590799/+subscriptions
>

Rafael David Tinoco (inaddy) wrote :

Hello everyone,

Kuba,

Could you please show more details on what is happening in your case ? Could you follow what I did in my previous comments, to show more logs from the nfs service(s) failure(s) ? Could you reproduce the issue - the startup problem - and show me a "systemctl status" on every possible nfs {service,target} so I can relate them ? There is a chance that what you're facing isn't, necessarily, the same problem.

Best,
Rafael

Kuba (jakub-nowacki) wrote :

Hi Rafael,

Below requested output, seems both services are dead and there are no log indication on what happened:

kuba@kubadesktop:~$ systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
kuba@kubadesktop:~$ systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: inactive (dead)

I've also checked dmesg for any hints, seems my problem is also related to cycle dependency, but it seems it's not related to this bug.

[ 3.460873] systemd[1]: remote-fs-pre.target: Found ordering cycle on remote-fs-pre.target/start
[ 3.460877] systemd[1]: remote-fs-pre.target: Found dependency on nfs-server.service/start
[ 3.460879] systemd[1]: remote-fs-pre.target: Found dependency on network.target/start
[ 3.460881] systemd[1]: remote-fs-pre.target: Found dependency on NetworkManager.service/start
[ 3.460883] systemd[1]: remote-fs-pre.target: Found dependency on dbus.service/start
[ 3.460885] systemd[1]: remote-fs-pre.target: Found dependency on basic.target/start
[ 3.460887] systemd[1]: remote-fs-pre.target: Found dependency on sockets.target/start
[ 3.460889] systemd[1]: remote-fs-pre.target: Found dependency on acpid.socket/start
[ 3.460891] systemd[1]: remote-fs-pre.target: Found dependency on sysinit.target/start
[ 3.460893] systemd[1]: remote-fs-pre.target: Found dependency on setkey.service/start
[ 3.460895] systemd[1]: remote-fs-pre.target: Found dependency on remote-fs.target/start
[ 3.460896] systemd[1]: remote-fs-pre.target: Found dependency on remote-fs-pre.target/start
[ 3.460899] systemd[1]: remote-fs-pre.target: Breaking ordering cycle by deleting job nfs-server.service/start
[ 3.460901] systemd[1]: nfs-server.service: Job nfs-server.service/start deleted to break ordering cycle starting with remote-fs-pre.target/start

Scott (scjones5) wrote :

I had been experiencing the same issue, as described in comment #5. I am using Ubuntu 16.04.2 4.4.0-83, nfs-kernel-server 1.2.8-9. The fix described in comments #6 and 9 along did not work but issuing

systemctl add-wants multi-user.target rpcbind.service
service nfs-kernel-server restart

as suggested in comment #13 solved the issue.

Kuba (jakub-nowacki) wrote :

Similar situation as reported above. This time however dmesg shows that there is a ordering cycle on nfs-server.service (depends on nfs-mountd.service)

kuba@kubadesktop:~$ systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
kuba@kubadesktop:~$ systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: inactive (dead)

Another output from dmesg related to nfs:
[ 3.366015] systemd[1]: nfs-server.service: Found ordering cycle on nfs-server.service/start
[ 3.366018] systemd[1]: nfs-server.service: Found dependency on nfs-mountd.service/start
[ 3.366021] systemd[1]: nfs-server.service: Found dependency on network.target/start
[ 3.366023] systemd[1]: nfs-server.service: Found dependency on NetworkManager.service/start
[ 3.366025] systemd[1]: nfs-server.service: Found dependency on basic.target/start
[ 3.366027] systemd[1]: nfs-server.service: Found dependency on sockets.target/start
[ 3.366029] systemd[1]: nfs-server.service: Found dependency on cups.socket/start
[ 3.366031] systemd[1]: nfs-server.service: Found dependency on sysinit.target/start
[ 3.366033] systemd[1]: nfs-server.service: Found dependency on setkey.service/start
[ 3.366035] systemd[1]: nfs-server.service: Found dependency on remote-fs.target/start
[ 3.366037] systemd[1]: nfs-server.service: Found dependency on remote-fs-pre.target/start
[ 3.366038] systemd[1]: nfs-server.service: Found dependency on nfs-server.service/start
[ 3.366040] systemd[1]: nfs-server.service: Breaking ordering cycle by deleting job nfs-mountd.service/start
[ 3.366043] systemd[1]: nfs-mountd.service: Job nfs-mountd.service/start deleted to break ordering cycle starting with nfs-server.service/start

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.