ypbind not able to socket activate rpcbind under systemd, fails at boot unless something else starts rpcbind

Bug #1558196 reported by John Sopko on 2016-03-16
136
This bug affects 28 people
Affects Status Importance Assigned to Milestone
nis (Debian)
Fix Released
Unknown
nis (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
rpcbind (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned

Bug Description

did apt-get update/upgrade March 16, 2016
Description: Ubuntu Xenial Xerus (development branch)
Release: 16.04

rpcbind does not start on boot, tried various systemd debugging steps with no clues. After boot systemctl start rpcbind works. There is a /etc/init.d/rpcbind and a /lib/systemd/system/rpcbind.service config files which is confusing.

Steve Langasek (vorlon) wrote :

Do you have anything on your system that depends on rpcbind running? The current version of the rpcbind package is socket-activated; perhaps the service isn't started because you have nothing installed that actually uses it?

John Sopko (sopko) wrote :

We use nis ybind which is started out of /etc/init.d/nis which fails because rpcbind is not running.

John Sopko (sopko) wrote :

Meant to type "ypbind", so I installed nfs-kernel-server and now rpcbind and ypbind start. Hate to install nfs-kernel-server on all our desktops and servers that do not need it but they do need ypbind. Thanks for your help.

On Wed, Mar 16, 2016 at 07:16:36PM -0000, John Sopko wrote:
> We use nis ybind which is started out of /etc/init.d/nis which fails
> because rpcbind is not running.

Ok, but rpcbind should start as soon as nis asks for it.

What does 'systemctl status rpcbind.socket' show on boot (i.e. before you've
manually started rpcbind)?

Would ybind for some reason be trying to connect to rpcbind via localhost,
instead of using the UNIX socket (/run/rpcbind.sock)?

rpcbind.socket is active, nis fails to start because it cannot contact the rpcbind server so I guess it is not using the socket to communicate. Reading the ypbind and ypbind.conf man pages does not have any info or options on registering with rpcbind.

root@tophat:~# systemctl status rpcbind.socket
* rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Thu 2016-03-17 07:49:58 EDT; 1min 1s ago
   Listen: /run/rpcbind.sock (Stream)

root@tophat:~# systemctl status rpcbind
* rpcbind.service - RPC bind portmap service
   Loaded: loaded (/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)
  Drop-In: /run/systemd/generator/rpcbind.service.d
           `-50-rpcbind-$portmap.conf
   Active: inactive (dead)

root@tophat:~# systemctl status nis
* nis.service - LSB: Start NIS client and server daemons.
   Loaded: loaded (/etc/init.d/nis; bad; vendor preset: enabled)
   Active: active (exited) since Thu 2016-03-17 07:50:33 EDT; 48s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1230 ExecStart=/etc/init.d/nis start (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 512)

John Sopko (sopko) wrote :

We are on the right track, I have tried so many things I need to reload the system to reset the default config for everything. Our 14.04 systems run rpc.statd but not all the other nfs stuff. So maybe enabling that service will be a work around, that is it will cause rpcbind to start. I will need some time to reload a default system. Thanks for your input.

John Sopko (sopko) wrote :

I re-installed the lasted 16.04 beta. Be default the rpc-statd service is not enabled, I enabled and rebooted, same problem rpcbind does not start and neither does the nis service. If I manualy start with systemctl start rpc-statd then rpcbind and rpcbind.statd start. If I systemctl start nis it does not start, I have to first stop withs ystemctl stop nis and then do a start. So still no luck getting nis to start without installing the nfs server.

Steve Langasek (vorlon) on 2016-03-17
summary: - rpcbind does not start on boot under systemd
+ ypbind not able to socket activate rpcbind under systemd, fails at boot
+ unless something else starts rpcbind
Steve Langasek (vorlon) on 2016-03-17
Changed in nis (Ubuntu):
status: New → Triaged
Changed in rpcbind (Ubuntu):
status: New → Triaged
John Sopko (sopko) wrote :

IMHO if the rpcbind service is enabled it should just start at boot time and not have to be self activated.

Found this but no real good solution.

"Regression: rpcbind doesn't start at boottime on systemd controlled machines."

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=805167

John Sopko (sopko) wrote :

Doing this forces rpcbind to start on boot and then nis starts correctly:

# /bin/systemctl add-wants multi-user.target rpcbind.service
Created symlink from /etc/systemd/system/multi-user.target.wants/rpcbind.service to /lib/systemd/system/rpcbind.service

Michael (shotoflove) wrote :

I'd like to confirm that on Ubuntu 16.04 the following allowed rpcbind to start on fresh bootup:
/bin/systemctl add-wants multi-user.target rpcbind.service

Niall Brosnan (niallb) wrote :

This has solved the issue for me also.
Ubuntu 16.04 with rpcbind 0.2.3-0.2 - apt updated on March 21st 15:15 GMT.

/bin/systemctl add-wants multi-user.target rpcbind.service

Henk Bos (henk-bos-x) wrote :

It does not work completely!
You can log in, however de UID/GID settings are not correct.
yptest and ypcat give the right results. But groups just returns the local group's and not the extra NIS-enabled groups.
The user is not able to access the server directory's if they need the extra NIS-enabled group settings.

John Sopko (sopko) wrote :

Does your /etc/nsswitch.conf have:

group: files nis

?

Henk Bos (henk-bos-x) wrote :

@John Sopko

yes, that is the trick!
Strange however that for all previous releases this parameter could be left at compat.
So I did not think of if to check this parameter, but thank you very much!

Changed in nis (Debian):
status: Unknown → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rpcbind - 0.2.3-0.4

---------------
rpcbind (0.2.3-0.4) unstable; urgency=medium

  * Non-maintainer upload.
  * debian/rcpbind.service: Add WantedBy=multi-user.target, this should ensure
    that rpcbind daemon is started before nis (Closes: #805167, LP: #1558196)

 -- Laurent Bigonville <email address hidden> Tue, 31 May 2016 11:55:08 +0200

Changed in rpcbind (Ubuntu):
status: Triaged → Fix Released
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in nis (Ubuntu Xenial):
status: New → Confirmed
Changed in rpcbind (Ubuntu Xenial):
status: New → Confirmed

I also use NIS, because it supports NFS-exports, mail-aliases, and netgroups in a lot of files.

Simply writing:
# /bin/systemctl add-wants multi-user.target rpcbind.service
is not enough. Sometimes boot is OK, sometimes not.

Starting manually rpcbind then ypbind is not OK. For instance, syslogd, mountd also use NIS.
So you have to restart a lot of services in case of NIS failure.

A statistic: the same installation rebooted 6 times was OK 4 times, fail 2 times...

Sean Clarke (sean-clarke) wrote :

Any idea when this package/fix is going to hit the Xenial repos? 23/07 and 16.04.1 and package still not available

Jean-Pierre Flori (jpflori) wrote :

Same issue here, my machines now get automatically updated to the new LTS and rpcbind/nis break.

Erik Kruus (ejkruus) wrote :
Download full text (4.0 KiB)

Jacques,

Ditto for rpcbind patch not being enough. I have traced the issue via syslogs
to nis starting too early, before kernel has brought network interfaces up.
**If** the network interfaces get up "fast", then boot succeeds. Otherwise
nfs fails and nameserver might never be there, and userids are unavailable, etc.
You may boot after the 5 min network timeout, but with drastically reduced
service.

What follows makes things work, but is UGLY and not the "right way".
--------------------------------------------------------------------
At our site IT recommends static ips set up with /etc/network/interfaces,
and the HACK I suggested involves overriding the default 'systemctl cat nis.service'
to put it after <email address hidden> and before NetworkManager.service.
This works, but is not a correct solution, because nis is starting after ifup
starts but before kernel reports the interface as 'ready'.

I also aid the boot by putting into 'interfaces' a
"post-up systemctl restart nis", just in case, and help some of the other
packages a bit by supplying dns-nameserver clause, and finally attempted
to cover my a** with an additional
  system-ctl add-wants NetworkManager.service nis.service

Now system boots with nis maybe 6 times in a row, and without timeouts, and
all nis stuff available -- good enough for me, but not generic since ifup
may not be there at your place.

I have NOT tested whether putting nis after (or maybe "Also") with
NetworkManager.service works, and if that worked, it might be a more general
solution.

-----For example only:
kruus@snake10$ cat nis.service
# /run/systemd/generator.late/nis.service
# Generated by [on a default install]
# systemctl cat nis.service > /etc/systemd/system/nis.service
# and some light editing.
#
# Also see the ifup dependency patched into
# /etc/systemd/system/nis.service.d/ifup.conf
# that reflects the static nec-labs route you set up in
# /etc/network/interfaces

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/nis
Description=LSB: Start NIS client and server daemons.
Before=multi-user.target
Before=graphical.target
Before=NetworkManager.service
Before=remote-fs.target
After=rpcbind.target
After=network-pre.target
# NO! After=remote-fs.target
# NO! Wants=network-online.target

[Install]
WantedBy=NetworkManager.service
# NO! WantedBy=nss-user-lookup.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/nis start
ExecStop=/etc/init.d/nis stop
ExecReload=/etc/init.d/nis reload

    /etc/systemd/system
kruus@snake10$ cat nis.service.d/ifup.conf
[Unit]
<email address hidden>
<email address hidden>
# ^^^^ your main network interface HERE.

    /etc/systemd/system
kruus@snake10$ cat /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto enp14s0
iface enp14s0 inet static
  address 138.15.169.213
  <snip-snip -- boring snake08--snake10 point-to-point on second port>

auto eno1
iface eno1 inet static
  address 138.15.169.213
  netmask 255.255.255.0
  gateway 138.15.169.254
# below is not required for general use:
  met...

Read more...

Erik Kruus (ejkruus) wrote :

oops.. typo ... nis fails (not nfs fails)

Jean-Pierre Flori (jpflori) wrote :

Any plan to incorporate at least the first fix described here into Xenial?
This bug just prevents any machine using nis to migrate to the new LTS.

Jean-Pierre Flori (jpflori) wrote :

(I mean the fix introduce in 0.2.3-0.4 which is only available in yakkety.)

cuihao (cuihao-leo) wrote :

Bump.

Almost one year passed, still broken in Xenial... Why not backport Yakkety or Debian's fix to Xenial?

Jean-Pierre Flori (jpflori) wrote :

I'm not sure anyone in charge cares...
The solution I felt back to is changing my authentication system...

bamyasi (iadzhubey) wrote :

Still broken on Ubuntu 16.04.2 LTS. Completely ruined crucial NFS server functionality. How it is even possible for such a critical bug to stay unfixed for years??

Ted Cabeen (ted-cabeen) wrote :

Debian will accept a Non-Maintainer Upload. Since the backport is so trivial (The patch from yakkety works perfectly), would that be possible here?

Paul Menzel (paulmenzel) wrote :

The referenced fix below [2] mentioned by Michael in [1] fixes the issue for my 16.04.

```
$ cat /etc/systemd/system/rpcbind.socket.d/override.conf
[Unit]
DefaultDependencies=no
Wants=rpcbind.target
Before=rpcbind.target
```

[1] https://bugs.debian.org/805167#88
[2] https://bugs.debian.org/812371#31

ChristianEhrhardt (paelzer) wrote :

The fix for 805167 is in Yakkety.
We need to consider that as an SRU for Xenial.

Changed in nis (Ubuntu):
status: Triaged → Fix Released
Changed in rpcbind (Ubuntu Xenial):
status: Confirmed → Triaged
tags: added: server-next
ChristianEhrhardt (paelzer) wrote :

As nis needs no change set that to invalid.

Changed in nis (Ubuntu Xenial):
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.