ypbind not able to socket activate rpcbind under systemd, fails at boot unless something else starts rpcbind

[John Sopko]

[Impact]

Due to a missing "WantedBy" directive in the .service file's [Install] section, rpcbind does not properly start when the system is booted, which can lead to problems if the user needs to mount some NFS locations on boot time.

[Test Case]

To reproduce the issue, one can do:

$ lxc launch images:ubuntu/xenial bug1558196-rpcbind --vm
$ lxc shell bug1558196-rpcbind
# apt update
# apt install rpcbind -y
# systemctl enable rpcbind.service
# reboot

After the reboot, you can confirm that the service is not running:

# systemctl status rpcbind.service
● rpcbind.service - RPC bind portmap service
   Loaded: loaded (/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)
  Drop-In: /run/systemd/generator/rpcbind.service.d
           └─50-rpcbind-$portmap.conf
   Active: inactive (dead)

[Regression Potential]

There is virtually no regression potential with this bug, aside from the regular "we will rebuild this package against newer versions of dependencies, which can introduce unwanted bugs". The fix is extremely simple and self-contained, and just guarantees that the package will start normally during boot time.

[Original Description]

did apt-get update/upgrade March 16, 2016
Description: Ubuntu Xenial Xerus (development branch)
Release: 16.04

rpcbind does not start on boot, tried various systemd debugging steps with no clues. After boot systemctl start rpcbind works. There is a /etc/init.d/rpcbind and a /lib/systemd/system/rpcbind.service config files which is confusing.

[Steve Langasek (vorlon)]

Do you have anything on your system that depends on rpcbind running? The current version of the rpcbind package is socket-activated; perhaps the service isn't started because you have nothing installed that actually uses it?

[John Sopko (sopko)]

We use nis ybind which is started out of /etc/init.d/nis which fails because rpcbind is not running.

[John Sopko (sopko)]

Meant to type "ypbind", so I installed nfs-kernel-server and now rpcbind and ypbind start. Hate to install nfs-kernel-server on all our desktops and servers that do not need it but they do need ypbind. Thanks for your help.

[Steve Langasek (vorlon)]

On Wed, Mar 16, 2016 at 07:16:36PM -0000, John Sopko wrote:
> We use nis ybind which is started out of /etc/init.d/nis which fails
> because rpcbind is not running.

Ok, but rpcbind should start as soon as nis asks for it.

What does 'systemctl status rpcbind.socket' show on boot (i.e. before you've
manually started rpcbind)?

Would ybind for some reason be trying to connect to rpcbind via localhost,
instead of using the UNIX socket (/run/rpcbind.sock)?

[John Sopko (sopko)]

rpcbind.socket is active, nis fails to start because it cannot contact the rpcbind server so I guess it is not using the socket to communicate. Reading the ypbind and ypbind.conf man pages does not have any info or options on registering with rpcbind.

root@tophat:~# systemctl status rpcbind.socket
* rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Thu 2016-03-17 07:49:58 EDT; 1min 1s ago
   Listen: /run/rpcbind.sock (Stream)

root@tophat:~# systemctl status rpcbind
* rpcbind.service - RPC bind portmap service
   Loaded: loaded (/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)
  Drop-In: /run/systemd/generator/rpcbind.service.d
           `-50-rpcbind-$portmap.conf
   Active: inactive (dead)

root@tophat:~# systemctl status nis
* nis.service - LSB: Start NIS client and server daemons.
   Loaded: loaded (/etc/init.d/nis; bad; vendor preset: enabled)
   Active: active (exited) since Thu 2016-03-17 07:50:33 EDT; 48s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1230 ExecStart=/etc/init.d/nis start (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 512)

[John Sopko (sopko)]

We are on the right track, I have tried so many things I need to reload the system to reset the default config for everything. Our 14.04 systems run rpc.statd but not all the other nfs stuff. So maybe enabling that service will be a work around, that is it will cause rpcbind to start. I will need some time to reload a default system. Thanks for your input.

[John Sopko (sopko)]

I re-installed the lasted 16.04 beta. Be default the rpc-statd service is not enabled, I enabled and rebooted, same problem rpcbind does not start and neither does the nis service. If I manualy start with systemctl start rpc-statd then rpcbind and rpcbind.statd start. If I systemctl start nis it does not start, I have to first stop withs ystemctl stop nis and then do a start. So still no luck getting nis to start without installing the nfs server.

[John Sopko (sopko)]

IMHO if the rpcbind service is enabled it should just start at boot time and not have to be self activated.

Found this but no real good solution.

"Regression: rpcbind doesn't start at boottime on systemd controlled machines."

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=805167

[John Sopko (sopko)]

Doing this forces rpcbind to start on boot and then nis starts correctly:

# /bin/systemctl add-wants multi-user.target rpcbind.service
Created symlink from /etc/systemd/system/multi-user.target.wants/rpcbind.service to /lib/systemd/system/rpcbind.service

[Michael (shotoflove)]

I'd like to confirm that on Ubuntu 16.04 the following allowed rpcbind to start on fresh bootup:
/bin/systemctl add-wants multi-user.target rpcbind.service

[Niall Brosnan (niallb)]

This has solved the issue for me also.
Ubuntu 16.04 with rpcbind 0.2.3-0.2 - apt updated on March 21st 15:15 GMT.

/bin/systemctl add-wants multi-user.target rpcbind.service

[Henk Bos (henk-bos-x)]

It does not work completely!
You can log in, however de UID/GID settings are not correct.
yptest and ypcat give the right results. But groups just returns the local group's and not the extra NIS-enabled groups.
The user is not able to access the server directory's if they need the extra NIS-enabled group settings.

[John Sopko (sopko)]

Does your /etc/nsswitch.conf have:

group: files nis

?

[Henk Bos (henk-bos-x)]

@John Sopko

yes, that is the trick!
Strange however that for all previous releases this parameter could be left at compat.
So I did not think of if to check this parameter, but thank you very much!

[Launchpad Janitor (janitor)]

This bug was fixed in the package rpcbind - 0.2.3-0.4

---------------
rpcbind (0.2.3-0.4) unstable; urgency=medium

  * Non-maintainer upload.
  * debian/rcpbind.service: Add WantedBy=multi-user.target, this should ensure
    that rpcbind daemon is started before nis (Closes: #805167, LP: #1558196)

 -- Laurent Bigonville <email address hidden> Tue, 31 May 2016 11:55:08 +0200

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jacques Beigbeder (jacques-beigbeder)]

I also use NIS, because it supports NFS-exports, mail-aliases, and netgroups in a lot of files.

Simply writing:
# /bin/systemctl add-wants multi-user.target rpcbind.service
is not enough. Sometimes boot is OK, sometimes not.

Starting manually rpcbind then ypbind is not OK. For instance, syslogd, mountd also use NIS.
So you have to restart a lot of services in case of NIS failure.

A statistic: the same installation rebooted 6 times was OK 4 times, fail 2 times...

[Sean Clarke (sean-clarke)]

Any idea when this package/fix is going to hit the Xenial repos? 23/07 and 16.04.1 and package still not available

[Jean-Pierre Flori (jpflori)]

Same issue here, my machines now get automatically updated to the new LTS and rpcbind/nis break.

[Erik Kruus (ejkruus)]

Jacques,

Ditto for rpcbind patch not being enough. I have traced the issue via syslogs
to nis starting too early, before kernel has brought network interfaces up.
**If** the network interfaces get up "fast", then boot succeeds. Otherwise
nfs fails and nameserver might never be there, and userids are unavailable, etc.
You may boot after the 5 min network timeout, but with drastically reduced
service.

What follows makes things work, but is UGLY and not the "right way".
--------------------------------------------------------------------
At our site IT recommends static ips set up with /etc/network/interfaces,
and the HACK I suggested involves overriding the default 'systemctl cat nis.service'
to put it after <email address hidden> and before NetworkManager.service.
This works, but is not a correct solution, because nis is starting after ifup
starts but before kernel reports the interface as 'ready'.

I also aid the boot by putting into 'interfaces' a
"post-up systemctl restart nis", just in case, and help some of the other
packages a bit by supplying dns-nameserver clause, and finally attempted
to cover my a** with an additional
  system-ctl add-wants NetworkManager.service nis.service

Now system boots with nis maybe 6 times in a row, and without timeouts, and
all nis stuff available -- good enough for me, but not generic since ifup
may not be there at your place.

I have NOT tested whether putting nis after (or maybe "Also") with
NetworkManager.service works, and if that worked, it might be a more general
solution.

-----For example only:
kruus@snake10$ cat nis.service
# /run/systemd/generator.late/nis.service
# Generated by [on a default install]
# systemctl cat nis.service > /etc/systemd/system/nis.service
# and some light editing.
#
# Also see the ifup dependency patched into
# /etc/systemd/system/nis.service.d/ifup.conf
# that reflects the static nec-labs route you set up in
# /etc/network/interfaces

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/nis
Description=LSB: Start NIS client and server daemons.
Before=multi-user.target
Before=graphical.target
Before=NetworkManager.service
Before=remote-fs.target
After=rpcbind.target
After=network-pre.target
# NO! After=remote-fs.target
# NO! Wants=network-online.target

[Install]
WantedBy=NetworkManager.service
# NO! WantedBy=nss-user-lookup.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/nis start
ExecStop=/etc/init.d/nis stop
ExecReload=/etc/init.d/nis reload

    /etc/systemd/system
kruus@snake10$ cat nis.service.d/ifup.conf
[Unit]
<email address hidden>
<email address hidden>
# ^^^^ your main network interface HERE.

    /etc/systemd/system
kruus@snake10$ cat /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto enp14s0
iface enp14s0 inet static
  address 138.15.169.213
  <snip-snip -- boring snake08--snake10 point-to-point on second port>

auto eno1
iface eno1 inet static
  address 138.15.169.213
  netmask 255.255.255.0
  gateway 138.15.169.254
# below is not required for general use:
  met...

[Erik Kruus (ejkruus)]

oops.. typo ... nis fails (not nfs fails)

[Jean-Pierre Flori (jpflori)]

Any plan to incorporate at least the first fix described here into Xenial?
This bug just prevents any machine using nis to migrate to the new LTS.

[Jean-Pierre Flori (jpflori)]

(I mean the fix introduce in 0.2.3-0.4 which is only available in yakkety.)

[cuihao (cuihao-leo)]

Bump.

Almost one year passed, still broken in Xenial... Why not backport Yakkety or Debian's fix to Xenial?

[Jean-Pierre Flori (jpflori)]

I'm not sure anyone in charge cares...
The solution I felt back to is changing my authentication system...

[bamyasi (iadzhubey)]

Still broken on Ubuntu 16.04.2 LTS. Completely ruined crucial NFS server functionality. How it is even possible for such a critical bug to stay unfixed for years??

[Ted Cabeen (ted-cabeen)]

Debian will accept a Non-Maintainer Upload. Since the backport is so trivial (The patch from yakkety works perfectly), would that be possible here?

[Paul Menzel (paulmenzel)]

The referenced fix below [2] mentioned by Michael in [1] fixes the issue for my 16.04.

```
$ cat /etc/systemd/system/rpcbind.socket.d/override.conf
[Unit]
DefaultDependencies=no
Wants=rpcbind.target
Before=rpcbind.target
```

[1] https://bugs.debian.org/805167#88
[2] https://bugs.debian.org/812371#31

[Christian Ehrhardt (paelzer)]

The fix for 805167 is in Yakkety.
We need to consider that as an SRU for Xenial.

[Christian Ehrhardt (paelzer)]

As nis needs no change set that to invalid.

[Tjd (tjd)]

@paelzer: Is this being considered for SRU? Could it still make it into 16.04.4?

[Eric G (erickg)]

$ cat /etc/systemd/system/rpcbind.socket.d/override.conf
[Unit]
DefaultDependencies=no
Wants=rpcbind.target
Before=rpcbind.target

Fixed xenial for me. Would be nice to have a fixed package for LTS.

[David Hansen (dchansen)]

date
Thu Jul 12 12:57:35 EDT 2018

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial

apt-cache policy rpcbind
rpcbind:
  Installed: 0.2.3-0.2
  Candidate: 0.2.3-0.2

This bug was opened on 2016-03-16 against Xenial.
It was posted that the rpcbind package was fixed on 2016-06-01.
It is now 2018-07-12 and the package is still not in the repositories.
Is rpcbind (0.2.3-0.4) really not going to be introduced into the repositories for the LTS it affects? Is this not the type of change an LTS branch is expected to receive?

[Robie Basak (racb)]

If this is still a valid and understood issue for Xenial then its status should be Triaged, not Incomplete. Otherwise it looks like it doesn't have enough information to be worked on. Please see https://wiki.ubuntu.com/Bugs/Status

[David Hansen (dchansen)]

This is still a valid issue, I just performed a do-release-upgrade on a v14.04 system yesterday and the same rpcbind issue reared it's ugly little head.

Reading the posts here should conclude that it is an understood issue for Xenial.

My post here was two years ago. This ticket was created four years ago. If rpcbind package v0.2.3-0.4 (Released four years ago!) really does fix this issue then what in the world is holding up it being included in Xenial? 16.04.06 and the package still has not been included yet.

What further information is needed to work on this?

[Robie Basak (racb)]

Hello John, or anyone else affected,

Accepted rpcbind into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rpcbind/0.2.3-0.2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted rpcbind (0.2.3-0.2ubuntu0.1) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

nfs-utils/1:1.2.8-9ubuntu12.3 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#rpcbind

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

ubuntu@lp1558196-x:~$ dpkg -l|grep rpcbind
ii rpcbind 0.2.3-0.2 amd64 converts RPC program numbers into universal addresses
ubuntu@lp1558196-x:~$ sudo reboot
...(reboot)...
ubuntu@lp1558196-x:~$ sudo systemctl status rpcbind.service
● rpcbind.service - RPC bind portmap service
   Loaded: loaded (/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)
  Drop-In: /run/systemd/generator/rpcbind.service.d
           └─50-rpcbind-$portmap.conf
   Active: inactive (dead)

ii rpcbind 0.2.3-0.2ubuntu0.1 amd64 converts RPC program numbers into universal addresses
ubuntu@lp1558196-x:~$ sudo reboot
...(reboot)...
ubuntu@lp1558196-x:~$ sudo systemctl status rpcbind.service
● rpcbind.service - RPC bind portmap service
   Loaded: loaded (/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/rpcbind.service.d
           └─50-rpcbind-$portmap.conf
   Active: active (running) since Mon 2020-08-10 15:19:02 UTC; 32s ago
 Main PID: 431 (rpcbind)
    Tasks: 1
   Memory: 836.0K
      CPU: 7ms
   CGroup: /system.slice/rpcbind.service
           └─431 /sbin/rpcbind -f -w

Aug 10 15:19:02 lp1558196-x systemd[1]: Starting RPC bind portmap service...
Aug 10 15:19:02 lp1558196-x rpcbind[431]: rpcbind: xdr_/run/rpcbind/rpcbind.xdr: failed
Aug 10 15:19:02 lp1558196-x rpcbind[431]: rpcbind: xdr_/run/rpcbind/portmap.xdr: failed
Aug 10 15:19:02 lp1558196-x systemd[1]: Started RPC bind portmap service.

extra verification note:
the 'xdr_/run/rpcbind/*.xdr: failed' messages logged on first start come from the read_warmstart() function, and since the 'warmstart' data is stored under /run it doesn't exist on bootup. The read_warmstart() function logs the 'error' that it can't find the file, but there is no impact to the functionality of rpcbind, since there is no 'warmstart' data to read in, the first rpcbind startup simply acts as a 'coldstart' which is the correct behavior.

[Dan Streetman (ddstreet)]

autopkgtest note: nfs-utils(s390x) fails, but appears to have always failed; the test runs that 'passed' back in 2017 just skipped the now-failing testcase due to lack of machine isolation back then:
"local-server-client SKIP Test requires machine-level isolation but testbed does not provide that"

so that autopkgtest failure can be ignored as unrelated to this.

[Sergio Durigan Junior (sergiodj)]

On Monday, August 10 2020, Dan Streetman wrote:

> autopkgtest note: nfs-utils(s390x) fails, but appears to have always
> failed; the test runs that 'passed' back in 2017 just skipped the
> now-failing testcase due to lack of machine isolation back then:
> "local-server-client SKIP Test requires machine-level isolation but testbed does not provide that"
>
> so that autopkgtest failure can be ignored as unrelated to this.

Thanks for the verification, Dan.

I did some investigation regarding this nfs-utils failure on s390x, and
posted my conclusions here:

  https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1705483/comments/1

As a follow-up, I filed the following MP for hints-ubuntu-xenial:

  https://code.launchpad.net/~sergiodj/britney/hints-ubuntu-xenial/+merge/388495

Which should force-badtest the failure. I'm still waiting on someone to
review and approve the MP.

Thanks,

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

[Launchpad Janitor (janitor)]

This bug was fixed in the package rpcbind - 0.2.3-0.2ubuntu0.1

---------------
rpcbind (0.2.3-0.2ubuntu0.1) xenial; urgency=medium

  * d/rpcbind.service: Add WantedBy=multi-user.target, this should ensure
    that rpcbind daemon is started before nis (Closes: #805167, LP: #1558196)
    Thanks to Laurent Bigonville <email address hidden>.

 -- Sergio Durigan Junior <email address hidden> Fri, 10 Jul 2020 14:26:18 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for rpcbind has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.