sockets.target:
A special target unit that sets up all socket units (see systemd.socket(5) for details) that shall be active after boot. Services that can be socket-activated shall add Wants= dependencies to this unit for their socket unit during installation. This is best configured via a WantedBy=sockets.target in the socket unit's "[Install]" section.
Lets see who is starting rpcbind.socket:
[Install]
WantedBy=sockets.target
Unit file will be linked as "wanted" by "sockets.target".
rpcbind.service depends on rpcbind.target (not with my patch) AND rpcbind.socket (started by basic.target with the sockets.target). Lets see rpcbind.target (its a special target made by a generator, its unit file doesn't contain anything):
rpcbind.target:
The portmapper/rpcbind pulls in this target and orders itself before it, to indicate its availability. systemd automatically adds dependencies of type After= for this target unit to all SysV init script service units with an LSB header referring to the "$portmap" facility.
So, systemd puts an After=rpcbind.target in all SysV scripts with LSB header containing "$portmap" as facility (insserv style):
$ systemctl status rpcbind.target
● rpcbind.target - RPC Port Mapper
Loaded: loaded (/etc/insserv.conf.d/rpcbind; static; vendor preset: enabled)
Drop-In: /run/systemd/generator/rpcbind.target.d └─50-hard-dependency-rpcbind-$portmap.conf
So rpcbind.target doesn't start "rpcbind.service" and any "rpcbind.target" usage seems broken to me. rpcbind was ever started by nfs-{kernel-server,mountd}.service (since they were relying on the rpcbind.target). rpcbind (the portmap) was being activated by a connection to the rpc socket.
But NOT the NETWORK socket, the LOCAL socket:
[Socket]
ListenStream=/run/rpcbind.sock
You can test this by stopping the "rpcbind.service" and trying:
So accessing "/run/rpcbind.sock" is what "started" the rpcbind and the race condition seen was born there, for sure. We have to make sure rpcbind is loaded BEFORE any service depending on it. Lets analyze next if my change (based only on backporting upstream code/fix) accomplishes that.
Summarizing
From systemd.special we have:
sockets.target: sockets. target in the socket unit's "[Install]" section.
A special target unit that sets up all socket units (see systemd.socket(5) for details) that shall be active after boot. Services that can be socket-activated shall add Wants= dependencies to this unit for their socket unit during installation. This is best configured via a WantedBy=
Lets see who is starting rpcbind.socket:
[Install] sockets. target
WantedBy=
Unit file will be linked as "wanted" by "sockets.target".
$ systemctl list-dependencies sockets.target forward. socket
sockets.target
● ├─acpid.socket
● ├─apport-
● ├─dbus.socket
● ├─dm-event.socket
● ├─rpcbind.socket
... <many others>
$ systemctl --reverse list-dependencies sockets.target
sockets.target
● └─basic.target
● └─multi-user.target
● └─graphical.target
basic.target starts the sockets.target.
sockets.target start rpcbind.socket.
rpcbind.target doesn't start rpcbind.service.
Lets who starts rpcbind.service:
[Install]
Also=rpcbind.socket
rpcbind. {socket, service} are enabled/disabled together (not started).
$ systemctl list-dependencies rpcbind.service fs-pre. target
rpcbind.service
● ├─rpcbind.socket
● ├─system.slice
● ├─remote-
● └─rpcbind.target
└─...
rpcbind.service depends on rpcbind.target (not with my patch) AND rpcbind.socket (started by basic.target with the sockets.target). Lets see rpcbind.target (its a special target made by a generator, its unit file doesn't contain anything):
rpcbind.target:
The portmapper/rpcbind pulls in this target and orders itself before it, to indicate its availability. systemd automatically adds dependencies of type After= for this target unit to all SysV init script service units with an LSB header referring to the "$portmap" facility.
So, systemd puts an After=rpcbind. target in all SysV scripts with LSB header containing "$portmap" as facility (insserv style):
$ systemctl status rpcbind.target conf.d/ rpcbind; static; vendor preset: enabled) generator/ rpcbind. target. d
└─50- hard-dependency -rpcbind- $portmap. conf
● rpcbind.target - RPC Port Mapper
Loaded: loaded (/etc/insserv.
Drop-In: /run/systemd/
/etc/insserv. conf.d/ rpcbind contains "rpcbind". generator/ rpcbind. service. d/50-rpcbind- \$portmap. conf contains:
/run/systemd/
[Unit] target rpcbind. target
Wants=rpcbind.
Before=
So rpcbind.target doesn't start "rpcbind.service" and any "rpcbind.target" usage seems broken to me. rpcbind was ever started by nfs-{kernel- server, mountd} .service (since they were relying on the rpcbind.target). rpcbind (the portmap) was being activated by a connection to the rpc socket.
But NOT the NETWORK socket, the LOCAL socket:
[Socket] /run/rpcbind. sock
ListenStream=
You can test this by stopping the "rpcbind.service" and trying:
$ rpcinfo -T tcp 127.0.0.1
rpcinfo: can't contact rpcbind: RPC: Remote system error - Connection refused
$ NETPATH= /run/rpcbind. sock rpcinfo
program version netid address service owner
100000 4 tcp6 ::.0.111 portmapper superuser
100000 3 tcp6 ::.0.111 portmapper superuser
100000 4 udp6 ::.0.111 portmapper superuser
100000 3 udp6 ::.0.111 portmapper superuser
100000 4 tcp 0.0.0.0.0.111 portmapper superuser
So accessing "/run/rpcbind.sock" is what "started" the rpcbind and the race condition seen was born there, for sure. We have to make sure rpcbind is loaded BEFORE any service depending on it. Lets analyze next if my change (based only on backporting upstream code/fix) accomplishes that.