gpsd should start after chrony if they are being used together

[Mark Shuttleworth]

When using GPSD to provide a time signal over a socket to chrony, it is necessary to start gpsd after chrony (I think this is so that gpsd finds the /var/run/chrony.ttySX.sock file when it launches). gpsd tends to want to drop privileges quickly which makes it hard for it to find this file later, aiui.

[Mark Shuttleworth (sabdfl)]

I think the existing gpsd.service has a bug in that it states After=chronyd.service when the correct service is chrony.service

[Christian Ehrhardt  (paelzer)]

Hi Mark,
today I'm not answering on my own, instead the Debian maintainer Vincent Blut already contacted me on this bug (as he has issues with Launchpad to answer on his own), so I'm forwarding his message:

"The fact that the gpsd.service mentions “After=chronyd.service” should not be an issue because when I created the chrony.service unit file for Debian, I set chronyd.service as an alias (i.e., Alias=chronyd.service), so the gpsd unit is correct on this front."

---

From here on I'm is myself again :-)
As reference, from /lib/systemd/system/gpsd.service
  # Needed with chrony SOCK refclock
  After=chronyd.service
And from /lib/systemd/system/chrony.service
  Alias=chronyd.service

So systemd-wise gpsd should start after chrony(d) as intended.

I assume you were running into issues with this setup that made you checking the service dependencies? So what happened exactly, gpsd didn't find the socket I assume?

If that is the case I have an assumption what the reason could be thou.
The definition of a type=forking unit is that it waits until the master PID it spawned exits signalizing that the init is complete.
Now for the MAAS and general issue that we want the server portion of chrony to run nice and clean without the user wrestling with the config we use a wrapper script that might (or not) add options.
I could think of this wrapper returning before chrony itself is fully set up, by that there could be a race where chrony is still initializing and the socket is not yet around while gpsd is already starting up and fails to find it.

To test this - if the issue is reproducible, one could exchange in the chrony service file
  ExecStart=/usr/lib/systemd/scripts/chronyd-starter.sh $DAEMON_OPTS
with
  ExecStart=/usr/sbin/chronyd $DAEMON_OPTS
If that makes it reliably working then we have to augment our script to properly wait.

It is a tight race thou, as the forking even on my overloaded laptop returns in 0m0.003s - 0m0.008s real time - anyway CPUs are fast and it is the best theory we have until you report more details on the issue you are seeing.

Hmm, actually I checked in detail how our wrapper script is working and it ends with the call to chronyd (to get its RC). But that implies it will only end the shell script when the program is done. So the theoretical race I have thought of does not exists.

Never the less there might not be an issue with our wrapper, but instead with chrony itself just returning its start PID too early before having set up the socket.
We could check that in your setup via appending
  rc=$?
  sleep 5s
  exit $rc
to the tail of /usr/lib/systemd/scripts/chronyd-starter.sh
That would grant it (for debugging) some more init time.

Note: in gpsd code I found that the socket path it will open depends on the device.
So if you use gpsd on ttyS0 then /var/run/chrony.ttyS0.sock is ok, but if you have anything else like ttyUSB0 then the path will differ.
From:
        (void)snprintf(chrony_path, sizeof (chrony_path),"/var/run/chrony.%s.sock", basename(session->gpsdata.dev.path));
Below that is a bunch of log messages:
       "PPS:%s chrony socket %s doesn't exist\n",
       "PPS:%s connect chrony socket failed: %s, error: %d...

[Mark Shuttleworth (sabdfl)]

I have an odd behaviour in that gpsd starts on one machine but not on
the other. I think this is just me being new to systemd, please let's
close this bug as invalid until I have a deeper root cause. Thanks for
the help!

 status invalid

[Christian Ehrhardt  (paelzer)]

Thanks Mark, I have seen your discussions on #systemd about it - perfect understatement of your skills :-).
Don't be afraid, I'm not tracking you, but among other IRC highlights, one on "chrony" helps me to ensure our users are good with it :-)

I find it puzzling as well why it should be "different" on both systems.
I have seen you ended with wondering about what the packaging does about it, so I wanted to dump that here FYI.

GPSD calls systemd helpers for the service which will end up in *inst maintainer scripts.
in d/rules are:
  dh_systemd_enable -pgpsd
  dh_installinit
  dh_systemd_start -pgpsd --restart-after-upgrade
That sequence should make it available and started after install in all cases.

The only interesting bit, it seems to be actually socket activated.
Therefore the "indirect" on gpsd, but you can check the socket via
  $ systemctl status gpsd.socket

I hope that might shed some light onto your issue.

[Mark Shuttleworth (sabdfl)]

On 05/17/2018 06:52 AM,  Christian Ehrhardt  wrote:
> GPSD calls systemd helpers for the service which will end up in *inst maintainer scripts.
> in d/rules are:
> dh_systemd_enable -pgpsd
> dh_installinit
> dh_systemd_start -pgpsd --restart-after-upgrade
> That sequence should make it available and started after install in all cases.

On various systems that I have tried, simply installing gpsd does not
cause it to run on boot, you need to connect to the socket before gpsd
is actually started. That makes it unhelpful for chrony because it needs
to start automatically on boot. The unit files correctly handle the
sequencing (gpsd after chrony) but I needed to add a symlink for
multi-user.target in order to get gpsd running on boot without having
something to connect to the socket. chrony doesn't connect to the
socket, chrony makes it's own different socket that gpsd detects and
writes to, independently of the gpsd socket.

The odd thing is that one of my systems already had the multi-user
target, which is why THAT system had gpsd running on boot. I have
nothing in my command history there showing me creating it. This machine
is an older machine that has been upgraded since at least Xenial,
possibly even pre-Xenial, so it may be that an older gpsd package did
something.

The only surprise is needing to manually create the multi-user.target
dependency for use with chrony. Ideally, the gpsd packaging would detect
cases where chrony is offering it's special socket, and gpsd would be
activated accordingly, but I can't think of an obvious way to depend on
the existence of a file matching a particular filename schema
('/var/run/chrony.ttyXXX.sock'). If systemd does support that sort of
funky file-exists dependency, then we could try adding that.

Mark