GPSD unable to use Kernel PPS

[Mark Shuttleworth]

[Impact]

 * Current GPSD apparmor isolation is too strict to use PPS devices
   properly.

 * backport changes we added to 20.10 to fix this

[Test Case]

 * Set up a PPS device with chrony/gpsd as described in [1]
   Check the log output.

   Bad case:
   gpsd:INFO: KPPS:/dev/ttyS0 device not found.
   gpsd:WARN: KPPS:/dev/ttyS0 kernel PPS unavailable, PPS accuracy will suffer

   Good case does not show the errors above. Check that gpsd properly
   initializes the device by ensuring this works for the whole stack
   and chrony ends up getting proper PPS time data (also in [1]).

[1]: https://ubuntu.com/server/docs/network-ntp

[Regression Potential]

 * As always with apparmor changes the regression risk comes in two way:
   - we allow more than before, that could be insecure but we have the +1
     from the security team and optimized to further reduce permissions.
   - we deny some access (to silence warnings) which could, if strictly
     required for un-tested use cases break these use-cases. Neither in the
     tests nor in the review/discussion such cases were identified.

[Other Info]

 * This is accepted in Debians packaging git, if not in Groovy in time I'll
   need to put an 3.20-8ubuntu1 there, but I can preparing the SRU
   independent to that.

----

I am working with gpsd and a pulse-per-second (PPS) signal on a serial port.

GPSD is able to detect the PPS signal but it is not able to use Kernel PPS for lower jitter. Here are the relevant log lines when PPS is being detected and enabled:

gpsd:INFO: KPPS:/dev/ttyS0 device not found.
gpsd:WARN: KPPS:/dev/ttyS0 kernel PPS unavailable, PPS accuracy will suffer
gpsd:PROG: PPS:/dev/ttyS0 thread launched
gpsd:INFO: PPS:/dev/ttyS0 ntpshm_link_activate: 1
gpsd:PROG: TPPS:/dev/ttyS0 ioctl(TIOCMIWAIT) succeeded, time: 1586594708.000179259, state: 0
gpsd:PROG: TPPS:/dev/ttyS0 Clear, cycle: 1586594708000179, duration: 1586594708000179 @ 1586594708.000179259
gpsd:PROG: PPS:/dev/ttyS0 Clear cycle: 1586594708000179, duration: 1586594708000179 @ 1586594708.000179259
gpsd:PROG: TPPS:/dev/ttyS0 ioctl(TIOCMIWAIT) succeeded, time: 1586594708.100154827, state: 64
gpsd:PROG: TPPS:/dev/ttyS0 Assert, cycle: 1586594708100154, duration: 99975 @ 1586594708.100154827
gpsd:PROG: PPS:/dev/ttyS0 Assert cycle: 1586594708100154, duration: 99975 @ 1586594708.100154827
gpsd:PROG: PPS:/dev/ttyS0 Assert ignored missing last_fixtime
gpsd:PROG: TPPS:/dev/ttyS0 ioctl(TIOCMIWAIT) succeeded, time: 1586594709.000117775, state: 0
gpsd:PROG: TPPS:/dev/ttyS0 Clear, cycle: 999938, duration: 899962 @ 1586594709.000117775
gpsd:PROG: PPS:/dev/ttyS0 Clear cycle: 999938, duration: 899962 @ 1586594709.000117775
gpsd:INFO: PPS:/dev/ttyS0 Clear hooks called clock: 1586594709.000117775 real: 1586594709.000000000: no fix
gpsd:PROG: PPS:/dev/ttyS0 Clear no fix @ 1586594709.000117775 offset -0.000117775
gpsd:PROG: TPPS:/dev/ttyS0 ioctl(TIOCMIWAIT) succeeded, time: 1586594709.100167645, state: 64
gpsd:PROG: TPPS:/dev/ttyS0 Assert, cycle: 1000012, duration: 100049 @ 1586594709.100167645
gpsd:PROG: PPS:/dev/ttyS0 Assert cycle: 1000012, duration: 100049 @ 1586594709.100167645
gpsd:PROG: PPS:/dev/ttyS0 Assert ignored this second already handled

The kernel does have PPS configured and the pps_ldisc module loaded. I think GPSd actually drove the loading of the module but I am not sure. I see the module being loaded and the serial port becoming a pps0 source during the boot sequence:

10:34:09 doorway kernel: [ 8.691161] pps_ldisc: PPS line discipline registered
10:34:09 doorway kernel: [ 8.691686] pps pps0: new PPS source serial0
10:34:09 doorway kernel: [ 8.691714] pps pps0: source "/dev/ttyS0" added

This is confirmed with ppsfind:

$ ppsfind ttyS0
pps0: name=serial0 path=/dev/ttyS0

And this pps0 device is working well:

$ sudo ppswatch /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
timestamp: 1586596125, sequence: 1418, offset: 85067
timestamp: 1586596125, sequence: 1418, offset: 85067
timestamp: 1586596126, sequence: 1419, offset: 81949
timestamp: 1586596126, sequence: 1419, offset: 81949
timestamp: 1586596127, sequence: 1420, offset: 80119
timestamp: 1586596127, sequence: 1420, offset: 80119
timestamp: 1586596128, sequence: 1421, offset: 81099
timestamp: 1586596128, sequence: 1421, offset: 81099

So, the kernel PPS subsystem is there and appears to be perfectly happy with the pps0 coming from /dev/ttyS0 but still GPSD appears unable to use all of that.

[Christian Ehrhardt  (paelzer)]

Hi Mark,
Disclaimer: my gps device has no PPS so I'm stabbing a bit in the dark here. I beg your pardon for that in advance.

By default devices are hot-added as they are detected e.g. via udev.
But for PPS there is a permission issue if you have it set up like it.

From [1]: "In order to present the smallest possible attack surface to privilege-escalation attempts, gpsd, if run as root, drops its root privileges very soon after startup - just after it has opened any serial device paths passed on the command line.
Thus, KPPS can only be used with devices passed that way, not with GPSes that are later presented to gpsd by the hotplug system. Those hotplug devices may, however, be able to use plain, non-kernel PPS. gpsd tries to automatically fall back to this when absence of root permissions makes KPPS unavailable."

I assume it is ok for your setup, but just to be sure you have set $DEVICES to include, maybe even both of them?
The hot adding is rather new and has other known bugs (e.g. 1873415) anyway and since your gpsd debug output lists the device you should be ok on that, but I wanted to ask to be sure.

How does "systemctl status gpsd.service" look like, does it have e.g.:
● gpsd.service - GPS (Global Positioning System) Daemon
...
             └─1085 /usr/sbin/gpsd /dev/ttyUSB0 /dev/pps0

I guess for debugging you might already run it directly with something like:
  $ sudo gpsd -D 5 -N /dev/ttyS0 /dev/pps0
Is that assumption right?

The error message that you see "kernel PPS unavailable, PPS accuracy will suffer" has a comment referencing to those potential permissions issues as well.
1263 /* some operations in init_kernel_pps() require root privs */
1264 (void)init_kernel_pps(&inner_context);
1265 if ( 0 <= inner_context.kernelpps_handle ) {

We might want/need to debug the details of that init_kernel_pps function if it returns an RC.
Any chance to log onto your system?

Also maybe anything in dmesg for apparmor denies?

[1]: https://gpsd.gitlab.io/gpsd/gpsd-time-service-howto.html#_gps_time

[Christian Ehrhardt  (paelzer)]

FYI I usually used a usb GPS to check gpsd's function e.g. on a merge of a new version.
But that one had no PPS capability.

I now ordered one that should have it (via some tweaks on the USB port).
With the current times a delivery from Asia might take a few extra day.
I'll debug this further once I hopefully can reproduce this.

[Christian Ehrhardt  (paelzer)]

I'm slowly but surely ramping up my test bed.
On a broken PPS device you'd get:
  ubuntu gpsd[2585]: gpsd:ERROR: KPPS:/dev/pps0 time_pps_setparams(mode=0x1001) failed: unknown error

That happens if e.g. a device announced PPS but since it didn't initialize the kernel driver (by not soldering the PPS signal to a ping for example in my case). You are clearly past that.

But with another of my devices I think I can recreate the error now:
  gpsd[2649]: gpsd:WARN: KPPS:/dev/ttyUSB1 kernel PPS unavailable, PPS accuracy will suffer

After reducing to just the device that seems to have PPS (as the others throw a lot of false-positives) I can reproduce this and later this week hopefully debug it.

[Christian Ehrhardt  (paelzer)]

I'm seeing gpsd to set up the PPS sources (and yield it on stop):
[ 801.277262] pps pps1: new PPS source usbserial0
[ 801.277308] pps pps1: source "/dev/ttyUSB0" added
...
[ 1706.141555] pps pps1: removed

[Christian Ehrhardt  (paelzer)]

See https://bugs.launchpad.net/ubuntu/+source/gpsd/+bug/1872175/comments/7
This makes it work for me, but needs security sign off and a few more tests.

TL;DR: some extra apparmor rules for PPS

[Christian Ehrhardt  (paelzer)]

FYI - fix submitted as part of PR https://salsa.debian.org/debian-gps-team/pkg-gpsd/-/merge_requests/5

[Launchpad Janitor (janitor)]

This bug was fixed in the package gpsd - 3.20-11

---------------
gpsd (3.20-11) unstable; urgency=medium

  [ Christian Ehrhardt ]
  * [232c8d73] d/rules: fix ubxtool to use python3 in the gpsd package (LP: #1878158)
    Signed-off-by: Christian Ehrhardt <email address hidden>

 -- Bernd Zeimetz <email address hidden> Tue, 12 May 2020 13:49:15 +0200

[Christian Ehrhardt  (paelzer)]

The auto-sync in groovy also included 3.20-8/3.20-9 which covers this issue here:

gpsd (3.20-10) unstable; urgency=medium

  [ Christian Ehrhardt ]
  * [7670b61e] exchange gpsd-client/tools content
  * [fb81a96a] examples also need to be in -clients instead of -tools then
  * [7f8d06a8] d/control[.in]: moving gpsctl still breaks gpsd-clients
    (<< 3.20-9) for upgraders never seeing the version in experimental

  [ Bernd Zeimetz ]
  * [b448fb08] Merge branch 'exchange-clients-and-tools' into 'master'
    exchange gpsd-client/tools content
    See merge request debian-gps-team/pkg-gpsd!6

 -- Bernd Zeimetz <email address hidden> Mon, 11 May 2020 14:40:08 +0200

gpsd (3.20-9) experimental; urgency=medium

  [ Christian Ehrhardt ]
  * [6a21e6bd] d/usr.sbin.gpsd: improve apparmor rules for PPS
               usage (LP: #1872175 LP: #1872178)
  * [c8b703b4] d/gpsd.default: add USBAUTO option (LP: #1873415)
  * [2fcf9754] split more uncommon tools to gpsd-tools (LP: #1872189)
  * [4a1454fb] d/control[.in]: have gpsd-tools depend on gpsd-clients
               as it extends on cgps
  * Move tools for local HW config into gpsd itself
    - [016cff82] move gpsctl to package gpsd
    - [b78132aa] d/control[.in]: move ubxtool and ntpshmmon to gpsd
  * python fixups for Lintian warnings
    - d/control[.in]: add python3 dependency to gpsd-dbg
    - d/rules: fix the py2/3 fixup applied post build
    - d/rules: let package gpsd be processed by dh_python3
    - d/control[.in]: gpsd needs python as depends

Therefore this is ready for an SRU to focal now

[Christian Ehrhardt  (paelzer)]

Waiting in Focal-unapproved since yesterday - FYI autopkgtest sniff tests are good as well in https://bileto.ubuntu.com/excuses/4047/focal.html

[Robie Basak (racb)]

Hello Mark, or anyone else affected,

Accepted gpsd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gpsd/3.20-8ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Christian Ehrhardt  (paelzer)]

FYI - I'll verify this tomorrow morning once I've freed up another system.
For the sake of completeness I want to check this on another arch than where I wrote the fixes. TO ensure this really helps in a generic way.

[Christian Ehrhardt  (paelzer)]

As expected before the fix I wasn't able to use any PPS.

Upgrading to gpsd 3.20-8ubuntu0.1 worked without any further breakage.

We see on upgrade that the apparmor profile is replaced:
[12408.110277] audit: type=1400 audit(1589444818.093:34): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/gpsd" pid=7561 comm="apparmor_parser"

Without the fix (prior to the upgrade) we see apparmor denies on e.g. libusb access and such.

Along the way I found some shortcomings in our gpsd/pps doc and fixed it in the serverguide (bug 1878573).

Eventually I can confirm that with 3.20-8ubuntu0.1 I can now use pps on 20.04 properly.
  #* PPS 0 4 37 11 +257us[ +159us] +/- 53us

[Launchpad Janitor (janitor)]

This bug was fixed in the package gpsd - 3.20-8ubuntu0.1

---------------
gpsd (3.20-8ubuntu0.1) focal; urgency=medium

  * d/usr.sbin.gpsd: improve apparmor rules for PPS usage
    (LP: #1872175 LP: #1872178)

 -- Christian Ehrhardt <email address hidden> Thu, 30 Apr 2020 12:40:23 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for gpsd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.