gpsd shutdown mem abort

[Markus Linnala]

I have Raspberry 4 with USB GPS dongle and use gpsd use the dongle. When I stop gpsd, kernel throws mem abort and gpsd process is stuck.

$ sudo systemctl status gpsd
● gpsd.service - GPS (Global Positioning System) Daemon
   Loaded: loaded (/etc/systemd/system/gpsd.service; disabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-02-08 18:28:13 EET; 1 day 22h ago
  Process: 1479 ExecStart=/usr/sbin/gpsd $GPSD_OPTIONS $DEVICES (code=exited, status=0/SUCCESS)
 Main PID: 1485 (gpsd)
    Tasks: 2 (limit: 4198)
   Memory: 2.1M
   CGroup: /system.slice/gpsd.service
           └─1485 /usr/sbin/gpsd -n -s 115200 /dev/ttyACM0

Feb 08 18:28:13 rpi4-01.lan systemd[1]: Starting GPS (Global Positioning System) Daemon...
Feb 08 18:28:13 rpi4-01.lan systemd[1]: Started GPS (Global Positioning System) Daemon.

Expected:

$ sudo systemctl stop gpsd
$ sudo systemctl status gpsd
...
<gpsd stopped and no process running>
<Also no kernel call trace in dmesg.>
$ sudo systemctl start gpsd
$ sudo systemctl status gpsd
<gpsd up and running>

What happens:

$ sudo systemctl stop gpsd
<is stuck>
<There is call trace in dmesg.>

GPSD data is used by chrony via SHM socket.

Gpsd version is locally built based on https://git.bzed.at/mirror/pkg-gpsd because I needed to fix AppArmor issues, configuration issues, fix some packaging issues with py3 and split gpsd-clients to gpsd-clients-tui so that I don't need to install whole lot of graphical packages just to see my gps status over ssh terminal (cgps).

Kernel call trace seems to be always essentially the same at the top. Only possibility to fix things is to reboot.

$ lsb_release -rd
Description: Ubuntu 19.10
Release: 19.10

$ cat /proc/cmdline
coherent_pool=1M 8250.nr_uarts=1 cma=64M cma=256M smsc95xx.macaddr=DC:A6:32:68:42:57 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 net.ifnames=0 dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 elevator=deadline rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 LANG=C.UTF-8 vconsole.keymap=fi page_poison=1 pti=on slab_nomerge slub_debug=P vsyscall=none init_on_alloc=1 init_on_free=1 log_buf_len=4M printk.devkmsg=on fixrtc rootfstype=ext4 root=UUID=477f585f-b79f-4454-a9d9-bdd65f94b946

$ apt-cache policy linux-raspi2 chrony gpsd
linux-raspi2:
  Installed: 5.3.0.1017.14
  Candidate: 5.3.0.1017.14
  Version table:
 *** 5.3.0.1017.14 500
        500 http://ports.ubuntu.com/ubuntu-ports eoan-updates/universe arm64 Packages
        500 http://ports.ubuntu.com/ubuntu-ports eoan-security/universe arm64 Packages
        100 /var/lib/dpkg/status
     5.3.0.1007.3 500
        500 http://ports.ubuntu.com/ubuntu-ports eoan/universe arm64 Packages
chrony:
  Installed: 3.5-2ubuntu2
  Candidate: 3.5-2ubuntu2
  Version table:
 *** 3.5-2ubuntu2 500
        500 http://ports.ubuntu.com/ubuntu-ports eoan/main arm64 Packages
        100 /var/lib/dpkg/status
gpsd:
  Installed: 3.20-4build2
  Candidate: 3.20-4build2
  Version table:
 *** 3.20-4build2 500
        500 http://<local-ip>/eoan-arm64 Packages
        100 /var/lib/dpkg/status
     3.20-3build19 500
        500 http://<local-ip>/eoan-arm64 Packages
     3.20-3build17 500
        500 http://<local-ip>/eoan-arm64 Packages
     3.20-3build16 500
        500 http://<local-ip>/eoan-arm64 Packages
     3.20-3build13 500
        500 http://<local-ip>/eoan-arm64 Packages
     3.20-3build8 500
        500 http://<local-ip>/eoan-arm64 Packages
     3.20-3build6 500
        500 http://<local-ip>/eoan-arm64 Packages
     3.20-3build4 500
        500 http://<local-ip>/eoan-arm64 Packages
     3.17-7 500
        500 http://ports.ubuntu.com/ubuntu-ports eoan/universe arm64 Packages

[ 0.000000] kernel: Linux version 5.3.0-1017-raspi2 (buildd@bos02-arm64-063) (gcc version 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2)) #19-Ubuntu SMP Thu Jan 1
[ 0.000000] kernel: Machine model: Raspberry Pi 4 Model B Rev 1.1
...
[ 0.346539] pps_core: LinuxPPS API ver. 1 registered
[ 0.346557] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <email address hidden>
...
[ 2.966789] usb 1-1.4: new full-speed USB device number 3 using xhci_hcd
[ 3.145578] usb 1-1.4: New USB device found, idVendor=1546, idProduct=01a7, bcdDevice= 1.00
[ 3.156878] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 3.156882] usb 1-1.4: Product: u-blox 7 - GPS/GNSS Receiver
[ 3.156887] usb 1-1.4: Manufacturer: u-blox AG - www.u-blox.com
...
[ 27.330812] pps_ldisc: PPS line discipline registered
[ 27.337024] pps pps0: new PPS source acm0
[ 27.341237] pps pps0: source "/dev/ttyACM0" added
...
[163075.844341] systemd[1]: Stopping Manage ttyACM0 for GPS daemon...
[163075.879783] gpsdctl[10713]: gpsd_control(action=remove, arg=/dev/ttyACM0)
[163075.883507] gpsdctl[10713]: reached a running gpsd
[163075.899234] systemd[1]: Stopping GPS (Global Positioning System) Daemon...
[163079.462980] kernel: pps pps0: removed
[163079.471169] kernel: Unable to handle kernel paging request at virtual address 006b6b6b6b6b6eb3
[163079.479456] kernel: Mem abort info:
[163079.482496] kernel: ESR = 0x96000004
[163079.482502] kernel: Exception class = DABT (current EL), IL = 32 bits
[163079.482505] kernel: SET = 0, FnV = 0
[163079.482512] kernel: EA = 0, S1PTW = 0
[163079.498497] kernel: Data abort info:
[163079.498500] kernel: ISV = 0, ISS = 0x00000004
[163079.498503] kernel: CM = 0, WnR = 0
[163079.498507] kernel: [006b6b6b6b6b6eb3] address between user and kernel address ranges
[163079.498512] kernel: Internal error: Oops: 96000004 [#1] SMP
[163079.498516] kernel: Modules linked in: pps_ldisc netconsole nls_ascii dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua btsdio bluetooth bcm2835_v4l2(CE
[163079.498597] kernel: genet gpio_regulator phy_generic fixed aes_neon_bs aes_neon_blk crypto_simd cryptd aes_arm64
[163079.610369] kernel: CPU: 2 PID: 1548 Comm: gpsd Tainted: G C E 5.3.0-1017-raspi2 #19-Ubuntu
[163079.610371] kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)
[163079.610374] kernel: pstate: 40400005 (nZcv daif +PAN -UAO)
[163079.610391] kernel: pc : module_put+0x24/0x100
[163079.644256] kernel: lr : cdev_put.part.0+0x2c/0x38
[163079.644259] kernel: sp : ffff000010b93c70
[163079.644260] kernel: x29: ffff000010b93c70 x28: ffffaabc70a41dc0
[163079.644264] kernel: x27: 0000000000000000 x26: ffffaabc58ea88d0
[163079.644267] kernel: x25: ffffaabc68c74060 x24: ffffaabc73bcf898
[163079.644269] kernel: x23: ffffaabc76597a60 x22: ffffaabc68c74060
[163079.644272] kernel: x21: ffff5b638471f7b4 x20: 0000000000000008
[163079.644275] kernel: x19: 6b6b6b6b6b6b6b6b x18: 0000000000000000
[163079.685076] kernel: x17: 0000000000000000 x16: 0000000000000000
[163079.685080] kernel: x15: 0000000000000004 x14: 0000000000000000
[163079.685083] kernel: x13: 0000000000000000 x12: ffffaabc578f26f0
[163079.685085] kernel: x11: ffffaabc578f26c8 x10: 0000000000000040
[163079.685088] kernel: x9 : 0000000000000000 x8 : 0000000000000001
[163079.685090] kernel: x7 : ffffaabc5874ea80 x6 : 0000000000000000
[163079.685092] kernel: x5 : 0000000000000000 x4 : 000000006b6b6b6b
[163079.685095] kernel: x3 : 000000006b6b6b6a x2 : 000000006b6b6b6b
[163079.685097] kernel: x1 : ffffaabc6a8f0d98 x0 : ffff5b638471f7b4
[163079.685100] kernel: Call trace:
[163079.685110] kernel: module_put+0x24/0x100
[163079.685119] kernel: cdev_put.part.0+0x2c/0x38
[163079.744716] kernel: cdev_put+0x24/0x30
[163079.744721] kernel: __fput+0x208/0x220
[163079.744724] kernel: ____fput+0x20/0x30
[163079.744729] kernel: task_work_run+0xd8/0x108
[163079.744734] kernel: do_exit+0x2f8/0xa80
[163079.744737] kernel: do_group_exit+0x48/0xb8
[163079.744748] kernel: __arm64_sys_exit_group+0x24/0x28
[163079.774584] kernel: el0_svc_common.constprop.0+0xe0/0x1e8
[163079.774587] kernel: el0_svc_handler+0x34/0xa0
[163079.774591] kernel: el0_svc+0x10/0x14
[163079.774603] kernel: Code: aa1e03f5 aa1e03e0 d503201f b4000213 (b9434a60)
[163079.798836] kernel: ---[ end trace d6816541b4d65710 ]---
[163079.798892] kernel: Fixing recursive fault but reboot is needed!

Bus 001 Device 003: ID 1546:01a7 U-Blox AG [u-blox 7]

Here is another call trace.

[ 350.112526] kernel: Unable to handle kernel paging request at virtual address 006b6b6b6b6b6eb3
[ 350.120764] kernel: Mem abort info:
[ 350.120768] kernel: ESR = 0x96000004
[ 350.120772] kernel: Exception class = DABT (current EL), IL = 32 bits
[ 350.120775] kernel: SET = 0, FnV = 0
[ 350.120777] kernel: EA = 0, S1PTW = 0
[ 350.120779] kernel: Data abort info:
[ 350.120782] kernel: ISV = 0, ISS = 0x00000004
[ 350.120785] kernel: CM = 0, WnR = 0
[ 350.120788] kernel: [006b6b6b6b6b6eb3] address between user and kernel address ranges
[ 350.120795] kernel: Internal error: Oops: 96000004 [#1] SMP
[ 350.120800] kernel: Modules linked in: pps_ldisc netconsole nls_ascii dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua btsdio bluetooth ecdh_generic ecc
[ 350.120896] kernel: gpio_regulator genet phy_generic fixed aes_neon_bs aes_neon_blk crypto_simd cryptd aes_arm64
[ 350.120909] kernel: CPU: 0 PID: 1729 Comm: gpsd Tainted: G C E 5.3.0-1017-raspi2 #19-Ubuntu
[ 350.120911] kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)
[ 350.120914] kernel: pstate: 40400005 (nZcv daif +PAN -UAO)
[ 350.120929] kernel: pc : module_put+0x24/0x100
[ 350.120933] kernel: lr : cdev_put.part.0+0x2c/0x38
[ 350.120935] kernel: sp : ffff000010bcbbb0
[ 350.120936] kernel: x29: ffff000010bcbbb0 x28: 0000000000000008
[ 350.120939] kernel: x27: ffff2caebe929000 x26: ffff837bafe9a290
[ 350.120942] kernel: x25: ffff837ba8c41330 x24: ffff837bb415b960
[ 350.120944] kernel: x23: ffff837bb6596660 x22: ffff837ba8c41330
[ 350.120947] kernel: x21: ffff2caebd91f7b4 x20: 0000000000000008
[ 350.120949] kernel: x19: 6b6b6b6b6b6b6b6b x18: 0000000000000000
[ 350.120951] kernel: x17: 0000000000000000 x16: 0000000000000000
[ 350.120953] kernel: x15: 0000000000000004 x14: 0000000000000000
[ 350.120955] kernel: x13: 0000000000000000 x12: ffff837bb41736e8
[ 350.120958] kernel: x11: ffff837bb41736c0 x10: 0000000000000040
[ 350.120960] kernel: x9 : 0000000000000000 x8 : 0000000000000001
[ 350.120962] kernel: x7 : ffff837ba7795080 x6 : 0000000000000000
[ 350.120965] kernel: x5 : 0000000000000000 x4 : 000000006b6b6b6b
[ 350.120967] kernel: x3 : 000000006b6b6b6a x2 : 000000006b6b6b6b
[ 350.120970] kernel: x1 : ffff837bb1651518 x0 : ffff2caebd91f7b4
[ 350.120973] kernel: Call trace:
[ 350.120977] kernel: module_put+0x24/0x100
[ 350.120980] kernel: cdev_put.part.0+0x2c/0x38
[ 350.120982] kernel: cdev_put+0x24/0x30
[ 350.120986] kernel: __fput+0x208/0x220
[ 350.120989] kernel: ____fput+0x20/0x30
[ 350.120994] kernel: task_work_run+0xd8/0x108
[ 350.120997] kernel: do_exit+0x2f8/0xa80
[ 350.121000] kernel: do_group_exit+0x48/0xb8
[ 350.121004] kernel: get_signal+0x15c/0x868
[ 350.121007] kernel: do_notify_resume+0x1d4/0x420
[ 350.121010] kernel: work_pending+0x8/0x14
[ 350.121018] kernel: Code: aa1e03f5 aa1e03e0 d503201f b4000213 (b9434a60)
[ 350.121021] kernel: ---[ end trace 5f7f45230e415b7c ]---
[ 350.121023] kernel: Fixing recursive fault but reboot is needed!
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu8.2
Architecture: arm64
DistroRelease: Ubuntu 19.10
Package: linux-raspi2 5.3.0.1017.14
PackageArchitecture: arm64
ProcVersionSignature: User Name 5.3.0-1017.19-raspi2 5.3.13
Tags: eoan
Uname: Linux 5.3.0-1017-raspi2 aarch64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
_MarkForUpload: True

[Markus Linnala (maage999)]

apport information

[Markus Linnala (maage999)]

apport information

[Markus Linnala (maage999)]

apport information

[Markus Linnala (maage999)]

 I downgraded gpsd to

$ apt-cache policy gpsd
gpsd:
  Installed: 3.17-7
  Candidate: 3.17-7
  Version table:
 *** 3.17-7 500
        500 http://ports.ubuntu.com/ubuntu-ports eoan/universe arm64 Packages
        100 /var/lib/dpkg/status

And this issue is still occurring.

I've added also latest dmesg.txt with this configuration.