systemd Caught <SEGV>, dumped core

[Arie Skliarouk]

Several days ago an ubuntu 18.04 in ec2 env updated systemd and udev to 237-3ubuntu10.31
This caused systemd to segfault and stop receiving any systemctl command (they timeout).

[ 198.482652] systemd[1]: segfault at 50 ip 00005630ddd60200 sp 00007ffc9c3ea890 error 4 in systemd (deleted)[5630ddcc1000+14e000]
[ 278.664729] systemd-journald[451]: Failed to send WATCHDOG=1 notification message: Connection refused
[ 368.462136] systemd-journald[451]: Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected
[ 458.496689] systemd-journald[451]: Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected

I downgraded to 237-3ubuntu10.29 to no avail.
Then I downgraded to 237-3ubuntu10:
apt install udev=237-3ubuntu10 systemd=237-3ubuntu10 systemd-sysv=237-3ubuntu10 libsystemd0:amd64=237-3ubuntu10 libudev1:amd64=237-3ubuntu10 libnss-systemd:amd64=237-3ubuntu10

and that did not help either.
The machine is unusable at the moment.

Attaching coredump of systemd (version 237-3ubuntu10.31)

[Arie Skliarouk (skliarie)]

This started happening after automatic upgrade:

2019-11-14 06:57:49 startup packages remove
2019-11-14 06:57:49 status installed linux-image-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:50 remove linux-image-4.15.0-1051-aws:amd64 4.15.0-1051.53 <none>
2019-11-14 06:57:50 status half-configured linux-image-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:50 status half-installed linux-image-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:52 status config-files linux-image-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:52 status config-files linux-image-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:52 status installed linux-modules-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:52 remove linux-modules-4.15.0-1051-aws:amd64 4.15.0-1051.53 <none>
2019-11-14 06:57:52 status half-configured linux-modules-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:52 status half-installed linux-modules-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:52 status config-files linux-modules-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:52 status config-files linux-modules-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:52 startup packages configure
2019-11-14 06:57:54 startup packages remove
2019-11-14 06:57:54 status installed linux-headers-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:54 remove linux-headers-4.15.0-1051-aws:amd64 4.15.0-1051.53 <none>
2019-11-14 06:57:54 status half-configured linux-headers-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:54 status half-installed linux-headers-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:56 status config-files linux-headers-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:56 status config-files linux-headers-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:56 status config-files linux-headers-4.15.0-1051-aws:amd64 4.15.0-1051.53
2019-11-14 06:57:56 status not-installed linux-headers-4.15.0-1051-aws:amd64 <none>
2019-11-14 06:57:56 startup packages configure
2019-11-14 06:57:59 startup archives unpack
2019-11-14 06:57:59 upgrade linux-libc-dev:amd64 4.15.0-69.78 4.15.0-70.79
2019-11-14 06:57:59 status half-configured linux-libc-dev:amd64 4.15.0-69.78
2019-11-14 06:57:59 status unpacked linux-libc-dev:amd64 4.15.0-69.78
2019-11-14 06:57:59 status half-installed linux-libc-dev:amd64 4.15.0-69.78
2019-11-14 06:57:59 status half-installed linux-libc-dev:amd64 4.15.0-69.78
2019-11-14 06:57:59 status unpacked linux-libc-dev:amd64 4.15.0-70.79
2019-11-14 06:57:59 status unpacked linux-libc-dev:amd64 4.15.0-70.79
2019-11-14 06:57:59 startup packages configure
2019-11-14 06:57:59 configure linux-libc-dev:amd64 4.15.0-70.79 <none>
2019-11-14 06:57:59 status unpacked linux-libc-dev:amd64 4.15.0-70.79
2019-11-14 06:57:59 status half-configured linux-libc-dev:amd64 4.15.0-70.79
2019-11-14 06:57:59 status installed linux-libc-dev:amd64 4.15.0-70.79
2019-11-14 06:58:01 startup archives unpack
2019-11-14 06:58:02 upgrade qemu-guest-agent:amd64 1:2.11+dfsg-1ubuntu7.15 1:2.11+dfsg-1ubuntu7.20
2019-11-14 06:58:02 status half-configured qemu-guest-agent:amd64 1:2.11+dfsg-1ubuntu7.15
2019-11-14 06:58:02 status unpacked qemu-guest-agent:...

[Dan Streetman (ddstreet)]

systemd crashed while trying to serialize, on a service that appears to not be provided by Ubuntu, '/lib/systemd/system/keepsessions.service'. It looks like it's part of a 'sitetools' package (the exec command is "/opt/sitetools/keepsessions").

This is probably a bug in systemd, but it would help to see the specific details of that service file; can you attach or paste it here?

[Launchpad Janitor (janitor)]

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]

[Arie Skliarouk (skliarie)]

Yes, it is our custom service:
# cat /var/tmp/keepsessions.service
[Unit]
Description=SiteTools keep sessions
Wants=network-online.target
Wants=docker.service
After=network-online.target
After=docker.service

[Service]
ExecStart=/opt/sitetools/keepsessions
ExecStartPost=/opt/sitetools/onstart
Restart=always
WatchdogSec=5m
Type=simple

[Install]
WantedBy=multi-user.target

It does usual stuff, like open/close files, connect to consul, etc. Written in python3, nothing special.