Ubuntu
systemd package

systemctl daemon-reload makes systemd crash

Jammy (22.04)
Bug #2100501

Bug #2100501 reported by Marcus Kool on 2025-02-27

6

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	systemd (Ubuntu)	Invalid	Undecided	Unassigned
	Jammy	New	Undecided	Unassigned

Bug Description

During the last months systemd has dumped core on two systems with Ubuntu 22.04.
This time it was triggered by "systemctl daemon-reload".

As you probably may know the only thing to do to resolve a system crash is to reboot the server which is *very* disturbing for our services.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: systemd 249.11-0ubuntu3.12 [modified: usr/lib/sysctl.d/50-pid-max.conf]
ProcVersionSignature: Ubuntu 5.15.0-131.141-generic 5.15.168
Uname: Linux 5.15.0-131-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu82.6
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Thu Feb 27 15:52:07 2025
Lsusb:
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
MachineType: ASUS System Product Name
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.15.0-131-generic root=UUID=1e666565-cb2f-4e78-90aa-0fdc62211613 ro consoleblank=0 systemd.show_status=true consoleblank=0
RebootRequiredPkgs: Error: path contained symlinks.
SourcePackage: systemd
SystemdFailedUnits: Error: command ['systemctl', 'status', '--full', 'Error:'] failed with exit code 1: Failed to get properties: Connection timed out
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 10/13/2022
dmi.bios.release: 5.17
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 9901
dmi.board.asset.tag: Default string
dmi.board.name: Pro WS 565-ACE
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev X.0x
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr9901:bd10/13/2022:br5.17:svnASUS:pnSystemProductName:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnProWS565-ACE:rvrRevX.0x:cvnDefaultstring:ct3:cvrDefaultstring:skuSKU:
dmi.product.family: To be filled by O.E.M.
dmi.product.name: System Product Name
dmi.product.sku: SKU
dmi.product.version: System Version
dmi.sys.vendor: ASUS
mtime.conffile..etc.systemd.journald.conf: 2024-02-21T20:35:18.546041
mtime.conffile..etc.systemd.resolved.conf: 2024-02-22T18:08:06.586859

Tags:

Revision history for this message

Marcus Kool (mhkool) wrote on 2025-02-27:

#1

CurrentDmesg.txt Edit (105.5 KiB, text/plain; charset="utf-8")
Dependencies.txt Edit (3.2 KiB, text/plain; charset="utf-8")
Lspci.txt Edit (85.0 KiB, text/plain; charset="utf-8")
Lspci-vt.txt Edit (2.9 KiB, text/plain; charset="utf-8")
Lsusb-v.txt Edit (10.9 KiB, text/plain; charset="utf-8")
ProcCpuinfo.txt Edit (49.2 KiB, text/plain; charset="utf-8")
ProcCpuinfoMinimal.txt Edit (1.5 KiB, text/plain; charset="utf-8")
ProcEnviron.txt Edit (280 bytes, text/plain; charset="utf-8")
ProcInterrupts.txt Edit (44.7 KiB, text/plain; charset="utf-8")
ProcModules.txt Edit (7.4 KiB, text/plain; charset="utf-8")
SystemdDelta.txt Edit (2.4 KiB, text/plain; charset="utf-8")
UdevDb.txt Edit (243.9 KiB, text/plain; charset="utf-8")
acpidump.txt Edit (730.2 KiB, text/plain; charset="utf-8")
modified.conffile..etc.systemd.journald.conf.txt Edit (1.3 KiB, text/plain; charset="utf-8")
modified.conffile..etc.systemd.resolved.conf.txt Edit (1.4 KiB, text/plain; charset="utf-8")

Revision history for this message

Nick Rosbrook (enr0n) wrote on 2025-02-27:

#2

Is this sporadic then? Or reproducible under some circumstances? And is there a crash file for systemd under /var/crash? That would hopefully have a stack trace which would make this easier to debug.

Changed in systemd (Ubuntu):
status:	New → Invalid
Changed in systemd (Ubuntu Jammy):
status:	New → Incomplete

Revision history for this message

Marcus Kool (mhkool) wrote on 2025-02-27 (last edit on 2025-02-27):

#3

The systemd crash has now happened 4 times over the last couple of months. We use systemctl sporadically and I do not recall if all crashes were caused by 'systemctl daemon-reload' so I cannot deny nor confirm that it is reproducible.

There is a crash file. It can be downloaded here:
https://files.urlfilterdb.com/crash/_usr_lib_systemd_systemd.0.crash

Revision history for this message

Marcus Kool (mhkool) wrote on 2025-03-02:

#4

I see that the status is still 'incomplete'.
Do you need more information?

Revision history for this message

Marcus Kool (mhkool) wrote on 2025-03-05:

#5

crash file provided

Changed in systemd (Ubuntu Jammy):
status:	Incomplete → New

Revision history for this message

Nick Rosbrook (enr0n) wrote on 2025-03-18:

#6

This is what I see re-tracing the crash:

root@jammy:~# apport-retrace --gdb _usr_lib_systemd_systemd.0.crash
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from /usr/lib/systemd/systemd...
Reading symbols from /usr/lib/debug/.build-id/2a/b06decf08c9378367b98f3e592473ea32a5b3c.debug...
warning: Can't open file /usr/lib/x86_64-linux-gnu/libcap.so.2.44 (deleted) during file-backed mapping note processing
[New LWP 117730]
warning: Section `.reg-xstate/117730' in core file too small.
[New LWP 1]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/sbin/init'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Section `.reg-xstate/117730' in core file too small.
#0 0x00007f641f0dd75b in kill () at ../sysdeps/unix/syscall-template.S:120
120 ../sysdeps/unix/syscall-template.S: No such file or directory.
[Current thread is 1 (LWP 117730)]
(gdb) bt
#0 0x00007f641f0dd75b in kill () at ../sysdeps/unix/syscall-template.S:120
#1 0x0000555c7065db1c in crash (sig=11) at ../src/core/main.c:257
#2 <signal handler called>
#3 0x0000555c706d7d73 in unit_active_state (u=<optimized out>) at ../src/core/unit.c:830
#4 0x0000555c706e7d86 in unit_may_gc (u=0x555c807d68e0) at ../src/core/unit.c:392
#5 0x0000555c706d69fe in unit_may_gc (u=0x555c807d68e0) at ../src/core/unit.c:389
#6 unit_add_to_gc_queue (u=0x555c807d68e0) at ../src/core/unit.c:468
#7 0x0000555c706e8ab1 in unit_clear_dependencies (u=0x555c80799e50) at ../src/core/unit.c:555
#8 unit_free (u=0x555c80799e50) at ../src/core/unit.c:698
#9 0x0000555c706a8d05 in unit_free (u=<optimized out>) at ../src/core/unit.c:654
#10 manager_clear_jobs_and_units (m=m@entry=0x555c806ac890) at ../src/core/manager.c:1421
#11 0x0000555c706ae01a in manager_clear_jobs_and_units (m=0x555c806ac890) at ../src/core/manager.c:1418
#12 manager_reload (m=0x555c806ac890) at ../src/core/manager.c:3826
#13 0x0000555c7065ca7b in invoke_main_loop (ret_error_message=0x7ffeda362240, ret_switch_root_init=<synthetic pointer>, ret_switch_root_dir=<synthetic pointer>, ret_fds=0x7ffeda362230, ret_shutdown_verb=<synthetic pointer>,
ret_retval=<synthetic pointer>, ret_reexecute=<synthetic pointer>, saved_rlimit_memlock=0x7ffeda362270, saved_rlimit_nofile=0x7ffeda362280, m=0x555c806ac890) at ../src/core/main.c:1934
#14 main (argc=1, argv=0x7ffeda362508) at ../src/core/main.c:2910

Which sounds a lot like bug 2098861.

This is what I see re-tracing the crash:

root@jammy:~# apport-retrace --gdb _usr_lib_systemd_systemd.0.crash 
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from /usr/lib/systemd/systemd...
Reading symbols from /usr/lib/debug/.build-id/2a/b06decf08c9378367b98f3e592473ea32a5b3c.debug...
warning: Can't open file /usr/lib/x86_64-linux-gnu/libcap.so.2.44 (deleted) during file-backed mapping note processing
[New LWP 117730]
warning: Section `.reg-xstate/117730' in core file too small.
[New LWP 1]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/sbin/init'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Section `.reg-xstate/117730' in core file too small.
#0  0x00007f641f0dd75b in kill () at ../sysdeps/unix/syscall-template.S:120
120	../sysdeps/unix/syscall-template.S: No such file or directory.
[Current thread is 1 (LWP 117730)]
(gdb) bt
#0  0x00007f641f0dd75b in kill () at ../sysdeps/unix/syscall-template.S:120
#1  0x0000555c7065db1c in crash (sig=11) at ../src/core/main.c:257
#2  <signal handler called>
#3  0x0000555c706d7d73 in unit_active_state (u=<optimized out>) at ../src/core/unit.c:830
#4  0x0000555c706e7d86 in unit_may_gc (u=0x555c807d68e0) at ../src/core/unit.c:392
#5  0x0000555c706d69fe in unit_may_gc (u=0x555c807d68e0) at ../src/core/unit.c:389
#6  unit_add_to_gc_queue (u=0x555c807d68e0) at ../src/core/unit.c:468
#7  0x0000555c706e8ab1 in unit_clear_dependencies (u=0x555c80799e50) at ../src/core/unit.c:555
#8  unit_free (u=0x555c80799e50) at ../src/core/unit.c:698
#9  0x0000555c706a8d05 in unit_free (u=<optimized out>) at ../src/core/unit.c:654
#10 manager_clear_jobs_and_units (m=m@entry=0x555c806ac890) at ../src/core/manager.c:1421
#11 0x0000555c706ae01a in manager_clear_jobs_and_units (m=0x555c806ac890) at ../src/core/manager.c:1418
#12 manager_reload (m=0x555c806ac890) at ../src/core/manager.c:3826
#13 0x0000555c7065ca7b in invoke_main_loop (ret_error_message=0x7ffeda362240, ret_switch_root_init=<synthetic pointer>, ret_switch_root_dir=<synthetic pointer>, ret_fds=0x7ffeda362230, ret_shutdown_verb=<synthetic pointer>, 
    ret_retval=<synthetic pointer>, ret_reexecute=<synthetic pointer>, saved_rlimit_memlock=0x7ffeda362270, saved_rlimit_nofile=0x7ffeda362280, m=0x555c806ac890) at ../src/core/main.c:1934
#14 main (argc=1, argv=0x7ffeda362508) at ../src/core/main.c:2910

Which sounds a lot like bug 2098861.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.