systemd dumps core on starting service: Assertion 'u->instance' failed at src/core/load-fragment.c:4798

Bug #1887744 reported by Jeff Turner
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd
Fix Released
Unknown
systemd (Ubuntu)
Medium
Dan Streetman
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Focal
Medium
Dan Streetman
Groovy
Medium
Dan Streetman
Hirsute
Medium
Dan Streetman

Bug Description

[impact]

if a non-template service file is created as a symlink to a template service file outside the normal unit file paths, systemd will fail assertion and crash

[test case]

su -
cat <<EOF > /crashsystemd@.service
[Unit]
Description=Template for instance %i

[Service]
ExecStart=/bin/true
EOF

cd /etc/systemd/system
ln -s /crashsystemd@.service crashsystemd.service
systemctl daemon-reload
systemctl start crashsystemd

[regression potential]

any regression would likely cause a failed assertion and crash, or failure to properly parse and/or load a service file that is a symlink

[scope]

this is needed in f/g/h

this is fixed upstream by commit 3aa57658434e7a95c6000bebb166c31f1c6d051b which is included in v248-rc1, so this is not fixed in any release yet

this was introduced by the changes in commit e8630e69523 which was first included in v243, so this bug doesn't exist in b or earlier

[original description]

On an Ubuntu 20.04 server, when I started a newly installed systemd service 'systemctl start testemail2jira.service', systemd trips on an assertion (from journalctl):

Jul 16 13:49:12 pangolin-jiraconf systemd[1]: Assertion 'u->instance' failed at src/core/load-fragment.c:4798, function unit_load_fragment(). Aborting.
Jul 16 13:49:13 pangolin-jiraconf systemd[1]: Caught <ABRT>, dumped core as pid 2906637.
Jul 16 13:49:13 pangolin-jiraconf systemd[1]: Freezing execution.

apport generates a core dump which I trust is attached.

The server still works, but my terminal keeps showing:

Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

and pauses for 25s after each command.

In case it's relevant, the <service> i was trying to start had just been symlinked into /etc/systemd/system/:

root@pangolin-jiraconf # ls -la /etc/systemd/system/testemail2jira.service
lrwxrwxrwx 1 root root 82 Jul 16 13:48 /etc/systemd/system/testemail2jira.service -> /opt/atlassian/jira/current/emailprocessing/testemail2jira/testemail2jira@.service

File contents:

[Unit]
Description=Check that email to %i JIRA works

[Service]
#Type=oneshot
# Note: I keep the config file, testemail2jira.json5 and rsa.* files in emailprocessing/, one directory below testemail2jira/, so that they can be version-controlled independently.
# If you want testemail2jira.json5 and rsa.* within testemail2jira/, then add testemail2jira/ to WorkingDirectory and remove it from the ExecStart command
WorkingDirectory=/opt/atlassian/%i/current/emailprocessing
ExecStart=/opt/atlassian/%i/current/emailprocessing/testemail2jira/venv/bin/python testemail2jira/testemail2jira.py
User=root

[Install]
WantedBy=multi-user.target

Perhaps the problem is that my service file is a template (testemail2jira@.service), but I symlinked it as a regular 'testemail2jira.service'.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: systemd 245.4-4ubuntu3.1
ProcVersionSignature: Ubuntu 5.4.0-39.43-generic 5.4.41
Uname: Linux 5.4.0-39-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu27.3
Architecture: amd64
CasperMD5CheckResult: skip
Date: Thu Jul 16 14:13:06 2020
InstallationDate: Installed on 2020-05-27 (49 days ago)
InstallationMedia: Ubuntu-Server 20.04 LTS "Focal Fossa" - Release amd64 (20200423)
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:

Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-39-generic root=/dev/sda ro console=ttyS0,19200n8 net.ifnames=0
SourcePackage: systemd
SystemdFailedUnits: Error: command ['systemctl', 'status', '--full', 'Error:'] failed with exit code 1: Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: rel-1.12.0-0-ga698c8995f-prebuilt.qemu.org
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-3.1
dmi.modalias: dmi:bvnSeaBIOS:bvrrel-1.12.0-0-ga698c8995f-prebuilt.qemu.org:bd04/01/2014:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-3.1:cvnQEMU:ct1:cvrpc-q35-3.1:
dmi.product.name: Standard PC (Q35 + ICH9, 2009)
dmi.product.version: pc-q35-3.1
dmi.sys.vendor: QEMU

Revision history for this message
Jeff Turner (jefft7610) wrote :
Revision history for this message
Dan Streetman (ddstreet) wrote :

> apport generates a core dump which I trust is attached.

unfortunately i don't see it attached; could you manually attach it please?

Revision history for this message
Jeff Turner (jefft7610) wrote :

Attached is /var/crash/_usr_lib_systemd_systemd.0.crash

Revision history for this message
Jeff Turner (jefft7610) wrote :

Replicated on my local Ubuntu 20.04 system with systemd 245.4-4ubuntu3.1.

It's a PEBCAK error where a user writes a .service file containing %i's, but instead of naming the file <email address hidden>, just names it foo.service. To replicate:

cd /etc/systemd/system
curl -s -o testemail2jira.service 'https://hg.sr.ht/~redradishtech/testemail2jira/raw/testemail2jira%40.service?rev=tip'
systemctl daemon-reload
systemctl start testemail2jira

On one of the last two steps everything freezes, errors start appearing:

[system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

and /var/crash/_usr_lib_systemd_systemd.0.crash is created.

Revision history for this message
Dan Streetman (ddstreet) wrote :

> To replicate

I can't reproduce it with those steps, but maybe it's because i dont have the same content under /opt/... that you do; can you provide simpler repro steps that doesn't require anything installed under /opt/... ?

Revision history for this message
Jeff Turner (jefft7610) wrote :

Sorry, turns out the /etc/systemd/system service file needs to be a symlink to trigger the bug. This will do it:

su -
cat <<EOF > /crashsystemd@.service
[Unit]
Description=Template for instance %i

[Service]
ExecStart=/bin/true
EOF

cd /etc/systemd/system
ln -s /crashsystemd@.service crashsystemd.service
systemctl daemon-reload
systemctl start crashsystemd

After the 'systemctl start crashsystemd' my terminal says:

Failed to start crashsystemd.service: Connection reset by peer
See system logs and 'systemctl status crashsystemd.service' for details.

and in the systemd logs I see:

Jul 17 23:20:33 jturner-desktop systemd[1]: Assertion 'u->instance' failed at src/core/load-fragment.c:4798, function unit_load_fragment(). Aborting.

Things are pretty broken now. 'systemctl' and 'reboot' commands won't work. Don't forget to delete the /etc/systemd/system/crashsystemd.service symlink before rebooting.

Revision history for this message
Dan Streetman (ddstreet) wrote :

> Sorry, turns out the /etc/systemd/system service file needs to be a
> symlink to trigger the bug. This will do it:

aha! and not only a symlink, but the target of the symlink needs to be outside the normal systemd service path locations, or it will reject loading it:

root@lp1887744-f:/etc/systemd/system# ls -l test*
lrwxrwxrwx 1 root root 13 Jul 20 18:17 test.service -> test@.service
-rw-r--r-- 1 root root 76 Jul 20 18:14 test@.service

root@lp1887744-f:/etc/systemd/system# journalctl --no-pager -q -b --grep test.service
Jul 20 18:17:33 lp1887744-f systemd[1]: /etc/systemd/system/test.service: symlink target name type "test@.service" does not match source, rejecting.

Changed in systemd (Ubuntu):
status: New → Confirmed
Changed in systemd:
status: Unknown → New
Changed in systemd:
status: New → Fix Released
Dan Streetman (ddstreet)
description: updated
Changed in systemd (Ubuntu Hirsute):
importance: Undecided → Medium
Changed in systemd (Ubuntu Groovy):
importance: Undecided → Medium
Changed in systemd (Ubuntu Focal):
importance: Undecided → Medium
Changed in systemd (Ubuntu Bionic):
importance: Undecided → Medium
Changed in systemd (Ubuntu Xenial):
importance: Undecided → Medium
Changed in systemd (Ubuntu Hirsute):
status: Confirmed → In Progress
Changed in systemd (Ubuntu Groovy):
status: New → In Progress
Changed in systemd (Ubuntu Focal):
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
status: New → In Progress
Changed in systemd (Ubuntu Xenial):
status: New → In Progress
Changed in systemd (Ubuntu Hirsute):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Groovy):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Focal):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Xenial):
assignee: nobody → Dan Streetman (ddstreet)
Dan Streetman (ddstreet)
description: updated
Changed in systemd (Ubuntu Xenial):
status: In Progress → Invalid
Changed in systemd (Ubuntu Bionic):
status: In Progress → Invalid
assignee: Dan Streetman (ddstreet) → nobody
Changed in systemd (Ubuntu Xenial):
assignee: Dan Streetman (ddstreet) → nobody
Changed in systemd (Ubuntu Bionic):
importance: Medium → Undecided
Changed in systemd (Ubuntu Xenial):
importance: Medium → Undecided
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Ok, so this is still not something we have in hirsute? Do we know when 248 will be pulled in? Is it still something we plan for hirsute?

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

I'll review this and consider it for inclusion as an SRU, but will not release them before it is available in hirsute (either via 248, or a cherry-pick on top of the current release).

tags: added: block-proposed-focal block-proposed-groovy
Changed in systemd (Ubuntu Groovy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-groovy
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Jeff, or anyone else affected,

Accepted systemd into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/246.6-1ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Jeff, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/246.6-1ubuntu1.3)

All autopkgtests for the newly accepted systemd (246.6-1ubuntu1.3) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

prometheus/2.20.0+ds-1 (amd64)
asterisk/1:16.12.0~dfsg-1 (amd64)
apt/2.1.10ubuntu0.2 (arm64)
flatpak/1.8.2-1ubuntu0.1 (ppc64el)
udisks2/2.9.1-2ubuntu1 (amd64)
gvfs/1.46.1-1ubuntu1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/245.4-4ubuntu3.6)

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.6) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

multipath-tools/0.8.3-1ubuntu2 (s390x)
munin/2.0.56-1ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Dan Streetman (ddstreet) wrote :

root@lp1887744-g:~# dpkg -l systemd|grep systemd
ii systemd 246.6-1ubuntu1.2 amd64 system and service manager
root@lp1887744-g:~# cat <<EOF > /crashsystemd@.service
> [Unit]
> Description=Template for instance %i
>
> [Service]
> ExecStart=/bin/true
> EOF
root@lp1887744-g:~#
root@lp1887744-g:~# cd /etc/systemd/system
root@lp1887744-g:/etc/systemd/system# ln -s /crashsystemd@.service crashsystemd.service
root@lp1887744-g:/etc/systemd/system# systemctl daemon-reload
root@lp1887744-g:/etc/systemd/system# systemctl start crashsystemd
Failed to start crashsystemd.service: Connection reset by peer
See system logs and 'systemctl status crashsystemd.service' for details.

root@lp1887744-g:~# dpkg -l systemd|grep systemd
ii systemd 246.6-1ubuntu1.3 amd64 system and service manager
root@lp1887744-g:~# cat <<EOF > /crashsystemd@.service
> [Unit]
> Description=Template for instance %i
>
> [Service]
> ExecStart=/bin/true
> EOF
root@lp1887744-g:~#
root@lp1887744-g:~# cd /etc/systemd/system
root@lp1887744-g:/etc/systemd/system# ln -s /crashsystemd@.service crashsystemd.service
root@lp1887744-g:/etc/systemd/system# systemctl daemon-reload
root@lp1887744-g:/etc/systemd/system# systemctl start crashsystemd
root@lp1887744-g:/etc/systemd/system#

Revision history for this message
Dan Streetman (ddstreet) wrote :

root@lp1887744-f:~# dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.5 amd64 system and service manager
root@lp1887744-f:~# cat <<EOF > /crashsystemd@.service
> [Unit]
> Description=Template for instance %i
>
> [Service]
> ExecStart=/bin/true
> EOF
root@lp1887744-f:~#
root@lp1887744-f:~# cd /etc/systemd/system
root@lp1887744-f:/etc/systemd/system# ln -s /crashsystemd@.service crashsystemd.service
root@lp1887744-f:/etc/systemd/system# systemctl daemon-reload
root@lp1887744-f:/etc/systemd/system# systemctl start crashsystemd
Failed to start crashsystemd.service: Connection reset by peer
See system logs and 'systemctl status crashsystemd.service' for details.

root@lp1887744-f:~# dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.6 amd64 system and service manager
root@lp1887744-f:~# cat <<EOF > /crashsystemd@.service
> [Unit]
> Description=Template for instance %i
>
> [Service]
> ExecStart=/bin/true
> EOF
root@lp1887744-f:~#
root@lp1887744-f:~# cd /etc/systemd/system
root@lp1887744-f:/etc/systemd/system# ln -s /crashsystemd@.service crashsystemd.service
root@lp1887744-f:/etc/systemd/system# systemctl daemon-reload
root@lp1887744-f:/etc/systemd/system# systemctl start crashsystemd
root@lp1887744-f:/etc/systemd/system#

tags: added: verification-done verification-done-focal verification-done-groovy
removed: verification-needed verification-needed-focal verification-needed-groovy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 247.3-3ubuntu2

---------------
systemd (247.3-3ubuntu2) hirsute; urgency=medium

  [ Dan Streetman ]
  * d/p/lp1887744-basic-unit-file-when-loading-linked-unit-files-use-l.patch:
    Use src name, not dst name, of symlinked unit files (LP: #1887744)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=667a881ab3bace6c4a7550067d4ecf2e4b9ece82

 -- Balint Reczey <email address hidden> Mon, 29 Mar 2021 12:22:06 +0200

Changed in systemd (Ubuntu Hirsute):
status: In Progress → Fix Released
Revision history for this message
Dan Streetman (ddstreet) wrote :
tags: removed: block-proposed-focal block-proposed-groovy
Revision history for this message
Dan Streetman (ddstreet) wrote :

removing block-proposed tags as the fix has been released in hirsute

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 246.6-1ubuntu1.3

---------------
systemd (246.6-1ubuntu1.3) groovy; urgency=medium

  * d/p/lp1918696-shared-seccomp-util-address-family-filtering-is-brok.patch:
    Stop attempting to restrict address families on ppc archs
    (LP: #1918696)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=34d1b2b4f4fdc349da37a07fe98418456167592a
  * d/p/lp1915887-Downgrade-a-couple-of-warnings-to-debug.patch:
    Downgrade some log messages so they stop spamming logs
    (LP: #1915887)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d092e70912009daf6f64182d86b3d92aed28722a
  * d/p/lp1887744-basic-unit-file-when-loading-linked-unit-files-use-l.patch:
    Use src name, not dst name, of symlinked unit files (LP: #1887744)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1e42633ccecc9c9157b19387c11ac1bf7ea7d161

 -- Dan Streetman <email address hidden> Wed, 10 Mar 2021 16:03:13 -0500

Changed in systemd (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 245.4-4ubuntu3.6

---------------
systemd (245.4-4ubuntu3.6) focal; urgency=medium

  * debian/patches/lp1916485-Newer-Glibc-use-faccessat2-to-implement-faccessat.patch:
    Add support for faccessat2 (LP: #1916485)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=affb2c6507dccfeed02820a2267639648e2a2260
  * d/p/lp1918696-shared-seccomp-util-address-family-filtering-is-brok.patch:
    Stop attempting to restrict address families on ppc archs
    (LP: #1918696)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=daff4b6604362fcb5d305682216d5ca15a4c5738
  * d/p/lp1891810-seccomp-util-add-new-syscalls-from-kernel-5.6-to-sys.patch:
    Add openat2() syscall to seccomp filter list
    (LP: #1891810)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=69c8a684e2513b2f6530e5a5cf15c83abfb7bc74
  * d/p/lp1915887-Downgrade-a-couple-of-warnings-to-debug.patch:
    Downgrade some log messages so they stop spamming logs
    (LP: #1915887)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3c2c4731b90ed430ca1790270e69cd125643b94b
  * d/p/lp1887744-basic-unit-file-when-loading-linked-unit-files-use-l.patch:
    Use src name, not dst name, of symlinked unit files (LP: #1887744)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=03770601097cfdc09adeadf5593083da69345409

 -- Dan Streetman <email address hidden> Wed, 17 Mar 2021 17:36:08 -0400

Changed in systemd (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.