systemd dumps core on starting service: Assertion 'u->instance' failed at src/core/load-fragment.c:4798

[Jeff Turner]

[impact]

if a non-template service file is created as a symlink to a template service file outside the normal unit file paths, systemd will fail assertion and crash

[test case]

su -
cat <<EOF > /crashsystemd@.service
[Unit]
Description=Template for instance %i

[Service]
ExecStart=/bin/true
EOF

cd /etc/systemd/system
ln -s /crashsystemd@.service crashsystemd.service
systemctl daemon-reload
systemctl start crashsystemd

[regression potential]

any regression would likely cause a failed assertion and crash, or failure to properly parse and/or load a service file that is a symlink

[scope]

this is needed in f/g/h

this is fixed upstream by commit 3aa57658434e7a95c6000bebb166c31f1c6d051b which is included in v248-rc1, so this is not fixed in any release yet

this was introduced by the changes in commit e8630e69523 which was first included in v243, so this bug doesn't exist in b or earlier

[original description]

On an Ubuntu 20.04 server, when I started a newly installed systemd service 'systemctl start testemail2jira.service', systemd trips on an assertion (from journalctl):

Jul 16 13:49:12 pangolin-jiraconf systemd[1]: Assertion 'u->instance' failed at src/core/load-fragment.c:4798, function unit_load_fragment(). Aborting.
Jul 16 13:49:13 pangolin-jiraconf systemd[1]: Caught <ABRT>, dumped core as pid 2906637.
Jul 16 13:49:13 pangolin-jiraconf systemd[1]: Freezing execution.

apport generates a core dump which I trust is attached.

The server still works, but my terminal keeps showing:

Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

and pauses for 25s after each command.

In case it's relevant, the <service> i was trying to start had just been symlinked into /etc/systemd/system/:

root@pangolin-jiraconf # ls -la /etc/systemd/system/testemail2jira.service
lrwxrwxrwx 1 root root 82 Jul 16 13:48 /etc/systemd/system/testemail2jira.service -> /opt/atlassian/jira/current/emailprocessing/testemail2jira/testemail2jira@.service

File contents:

[Unit]
Description=Check that email to %i JIRA works

[Service]
#Type=oneshot
# Note: I keep the config file, testemail2jira.json5 and rsa.* files in emailprocessing/, one directory below testemail2jira/, so that they can be version-controlled independently.
# If you want testemail2jira.json5 and rsa.* within testemail2jira/, then add testemail2jira/ to WorkingDirectory and remove it from the ExecStart command
WorkingDirectory=/opt/atlassian/%i/current/emailprocessing
ExecStart=/opt/atlassian/%i/current/emailprocessing/testemail2jira/venv/bin/python testemail2jira/testemail2jira.py
User=root

[Install]
WantedBy=multi-user.target

Perhaps the problem is that my service file is a template (testemail2jira@.service), but I symlinked it as a regular 'testemail2jira.service'.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: systemd 245.4-4ubuntu3.1
ProcVersionSignature: Ubuntu 5.4.0-39.43-generic 5.4.41
Uname: Linux 5.4.0-39-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu27.3
Architecture: amd64
CasperMD5CheckResult: skip
Date: Thu Jul 16 14:13:06 2020
InstallationDate: Installed on 2020-05-27 (49 days ago)
InstallationMedia: Ubuntu-Server 20.04 LTS "Focal Fossa" - Release amd64 (20200423)
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:

Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-39-generic root=/dev/sda ro console=ttyS0,19200n8 net.ifnames=0
SourcePackage: systemd
SystemdFailedUnits: Error: command ['systemctl', 'status', '--full', 'Error:'] failed with exit code 1: Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: rel-1.12.0-0-ga698c8995f-prebuilt.qemu.org
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-3.1
dmi.modalias: dmi:bvnSeaBIOS:bvrrel-1.12.0-0-ga698c8995f-prebuilt.qemu.org:bd04/01/2014:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-3.1:cvnQEMU:ct1:cvrpc-q35-3.1:
dmi.product.name: Standard PC (Q35 + ICH9, 2009)
dmi.product.version: pc-q35-3.1
dmi.sys.vendor: QEMU

[Dan Streetman (ddstreet)]

> apport generates a core dump which I trust is attached.

unfortunately i don't see it attached; could you manually attach it please?

[Jeff Turner (jefft7610)]

Attached is /var/crash/_usr_lib_systemd_systemd.0.crash

[Jeff Turner (jefft7610)]

Replicated on my local Ubuntu 20.04 system with systemd 245.4-4ubuntu3.1.

It's a PEBCAK error where a user writes a .service file containing %i's, but instead of naming the file <email address hidden>, just names it foo.service. To replicate:

cd /etc/systemd/system
curl -s -o testemail2jira.service 'https://hg.sr.ht/~redradishtech/testemail2jira/raw/testemail2jira%40.service?rev=tip'
systemctl daemon-reload
systemctl start testemail2jira

On one of the last two steps everything freezes, errors start appearing:

[system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

and /var/crash/_usr_lib_systemd_systemd.0.crash is created.

[Dan Streetman (ddstreet)]

> To replicate

I can't reproduce it with those steps, but maybe it's because i dont have the same content under /opt/... that you do; can you provide simpler repro steps that doesn't require anything installed under /opt/... ?

[Jeff Turner (jefft7610)]

Sorry, turns out the /etc/systemd/system service file needs to be a symlink to trigger the bug. This will do it:

su -
cat <<EOF > /crashsystemd@.service
[Unit]
Description=Template for instance %i

[Service]
ExecStart=/bin/true
EOF

cd /etc/systemd/system
ln -s /crashsystemd@.service crashsystemd.service
systemctl daemon-reload
systemctl start crashsystemd

After the 'systemctl start crashsystemd' my terminal says:

Failed to start crashsystemd.service: Connection reset by peer
See system logs and 'systemctl status crashsystemd.service' for details.

and in the systemd logs I see:

Jul 17 23:20:33 jturner-desktop systemd[1]: Assertion 'u->instance' failed at src/core/load-fragment.c:4798, function unit_load_fragment(). Aborting.

Things are pretty broken now. 'systemctl' and 'reboot' commands won't work. Don't forget to delete the /etc/systemd/system/crashsystemd.service symlink before rebooting.

[Dan Streetman (ddstreet)]

> Sorry, turns out the /etc/systemd/system service file needs to be a
> symlink to trigger the bug. This will do it:

aha! and not only a symlink, but the target of the symlink needs to be outside the normal systemd service path locations, or it will reject loading it:

root@lp1887744-f:/etc/systemd/system# ls -l test*
lrwxrwxrwx 1 root root 13 Jul 20 18:17 test.service -> test@.service
-rw-r--r-- 1 root root 76 Jul 20 18:14 test@.service

root@lp1887744-f:/etc/systemd/system# journalctl --no-pager -q -b --grep test.service
Jul 20 18:17:33 lp1887744-f systemd[1]: /etc/systemd/system/test.service: symlink target name type "test@.service" does not match source, rejecting.

[Łukasz Zemczak (sil2100)]

Ok, so this is still not something we have in hirsute? Do we know when 248 will be pulled in? Is it still something we plan for hirsute?

[Łukasz Zemczak (sil2100)]

I'll review this and consider it for inclusion as an SRU, but will not release them before it is available in hirsute (either via 248, or a cherry-pick on top of the current release).

[Łukasz Zemczak (sil2100)]

Hello Jeff, or anyone else affected,

Accepted systemd into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/246.6-1ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Jeff, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (246.6-1ubuntu1.3) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

prometheus/2.20.0+ds-1 (amd64)
asterisk/1:16.12.0~dfsg-1 (amd64)
apt/2.1.10ubuntu0.2 (arm64)
flatpak/1.8.2-1ubuntu0.1 (ppc64el)
udisks2/2.9.1-2ubuntu1 (amd64)
gvfs/1.46.1-1ubuntu1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.6) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

multipath-tools/0.8.3-1ubuntu2 (s390x)
munin/2.0.56-1ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

root@lp1887744-g:~# dpkg -l systemd|grep systemd
ii systemd 246.6-1ubuntu1.2 amd64 system and service manager
root@lp1887744-g:~# cat <<EOF > /crashsystemd@.service
> [Unit]
> Description=Template for instance %i
>
> [Service]
> ExecStart=/bin/true
> EOF
root@lp1887744-g:~#
root@lp1887744-g:~# cd /etc/systemd/system
root@lp1887744-g:/etc/systemd/system# ln -s /crashsystemd@.service crashsystemd.service
root@lp1887744-g:/etc/systemd/system# systemctl daemon-reload
root@lp1887744-g:/etc/systemd/system# systemctl start crashsystemd
Failed to start crashsystemd.service: Connection reset by peer
See system logs and 'systemctl status crashsystemd.service' for details.

root@lp1887744-g:~# dpkg -l systemd|grep systemd
ii systemd 246.6-1ubuntu1.3 amd64 system and service manager
root@lp1887744-g:~# cat <<EOF > /crashsystemd@.service
> [Unit]
> Description=Template for instance %i
>
> [Service]
> ExecStart=/bin/true
> EOF
root@lp1887744-g:~#
root@lp1887744-g:~# cd /etc/systemd/system
root@lp1887744-g:/etc/systemd/system# ln -s /crashsystemd@.service crashsystemd.service
root@lp1887744-g:/etc/systemd/system# systemctl daemon-reload
root@lp1887744-g:/etc/systemd/system# systemctl start crashsystemd
root@lp1887744-g:/etc/systemd/system#

[Dan Streetman (ddstreet)]

root@lp1887744-f:~# dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.5 amd64 system and service manager
root@lp1887744-f:~# cat <<EOF > /crashsystemd@.service
> [Unit]
> Description=Template for instance %i
>
> [Service]
> ExecStart=/bin/true
> EOF
root@lp1887744-f:~#
root@lp1887744-f:~# cd /etc/systemd/system
root@lp1887744-f:/etc/systemd/system# ln -s /crashsystemd@.service crashsystemd.service
root@lp1887744-f:/etc/systemd/system# systemctl daemon-reload
root@lp1887744-f:/etc/systemd/system# systemctl start crashsystemd
Failed to start crashsystemd.service: Connection reset by peer
See system logs and 'systemctl status crashsystemd.service' for details.

root@lp1887744-f:~# dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.6 amd64 system and service manager
root@lp1887744-f:~# cat <<EOF > /crashsystemd@.service
> [Unit]
> Description=Template for instance %i
>
> [Service]
> ExecStart=/bin/true
> EOF
root@lp1887744-f:~#
root@lp1887744-f:~# cd /etc/systemd/system
root@lp1887744-f:/etc/systemd/system# ln -s /crashsystemd@.service crashsystemd.service
root@lp1887744-f:/etc/systemd/system# systemctl daemon-reload
root@lp1887744-f:/etc/systemd/system# systemctl start crashsystemd
root@lp1887744-f:/etc/systemd/system#

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 247.3-3ubuntu2

---------------
systemd (247.3-3ubuntu2) hirsute; urgency=medium

  [ Dan Streetman ]
  * d/p/lp1887744-basic-unit-file-when-loading-linked-unit-files-use-l.patch:
    Use src name, not dst name, of symlinked unit files (LP: #1887744)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=667a881ab3bace6c4a7550067d4ecf2e4b9ece82

 -- Balint Reczey <email address hidden> Mon, 29 Mar 2021 12:22:06 +0200

[Dan Streetman (ddstreet)]

fixed in hirsute in version:
https://launchpad.net/ubuntu/+source/systemd/247.3-3ubuntu2

[Dan Streetman (ddstreet)]

removing block-proposed tags as the fix has been released in hirsute

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 246.6-1ubuntu1.3

---------------
systemd (246.6-1ubuntu1.3) groovy; urgency=medium

  * d/p/lp1918696-shared-seccomp-util-address-family-filtering-is-brok.patch:
    Stop attempting to restrict address families on ppc archs
    (LP: #1918696)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=34d1b2b4f4fdc349da37a07fe98418456167592a
  * d/p/lp1915887-Downgrade-a-couple-of-warnings-to-debug.patch:
    Downgrade some log messages so they stop spamming logs
    (LP: #1915887)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d092e70912009daf6f64182d86b3d92aed28722a
  * d/p/lp1887744-basic-unit-file-when-loading-linked-unit-files-use-l.patch:
    Use src name, not dst name, of symlinked unit files (LP: #1887744)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1e42633ccecc9c9157b19387c11ac1bf7ea7d161

 -- Dan Streetman <email address hidden> Wed, 10 Mar 2021 16:03:13 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 245.4-4ubuntu3.6

---------------
systemd (245.4-4ubuntu3.6) focal; urgency=medium

  * debian/patches/lp1916485-Newer-Glibc-use-faccessat2-to-implement-faccessat.patch:
    Add support for faccessat2 (LP: #1916485)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=affb2c6507dccfeed02820a2267639648e2a2260
  * d/p/lp1918696-shared-seccomp-util-address-family-filtering-is-brok.patch:
    Stop attempting to restrict address families on ppc archs
    (LP: #1918696)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=daff4b6604362fcb5d305682216d5ca15a4c5738
  * d/p/lp1891810-seccomp-util-add-new-syscalls-from-kernel-5.6-to-sys.patch:
    Add openat2() syscall to seccomp filter list
    (LP: #1891810)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=69c8a684e2513b2f6530e5a5cf15c83abfb7bc74
  * d/p/lp1915887-Downgrade-a-couple-of-warnings-to-debug.patch:
    Downgrade some log messages so they stop spamming logs
    (LP: #1915887)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3c2c4731b90ed430ca1790270e69cd125643b94b
  * d/p/lp1887744-basic-unit-file-when-loading-linked-unit-files-use-l.patch:
    Use src name, not dst name, of symlinked unit files (LP: #1887744)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=03770601097cfdc09adeadf5593083da69345409

 -- Dan Streetman <email address hidden> Wed, 17 Mar 2021 17:36:08 -0400