systemd loops trying to start systemd-ask-password-plymouth

[Walter]

Short story:
- after boot
- when a systemd service wants a password
- systemd-ask-password is invoked
- systemd-ask-password-plymouth.path is reached
- systemd-ask-password-plymouth.service is started

Except:
- the conditions for the latter are (apparently) not met

End result:
- 40.000x "Condition check resulted in Forward Password Requests to Plymouth being skipped."
- 1.5 minutes of 100% cpu usage

----------------
AFFECTED VERSION
----------------

systemd 249.11-0ubuntu3.1 on Ubuntu/Jammy 22.04

----------------
HOW TO REPRODUCE
----------------

If I leave the `systemd-ask-password-plymouth.path` unmasked/enabled and reboot. OpenVPN (calling systemd-ask-password) will trigger the condition.

This causes high CPU usage for 1.5 minutes.

It looks like /run/plymouth/pid exists (for the .path file) at first, when that is invoked, but then the resultant target (the .service file) checks again, and finds that it is gone.

Manually reproducing:

# ls /run/systemd/ask-password
ask.5hW6rb sck.79cfe1203518610

# mkdir /run/plymouth/pid
# systemctl start systemd-ask-password-plymouth.service systemd-ask-password-plymouth.path

# systemctl show --value --property=MainPID systemd-ask-password-plymouth.service
24777

# rmdir /run/plymouth/pid ; kill 24777

Result: systemd going into a loop.

Stop the loop with:

# systemctl stop systemd-ask-password-plymouth.path

--------
ANALYSIS
--------

It looks like this:
https://github.com/systemd/systemd/issues/21025

which is fixed by:
https://github.com/systemd/systemd/pull/21030

Alternative bug reports:
https://bugzilla.redhat.com/show_bug.cgi?id=1919538

Systemd's own analysis of the situation:

# systemd-analyze critical-chain systemd-ask-password-plymouth.service
...
systemd-ask-password-plymouth.service @1min 35.823s
└─systemd-ask-password-plymouth.path @593ms
  └─plymouth-start.service @571ms +21ms
    └─systemd-udevd.service @450ms +119ms
      └─systemd-tmpfiles-setup-dev.service @429ms +18ms
        └─systemd-sysusers.service @382ms +46ms
          └─systemd-remount-fs.service @353ms +22ms
            └─systemd-journald.socket @318ms
              └─system.slice @264ms
                └─-.slice @264ms

-----------
WORKAROUNDS
-----------

This works as long as you don't need the plymouth-ask-password:

# systemctl disable systemd-ask-password-plymouth.path
# systemctl mask systemd-ask-password-plymouth.path

Could you get the relevant patches from upstream sorted in Jammy?

Thanks!

Walter Doekes
OSSO B.V.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Nick Rosbrook (enr0n)]

I believe the commit you referenced has been in jammy since 249.9-0ubuntu1.

Can you please attach some journalctl output from a boot that demonstrates this issue?

[Walter (wdoekes)]

Funny you should ask. I tempted to ask where you think I got the output from in the first place. But I shall refrain from sarcasm as that won't do any good.

This does put me in a bit of a pickle. Because in order for the journalctl output to be more useful than just the 40.000 message I referenced, I should leave it untouched. But I cannot give it to you unsanitized either.

I'm attaching the log uniq -c'd, starting just before the issue at hand, so I could inspect it for private details. Observe that the `1501` on the left means 1501x the same message in lines like:

   1501 jun 04 15:01:55 myhostname systemd[1]: Condition check resulted in Forward Password Requests to Plymouth being skipped.

I hope this is convincing enough that I'm not making this up.

As for the:

> I believe the commit you referenced has been in jammy since 249.9-0ubuntu1.

Okay. Well. If I apt-get source systemd, I would assume that I'd get that as a patch file. But grepping for "unit_has_failed_condition_or_assert" in ./debian/patches yields nothing. Grepping for -i "condition": no relevant hits.

Are you sure this patch is included? If so, then only parts if it maybe?

Regards,
Walter

[Nick Rosbrook (enr0n)]

I think I misunderstood the first time I looked at the upstream issue. We have the commit that reverts the regressing commit, but not the follow-on patch (which was not backported to the v250
-stable branch).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Raphaël Droz (raphael-droz)]

Definitely needs a backport (to 22.04)

journalctl -g 'Forward Password Requests to Plymouth being skipped'|wc -l
1342502

(Was for a systemd-cryptsetup@*.service for which systemd-ask-password behavior seems wrong, but whatever the actual cause is, this loop is a very dangerous disk-intensive free-space eater)