Comment 0 for bug 1950787

systemd-sysusers.service/systemd.exec fails to start in privileged containers, due to being unable to properly mount /dev for passing credentials, caused by the following config in the .service unit:
# Optionally, pick up a root password and shell for the root user from a
# credential passed to the service manager. This is useful for importing this
# data from nspawn's --set-credential= switch.
LoadCredential=passwd.hashed-password.root
LoadCredential=passwd.plaintext-password.root
LoadCredential=passwd.shell.root

Reproducer:
$ lxc profile set default security.privileged "true"
$ lxc launch ubuntu-daily:jammy test
$ lxc exec test bash
# add-apt-repository ppa:ci-train-ppa-service/4704
# apt install systemd # install systemd 249.5-2ubuntu1
# systemctl restart systemd-sysusers
# systemctl status systemd-sysusers
# system --status=failed
$ lxc profile set default security.privileged "false"

A workaround is to disable it via:
$ cat /etc/systemd/system/systemd-sysusers.service.d/override.conf:
[Service]
LoadCredential=

Interesting logs:
Nov 12 12:09:44 test systemd[1]: systemd-journald.service: Added fd 42 (n/a) to fd store.
Nov 12 12:09:44 test systemd[431]: Mounting /dev (MS_REC|MS_SLAVE "")...
Nov 12 12:09:44 test systemd[431]: Failed to mount n/a (type n/a) on /dev (MS_REC|MS_SLAVE ""): Permission denied
Nov 12 12:09:44 test systemd[430]: (sd-mkdcreds) failed with exit status 1.
Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed to set up credentials: Protocol error
Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed at step CREDENTIALS spawning

Debug logs:
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Job 350 systemd-sysusers.service/restart finished, result=done
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Converting job systemd-sysusers.service/restart -> systemd-sysusers.service/start
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: ConditionNeedsUpdate=/etc succeeded.
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Passing 0 fds to service
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: About to execute systemd-sysusers
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Forked systemd-sysusers as 430
Nov 12 12:09:44 test systemd[430]: Successfully forked off '(sd-mkdcreds)' as PID 431.
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=7 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=8 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2893 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2894 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Changed failed -> start
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=9 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2895 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Starting Create System Users...
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=10 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=11 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2896 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2897 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/job/350 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=12 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/job/350 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2898 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: systemd-journald.service: Got notification message from PID 59 (FDSTORE=1)
Nov 12 12:09:44 test systemd[1]: systemd-journald.service: Added fd 42 (n/a) to fd store.
Nov 12 12:09:44 test systemd[431]: Mounting /dev (MS_REC|MS_SLAVE "")...
Nov 12 12:09:44 test systemd[431]: Failed to mount n/a (type n/a) on /dev (MS_REC|MS_SLAVE ""): Permission denied
Nov 12 12:09:44 test systemd[430]: (sd-mkdcreds) failed with exit status 1.
Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed to set up credentials: Protocol error
Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed at step CREDENTIALS spawning systemd-sysusers: Protocol error
Nov 12 12:09:44 test systemd[1]: Received SIGCHLD from PID 430 ((sysusers)).
Nov 12 12:09:44 test systemd[1]: Child 430 ((sysusers)) died (code=exited, status=243/CREDENTIALS)
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Child 430 belongs to systemd-sysusers.service.
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Main process exited, code=exited, status=243/CREDENTIALS
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Failed with result 'exit-code'.
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Service will not restart (restart setting)
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Changed start -> failed
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=13 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2899 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Job 350 systemd-sysusers.service/start finished, result=failed
Nov 12 12:09:44 test systemd[1]: Failed to start Create System Users.