sshd not starting because of missing /run/sshd

[Christopher Beland]

I recently upgraded from Ubuntu to 22.04 to 23.04 using do-release-upgrade (twice because I upgraded via 22.10). After doing so, sshd stopped working. The output of "journalctl -xeu ssh.service" includes the line:

Jun 06 01:51:05 big-bucks sshd[4544]: Missing privilege separation directory: /run/sshd

This fixed the problem: sudo mkdir /run/sshd

It seems like either the upgrade sequence or the service startup sequence should have done this automatically.

Currently running: ssh/lunar 1:9.0p1-1ubuntu8 all

There was also a problem reported with /run/sshd on bug #1991283, but I guess the fix for that didn't prevent this.

ProblemType: Bug
DistroRelease: Ubuntu 23.04
Package: ssh (not installed)
Uname: Linux 6.2.6-76060206-generic x86_64
ApportVersion: 2.26.1-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Tue Jun 6 02:37:21 2023
SourcePackage: openssh
UpgradeStatus: Upgraded to lunar on 2023-06-04 (1 days ago)

[Steve Langasek (vorlon)]

Did this happen before or after you rebooted into Ubuntu 23.04?

If before the reboot, /run/sshd should already have been present on the system, created by the previous version of /lib/systemd/system/ssh.service.

If after the reboot, it should be created by /usr/lib/tmpfiles.d/openssh-server.conf, via /lib/systemd/system/systemd-tmpfiles-setup.service.

/lib/systemd/system/systemd-tmpfiles-setup.service is a dependency of sysinit.target, so this directory should be created on boot before the ssh socket connection starts up.

So I don't see why this directory would be missing for you.

[Christopher Beland (cbeland)]

I don't know whether sshd was running immediately after the install; I noticed this problem a couple reboots after the install process was complete. Are there any logs I can check that would be helpful to reconstruct the sequence of events?

[Steve Langasek (vorlon)]

If you saw this problem after reboots, then the question is why tmpfiles handling is not working as expected.

I don't know if these jobs log anything relevant, but a place to start is:

 - output of systemctl status systemd-tmpfiles-setup
 - output of journalctl -b 0 --no-pager -lu systemd-tmpfiles-setup

[Steve Langasek (vorlon)]

Opened a bug task on systemd as well.

[Christopher Beland (cbeland)]

systemctl status systemd-tmpfiles-setup says:

 systemd-tmpfiles-setup.service - Create Volatile Files and Directories
     Loaded: loaded (/lib/systemd/system/systemd-tmpfiles-setup.service; static)
     Active: active (exited) since Tue 2023-06-06 01:46:37 EDT; 23h ago
       Docs: man:tmpfiles.d(5)
             man:systemd-tmpfiles(8)
   Main PID: 536 (code=exited, status=73)
        CPU: 62ms

Jun 06 01:46:37 big-bucks systemd-tmpfiles[536]: Detected unsafe path transition / (owned by beland) → /var (owned by root) during canonicalization of /var/log/journal.
Jun 06 01:46:37 big-bucks systemd-tmpfiles[536]: Detected unsafe path transition / (owned by beland) → /var (owned by root) during canonicalization of /var/log/journal.
Jun 06 01:46:37 big-bucks systemd-tmpfiles[536]: Detected unsafe path transition / (owned by beland) → /var (owned by root) during canonicalization of /var/log/journal/0bbe1b93d943a0a5>
Jun 06 01:46:37 big-bucks systemd-tmpfiles[536]: Detected unsafe path transition / (owned by beland) → /var (owned by root) during canonicalization of /var/log/journal/0bbe1b93d943a0a5>
Jun 06 01:46:37 big-bucks systemd-tmpfiles[536]: Detected unsafe path transition / (owned by beland) → /var (owned by root) during canonicalization of /var/log/journal/0bbe1b93d943a0a5>
Jun 06 01:46:37 big-bucks systemd-tmpfiles[536]: Detected unsafe path transition / (owned by beland) → /run (owned by root) during canonicalization of /run/log/journal.
Jun 06 01:46:37 big-bucks systemd-tmpfiles[536]: Detected unsafe path transition / (owned by beland) → /run (owned by root) during canonicalization of /run/log/journal.
Jun 06 01:46:37 big-bucks systemd-tmpfiles[536]: Detected unsafe path transition / (owned by beland) → /var (owned by root) during canonicalization of /var/log/journal/0bbe1b93d943a0a5>
Jun 06 01:46:37 big-bucks systemd-tmpfiles[536]: Detected unsafe path transition / (owned by beland) → /var (owned by root) during canonicalization of /var/log/journal/0bbe1b93d943a0a5>
Jun 06 01:46:37 big-bucks systemd[1]: Finished systemd-tmpfiles-setup.service - Create Volatile Files and Directories.

[Steve Langasek (vorlon)]

> Jun 06 01:46:37 big-bucks systemd-tmpfiles[536]: Detected unsafe path transition / (owned by beland) → /var (owned by root) during canonicalization of /var.

Well, that's fairly clear - you have insecure permissions on your root filesystem and systemd-tmpfiles is failing to act. You'll want to 'chown root:root /' to fix this; afterwards everything should work normally.

[Christopher Beland (cbeland)]

My naughty chickens have come home to roost! Thanks for the quick diagnosis.