systemd user daemon fails with Permission denied when creating transient scope

[Maciej Borzecki]

Observed on 18.04. Systemd user instance fails when trying to create a transient scope when logged in through ssh as a regular user

Specifically this fails:
$ systemd-run --user --scope ls
Job for run-rc78f932ad730440490bd7bc17f9d5c8c.scope failed.
See "systemctl status run-rc78f932ad730440490bd7bc17f9d5c8c.scope" and "journalctl -xe" for details.

Inspecting journal shows:
Oct 05 10:38:16 ubuntu systemd[1437]: run-rc78f932ad730440490bd7bc17f9d5c8c.scope: Failed to add PIDs to scope's control group: Permission denied
Oct 05 10:38:16 ubuntu systemd[1437]: run-rc78f932ad730440490bd7bc17f9d5c8c.scope: Failed with result 'resources'.
Oct 05 10:38:16 ubuntu systemd[1437]: Failed to start /bin/ls.
Oct 05 10:38:16 ubuntu polkitd(authority=local)[1244]: Unregistered Authentication Agent for unix-process:7425:200857 (system bus name :1.106, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

Further strace shows that there is an EACCES when writing the PID of the forked process to cgroup procs:

1437 openat(AT_FDCWD, "/sys/fs/cgroup/pids/user.slice/user-999.slice/[e-mail]/run-r067b0361ac97410886bbb3eec1c3848d.scope/pids.max", O_WRONLY|O_NOCTTY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1437 newfstatat(AT_FDCWD, "/sys/fs/cgroup/unified", {st_dev=makedev(0, 32), st_ino=1, st_mode=S_IFDIR|0555, st_nlink=5, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0, st_atime=1633428300 /* 2021-10-05T10:05:00.336000000+0000 */, st_atime_nsec=336000000, st_mtime=1633428300 /* 2021-10-05T10:05:00.336000000+0000 */, st_mtime_nsec=336000000, st_ctime=1633428300 /* 2021-10-05T10:05:00.336000000+0000 */, st_ctime_nsec=336000000}, AT_SYMLINK_NOFOLLOW) = 0
1437 openat(AT_FDCWD, "/sys/fs/cgroup/unified/user.slice/user-999.slice/[e-mail]/run-r067b0361ac97410886bbb3eec1c3848d.scope/cgroup.procs", O_WRONLY|O_NOCTTY|O_CLOEXEC) = 34
1437 fcntl(34, F_GETFL) = 0x8001 (flags O_WRONLY|O_LARGEFILE)
1437 fstat(34, {st_dev=makedev(0, 32), st_ino=2358, st_mode=S_IFREG|0644, st_nlink=1, st_uid=999, st_gid=999, st_blksize=4096, st_blocks=0, st_size=0, st_atime=1633430486 /* 2021-10-05T10:41:26.701277147+0000 */, st_atime_nsec=701277147, st_mtime=1633430486 /* 2021-10-05T10:41:26.701277147+0000 */, st_mtime_nsec=701277147, st_ctime=1633430486 /* 2021-10-05T10:41:26.701277147+0000 */, st_ctime_nsec=701277147}) = 0
1437 write(34, "7461\n", 5) = -1 EACCES (Permission denied)
1437 close(34) = 0

Full strace of the failed attempt: https://paste.ubuntu.com/p/4vwtYQ7mww/

When executing the same command from a gnome terminal, the scope is created successfuly. Full trace of successful execution: https://paste.ubuntu.com/p/XjJ8mfxSXn/

The relevant bit from the happy execution path:

openat(AT_FDCWD, "/sys/fs/cgroup/pids/user.slice/user-999.slice/[e-mail]/run-rd9ebe0f0326b482e82ca374c5ae613cd.scope/pids.max", O_WRONLY|O_NOCTTY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/sys/fs/cgroup/unified", {st_dev=makedev(0, 32), st_ino=1, st_mode=S_IFDIR|0555, st_nlink=5, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0, st_atime=1633428300 /* 2021-10-05T10:05:00.336000000+0000 */, st_atime_nsec=336000000, st_mtime=1633428300 /* 2021-10-05T10:05:00.336000000+0000 */, st_mtime_nsec=336000000, st_ctime=1633428300 /* 2021-10-05T10:05:00.336000000+0000 */, st_ctime_nsec=336000000}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/sys/fs/cgroup/unified/user.slice/user-999.slice/[e-mail]/run-rd9ebe0f0326b482e82ca374c5ae613cd.scope/cgroup.procs", O_WRONLY|O_NOCTTY|O_CLOEXEC) = 34
fcntl(34, F_GETFL) = 0x8001 (flags O_WRONLY|O_LARGEFILE)
fstat(34, {st_dev=makedev(0, 32), st_ino=2298, st_mode=S_IFREG|0644, st_nlink=1, st_uid=999, st_gid=999, st_blksize=4096, st_blocks=0, st_size=0, st_atime=1633429609 /* 2021-10-05T10:26:49.619626843+0000 */, st_atime_nsec=619626843, st_mtime=1633429609 /* 2021-10-05T10:26:49.619626843+0000 */, st_mtime_nsec=619626843, st_ctime=1633429609 /* 2021-10-05T10:26:49.619626843+0000 */, st_ctime_nsec=619626843}) = 0
write(34, "7410\n", 5) = 5
close(34) = 0

23838 write(31, "24075\n", 6) = -1 EACCES (Permission denied)

$ lsb_release -rd
Description: Ubuntu 18.04.6 LTS
Release: 18.04

$ dpkg -l systemd\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-========================================================================================
ii systemd 237-3ubuntu10.52 amd64 system and service manager
un systemd-container <none> <none> (no description available)
un systemd-shim <none> <none> (no description available)
ii systemd-sysv 237-3ubuntu10.52 amd64 system and service manager - SysV links

[Maciej Borzecki (maciek-borzecki)]

Possibly related: https://github.com/systemd/systemd/issues/3388

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.