systemd-logind leaves leftover sessions and scope files
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
dbus (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Xenial |
Fix Released
|
Medium
|
Heitor Alves de Siqueira | ||
systemd (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Xenial |
Fix Released
|
Medium
|
Heitor Alves de Siqueira |
Bug Description
[Impact]
Scope file leakage can cause SSH delays and reduce performance in systemd
[Description]
The current systemd-logind version present in Xenial can leave abandoned SSH
sessions and scope files in cases where the host sees a lot of concurrent SSH
connections. These leftover sessions can slow down systemd performance
greatly, and can have an impact on sshd handling a great number of concurrent
connections.
To fix this issue, patches are needed in both dbus and systemd. These improve the
performance of the communication between dbus and systemd, so that they can
handle a better volume of events (e.g. SSH logins). All of those patches are
already present from Bionic onwards, so we only need those fixes for Xenial.
== Systemd ==
Upstream patches:
- core: use an AF_UNIX/SOCK_DGRAM socket for cgroup agent notification (d8fdc62037b5)
$ git describe --contains d8fdc62037b5
v230~71^2~2
$ rmadison systemd
systemd | 229-4ubuntu4 | xenial | source, ...
systemd | 229-4ubuntu21.21 | xenial-security | source, ...
systemd | 229-4ubuntu21.22 | xenial-updates | source, ... <--------
systemd | 237-3ubuntu10 | bionic | source, ...
systemd | 237-3ubuntu10.29 | bionic-security | source, ...
systemd | 237-3ubuntu10.29 | bionic-updates | source, ...
systemd | 237-3ubuntu10.31 | bionic-proposed | source, ...
== DBus ==
Upstream patches:
- Only read one message at a time if there are fds pending (892f084eeda0)
- bus: Fix timeout restarts (529600397bca)
- DBusMainLoop: ensure all required timeouts are restarted (446b0d9ac75a)
$ git describe --contains 892f084eeda0 529600397bca 446b0d9ac75a
dbus-1.11.10~44
dbus-1.11.10~45
dbus-1.11.16~2
$ rmadison dbus
dbus | 1.10.6-1ubuntu3 | xenial | source, ...
dbus | 1.10.6-1ubuntu3.4 | xenial-security | source, ...
dbus | 1.10.6-1ubuntu3.4 | xenial-updates | source, ... <--------
dbus | 1.12.2-1ubuntu1 | bionic | source, ...
dbus | 1.12.2-1ubuntu1.1 | bionic-security | source, ...
dbus | 1.12.2-1ubuntu1.1 | bionic-updates | source, ...
[Test Case]
1) Simulate a lot of concurrent SSH connections with e.g. a for loop:
multipass@
2) Check for leaked sessions in /run/systemd/
multipass@
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/
...
[Regression Potential]
As the patches change the communication socket between dbus and systemd, possible regressions could cause systemd to not be notified of dbus events and vice-versa. We could see units not getting started properly, and communication between different services break down (e.g. between systemd-logind and other processes).
In this case, the regression potential should be low as these patches have seen extensive testing both upstream and in more recent releases of Ubuntu. Nonetheless, these new packages will be rigorously tested through autopkgtest to avoid any possible Xenial-specific regressions.
Changed in dbus (Ubuntu): | |
status: | New → Fix Released |
Changed in systemd (Ubuntu): | |
status: | New → Fix Released |
Changed in dbus (Ubuntu Xenial): | |
assignee: | nobody → Heitor Alves de Siqueira (halves) |
Changed in systemd (Ubuntu Xenial): | |
assignee: | nobody → Heitor Alves de Siqueira (halves) |
description: | updated |
Changed in dbus (Ubuntu Xenial): | |
status: | New → In Progress |
Changed in systemd (Ubuntu Xenial): | |
status: | New → In Progress |
description: | updated |
description: | updated |
Changed in systemd (Ubuntu Xenial): | |
importance: | Undecided → Medium |
Changed in systemd (Ubuntu): | |
importance: | Undecided → Medium |
Changed in dbus (Ubuntu): | |
importance: | Undecided → Medium |
Changed in dbus (Ubuntu Xenial): | |
importance: | Undecided → Medium |
tags: | added: ddstreet sts-sponsor-ddstreet systemd xenial |
Changed in systemd (Ubuntu Xenial): | |
status: | Incomplete → In Progress |
tags: | removed: ddstreet sts-sponsor-ddstreet |
The patches are on the medium-to-large size, but I have reviewed them and as far as I can tell they appear correct. The systemd patch is needed to fix the cgroup-agent from overrunning the dbus socket connection queue, and the dbus patches are needed to prevent a highly loaded dbus message queue from timing out a long queue of incoming messages.