| 2021-06-30 13:37:16 |
Nicolas Bock |
bug |
|
|
added bug |
| 2021-06-30 13:37:33 |
Nicolas Bock |
systemd (Ubuntu): assignee |
|
Dan Streetman (ddstreet) |
|
| 2021-06-30 13:56:49 |
Nicolas Bock |
description |
On a system that is monitored via telegraf I found many abandoned systemd session which I believe are created by a potential race where systemd is reloading unit files and at the same time a user is connecting to the system via ssh or is executing the su command.
The simple reproducer
for i in {1..100}; do ssh localhost sudo systemctl daemon-reload & ssh localhost sleep 1 & done
jobs -p | xargs --verbose --no-run-if-empty kill -KILL
systemctl status --all 2> /dev/null | grep --before-context 3 abandoned
will produce something similar to
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
--
● session-273.scope - Session 273 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-273.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-274.scope - Session 274 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-274.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-30.scope - Session 30 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-30.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 10:05:56 UTC; 3h 30min ago
--
● session-302.scope - Session 302 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-302.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:04 UTC; 4min 6s ago
--
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
The system in question is running Bionic, systemd-237-3ubuntu10.48 |
On a system that is monitored via telegraf I found many abandoned systemd session which I believe are created by a potential race where systemd is reloading unit files and at the same time a user is connecting to the system via ssh or is executing the su command.
The simple reproducer
$ for i in {1..100}; do sleep 0.2; ssh localhost sudo systemctl daemon-reload & ssh localhost sleep 1 & done
$ systemctl status --all 2> /dev/null | grep --before-context 3 abandoned
will produce something similar to
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
--
● session-273.scope - Session 273 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-273.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-274.scope - Session 274 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-274.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-30.scope - Session 30 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-30.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 10:05:56 UTC; 3h 30min ago
--
● session-302.scope - Session 302 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-302.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:04 UTC; 4min 6s ago
--
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
The system in question is running Bionic, systemd-237-3ubuntu10.48 |
|
| 2021-06-30 13:59:01 |
Nicolas Bock |
description |
On a system that is monitored via telegraf I found many abandoned systemd session which I believe are created by a potential race where systemd is reloading unit files and at the same time a user is connecting to the system via ssh or is executing the su command.
The simple reproducer
$ for i in {1..100}; do sleep 0.2; ssh localhost sudo systemctl daemon-reload & ssh localhost sleep 1 & done
$ systemctl status --all 2> /dev/null | grep --before-context 3 abandoned
will produce something similar to
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
--
● session-273.scope - Session 273 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-273.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-274.scope - Session 274 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-274.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-30.scope - Session 30 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-30.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 10:05:56 UTC; 3h 30min ago
--
● session-302.scope - Session 302 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-302.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:04 UTC; 4min 6s ago
--
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
The system in question is running Bionic, systemd-237-3ubuntu10.48 |
On a system that is monitored via telegraf I found many abandoned systemd session which I believe are created by a potential race where systemd is reloading unit files and at the same time a user is connecting to the system via ssh or is executing the su command.
The simple reproducer
$ for i in {1..100}; do sleep 0.2; ssh localhost sudo systemctl daemon-reload & ssh localhost sleep 1 & done
Wait > 1 second
$ jobs -p | xargs --verbose --no-run-if-empty kill -KILL
To clean out STOPPED jobs and
$ systemctl status --all 2> /dev/null | grep --before-context 3 abandoned
will produce something similar to
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
--
● session-273.scope - Session 273 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-273.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-274.scope - Session 274 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-274.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-30.scope - Session 30 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-30.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 10:05:56 UTC; 3h 30min ago
--
● session-302.scope - Session 302 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-302.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:04 UTC; 4min 6s ago
--
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
The system in question is running Bionic, systemd-237-3ubuntu10.48 |
|
| 2021-06-30 14:51:19 |
Cinzia Innocenzi |
bug |
|
|
added subscriber Cinzia Innocenzi |
| 2021-07-06 09:01:24 |
Launchpad Janitor |
systemd (Ubuntu): status |
New |
Confirmed |
|
| 2021-07-06 09:01:26 |
Trent Lloyd |
bug |
|
|
added subscriber Trent Lloyd |
| 2021-07-06 18:42:41 |
Dan Streetman |
nominated for series |
|
Ubuntu Hirsute |
|
| 2021-07-06 18:42:41 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Hirsute) |
|
| 2021-07-06 18:42:41 |
Dan Streetman |
nominated for series |
|
Ubuntu Focal |
|
| 2021-07-06 18:42:41 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Focal) |
|
| 2021-07-06 18:42:41 |
Dan Streetman |
nominated for series |
|
Ubuntu Impish |
|
| 2021-07-06 18:42:41 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Impish) |
|
| 2021-07-06 18:42:41 |
Dan Streetman |
nominated for series |
|
Ubuntu Bionic |
|
| 2021-07-06 18:42:41 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Bionic) |
|
| 2021-07-06 18:42:41 |
Dan Streetman |
nominated for series |
|
Ubuntu Groovy |
|
| 2021-07-06 18:42:41 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Groovy) |
|
| 2021-07-06 18:42:52 |
Dan Streetman |
systemd (Ubuntu Hirsute): assignee |
|
Dan Streetman (ddstreet) |
|
| 2021-07-06 18:42:53 |
Dan Streetman |
systemd (Ubuntu Groovy): assignee |
|
Dan Streetman (ddstreet) |
|
| 2021-07-06 18:42:55 |
Dan Streetman |
systemd (Ubuntu Focal): assignee |
|
Dan Streetman (ddstreet) |
|
| 2021-07-06 18:42:56 |
Dan Streetman |
systemd (Ubuntu Bionic): assignee |
|
Dan Streetman (ddstreet) |
|
| 2021-07-06 18:43:01 |
Dan Streetman |
systemd (Ubuntu Impish): importance |
Undecided |
Medium |
|
| 2021-07-06 18:43:02 |
Dan Streetman |
systemd (Ubuntu Hirsute): importance |
Undecided |
Medium |
|
| 2021-07-06 18:43:04 |
Dan Streetman |
systemd (Ubuntu Groovy): importance |
Undecided |
Medium |
|
| 2021-07-06 18:43:05 |
Dan Streetman |
systemd (Ubuntu Focal): importance |
Undecided |
Medium |
|
| 2021-07-06 18:43:06 |
Dan Streetman |
systemd (Ubuntu Bionic): importance |
Undecided |
Medium |
|
| 2021-07-11 22:07:44 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2021-07-12 01:13:20 |
Dan Streetman |
bug watch added |
|
https://github.com/systemd/systemd/issues/20198 |
|
| 2021-07-12 01:13:20 |
Dan Streetman |
bug task added |
|
systemd |
|
| 2021-07-12 01:18:58 |
Dan Streetman |
description |
On a system that is monitored via telegraf I found many abandoned systemd session which I believe are created by a potential race where systemd is reloading unit files and at the same time a user is connecting to the system via ssh or is executing the su command.
The simple reproducer
$ for i in {1..100}; do sleep 0.2; ssh localhost sudo systemctl daemon-reload & ssh localhost sleep 1 & done
Wait > 1 second
$ jobs -p | xargs --verbose --no-run-if-empty kill -KILL
To clean out STOPPED jobs and
$ systemctl status --all 2> /dev/null | grep --before-context 3 abandoned
will produce something similar to
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
--
● session-273.scope - Session 273 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-273.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-274.scope - Session 274 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-274.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-30.scope - Session 30 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-30.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 10:05:56 UTC; 3h 30min ago
--
● session-302.scope - Session 302 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-302.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:04 UTC; 4min 6s ago
--
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
The system in question is running Bionic, systemd-237-3ubuntu10.48 |
[impact]
systemd may leak sessions, leaving empty cgroups around as well as abandoned session scopes.
[test case]
on a system where the user has a ssh key that allows noninteractive login to localhost, and also has noninteractive sudo, run:
$ for i in {1..100}; do sudo -b -i -u ubuntu ssh localhost -- sleep 1; done; for i in {1..20}; do echo 'Reloading...'; sudo systemctl daemon-reload; done
check the sessions to see there have been leaked sessions:
$ loginctl list-sessions
SESSION UID USER SEAT TTY
1 1000 ubuntu ttyS0
350 1000 ubuntu
351 1000 ubuntu
360 1000 ubuntu
...
to verify the sessions were leaked, clear them out with:
$ echo '' | sudo tee /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-*.scope/cgroup.events
that should result in all the leaked sessions being cleaned up.
[regression potential]
issues during systemd pid1 reexec/reload, or issues while cleaning up sessions, including leaking sessions/cgroups
[scope]
this is needed for all releases
upstream bug linked above, and upstream PR:
https://github.com/systemd/systemd/pull/20199
[original description]
On a system that is monitored via telegraf I found many abandoned systemd session which I believe are created by a potential race where systemd is reloading unit files and at the same time a user is connecting to the system via ssh or is executing the su command.
The simple reproducer
$ for i in {1..100}; do sleep 0.2; ssh localhost sudo systemctl daemon-reload & ssh localhost sleep 1 & done
Wait > 1 second
$ jobs -p | xargs --verbose --no-run-if-empty kill -KILL
To clean out STOPPED jobs and
$ systemctl status --all 2> /dev/null | grep --before-context 3 abandoned
will produce something similar to
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
--
● session-273.scope - Session 273 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-273.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-274.scope - Session 274 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-274.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-30.scope - Session 30 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-30.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 10:05:56 UTC; 3h 30min ago
--
● session-302.scope - Session 302 of user ubuntu
Loaded: loaded (/run/systemd/transient/session-302.scope; transient)
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:04 UTC; 4min 6s ago
--
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
The system in question is running Bionic, systemd-237-3ubuntu10.48 |
|
| 2021-07-12 01:19:12 |
Dan Streetman |
systemd (Ubuntu Impish): status |
Confirmed |
In Progress |
|
| 2021-07-12 01:19:14 |
Dan Streetman |
systemd (Ubuntu Hirsute): status |
New |
In Progress |
|
| 2021-07-12 01:19:16 |
Dan Streetman |
systemd (Ubuntu Groovy): status |
New |
In Progress |
|
| 2021-07-12 01:19:17 |
Dan Streetman |
systemd (Ubuntu Focal): status |
New |
In Progress |
|
| 2021-07-12 07:38:39 |
Bug Watch Updater |
systemd: status |
Unknown |
New |
|
| 2021-07-13 08:24:08 |
Brett Milford |
bug |
|
|
added subscriber Brett Milford |
| 2021-07-23 11:40:43 |
Launchpad Janitor |
systemd (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2021-07-23 11:41:29 |
Ian Johnson |
bug task added |
|
snapd |
|
| 2021-07-23 16:45:06 |
Chris Johnston |
bug |
|
|
added subscriber Chris Johnston |
| 2021-08-13 17:16:59 |
Dan Streetman |
systemd (Ubuntu Impish): assignee |
Dan Streetman (ddstreet) |
Heather Lemon (hypothetical-lemon) |
|
| 2021-08-13 17:17:07 |
Dan Streetman |
systemd (Ubuntu Hirsute): assignee |
Dan Streetman (ddstreet) |
Heather Lemon (hypothetical-lemon) |
|
| 2021-08-13 17:17:20 |
Dan Streetman |
systemd (Ubuntu Groovy): assignee |
Dan Streetman (ddstreet) |
|
|
| 2021-08-13 17:17:27 |
Dan Streetman |
systemd (Ubuntu Focal): assignee |
Dan Streetman (ddstreet) |
Heather Lemon (hypothetical-lemon) |
|
| 2021-08-13 17:17:35 |
Dan Streetman |
systemd (Ubuntu Bionic): assignee |
Dan Streetman (ddstreet) |
Heather Lemon (hypothetical-lemon) |
|
| 2021-08-13 17:17:42 |
Dan Streetman |
systemd (Ubuntu Groovy): status |
In Progress |
Won't Fix |
|
| 2021-08-23 14:58:04 |
Heather Lemon |
merge proposal linked |
|
https://code.launchpad.net/~hypothetical-lemon/ubuntu/+source/systemd/+git/systemd/+merge/407551 |
|
| 2021-08-24 14:52:38 |
Heather Lemon |
systemd (Ubuntu Bionic): assignee |
Heather Lemon (hypothetical-lemon) |
Dan Streetman (ddstreet) |
|
| 2021-08-24 14:52:47 |
Heather Lemon |
systemd (Ubuntu Focal): assignee |
Heather Lemon (hypothetical-lemon) |
Dan Streetman (ddstreet) |
|
| 2021-08-24 14:52:54 |
Heather Lemon |
systemd (Ubuntu Hirsute): assignee |
Heather Lemon (hypothetical-lemon) |
Dan Streetman (ddstreet) |
|
| 2021-08-24 14:53:03 |
Heather Lemon |
systemd (Ubuntu Impish): assignee |
Heather Lemon (hypothetical-lemon) |
Dan Streetman (ddstreet) |
|
| 2021-08-25 13:09:39 |
Dan Streetman |
systemd (Ubuntu Impish): status |
In Progress |
Fix Committed |
|
| 2021-08-25 14:17:49 |
Dan Streetman |
systemd (Ubuntu Bionic): status |
Confirmed |
In Progress |
|
| 2021-09-01 01:34:51 |
Chris Halse Rogers |
systemd (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2021-09-01 01:34:56 |
Chris Halse Rogers |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-09-01 01:34:58 |
Chris Halse Rogers |
bug |
|
|
added subscriber SRU Verification |
| 2021-09-01 01:35:06 |
Chris Halse Rogers |
tags |
|
verification-needed verification-needed-bionic |
|
| 2021-09-01 05:41:54 |
Chris Halse Rogers |
systemd (Ubuntu Hirsute): status |
In Progress |
Fix Committed |
|
| 2021-09-01 05:42:05 |
Chris Halse Rogers |
tags |
verification-needed verification-needed-bionic |
verification-needed verification-needed-bionic verification-needed-hirsute |
|
| 2021-09-02 12:48:05 |
Łukasz Zemczak |
systemd (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2021-09-02 12:48:13 |
Łukasz Zemczak |
tags |
verification-needed verification-needed-bionic verification-needed-hirsute |
verification-needed verification-needed-bionic verification-needed-focal verification-needed-hirsute |
|
| 2021-09-05 21:30:08 |
Dan Streetman |
tags |
verification-needed verification-needed-bionic verification-needed-focal verification-needed-hirsute |
verification-done verification-done-bionic verification-done-focal verification-done-hirsute |
|
| 2021-09-07 20:57:07 |
Brian Murray |
tags |
verification-done verification-done-bionic verification-done-focal verification-done-hirsute |
verification-done-bionic verification-done-focal verification-needed verification-needed-hirsute |
|
| 2021-09-07 21:06:36 |
Brian Murray |
tags |
verification-done-bionic verification-done-focal verification-needed verification-needed-hirsute |
verification-done-bionic verification-needed verification-needed-focal verification-needed-hirsute |
|
| 2021-09-09 13:10:10 |
Dan Streetman |
tags |
verification-done-bionic verification-needed verification-needed-focal verification-needed-hirsute |
verification-done verification-done-bionic verification-done-focal verification-done-hirsute |
|
| 2021-09-15 03:16:12 |
Chris Halse Rogers |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2021-09-15 03:23:41 |
Launchpad Janitor |
systemd (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2021-09-15 03:24:04 |
Launchpad Janitor |
systemd (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-09-15 03:24:29 |
Launchpad Janitor |
systemd (Ubuntu Hirsute): status |
Fix Committed |
Fix Released |
|
| 2021-09-30 12:46:22 |
Maciej Borzecki |
snapd: status |
New |
Invalid |
|
| 2022-01-12 18:38:44 |
Dan Streetman |
systemd (Ubuntu Impish): status |
Fix Committed |
Fix Released |
|
| 2022-01-12 18:38:48 |
Dan Streetman |
systemd (Ubuntu): status |
Fix Committed |
Fix Released |
|
