logind session files fill up /run space

[Daniel GALAMBOS]

Under Ubuntu 14.10 (server), systemd fills up /run/systemd/sessions directory with files.
The files are 240-260 bytes, but there are many
 As per default install, /run is on tmpfs with 50 MB size. After a couple of days, /run/ runs out of free space, which can cause not so funny errors, like failed kernel update (kernel postinst tries to create files here, then fails)

As I see, thesefiles are releated to ssh sesssions, and yes there are plenty SSH connection to this host. Shouldn't systemd remove these old session files?

# lsb_release -rd
Description: Ubuntu 14.10
Release: 14.10

# apt-cache policy systemd
systemd:
  Installed: 208-8ubuntu8.2
  Candidate: 208-8ubuntu8.2
  Version table:
 *** 208-8ubuntu8.2 0
        500 http://hu.archive.ubuntu.com/ubuntu/ utopic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     208-8ubuntu8 0
        500 http://hu.archive.ubuntu.com/ubuntu/ utopic/main amd64 Packages

[Martin Pitt (pitti)]

It should remove them, yes. I wonder if all these sessions are still open, i. e. are there any leftover processes in them? Can you please give me the output of "loginctl", and then "loginctl <id>" for a few session IDs? (local ones look like "c2", remote ones should just be a number).

[Martin Pitt (pitti)]

I. e. it should remove them as soon as the sessions are actually closed. Otherwise, if you log out, they merely go into state "closing", and wait until all remaining processes are going away. If you start screen sessions or other background jobs, they'll stay around for those.

[Daniel GALAMBOS (dancsa)]

The server (vm) used as a management gateway to other service. Sessions opened by SSH. Remote machine opens a SSH connection with a key, by SSH force command, a python script opens, do the stuffs, and exits.

Sessions are in 'closing' state, but no processes (not even zombies) are exist under this user.

root@ad:/run/systemd/sessions# grep -r 'STATE' . | grep closing | wc -l
12419

root@ad:/run/systemd/sessions# grep -r 'STATE' . | grep -v closing | wc -l
1

root@ad:/run/systemd/sessions# loginctl list-users
       UID USER
      1001 ad-sync-prod
         0 root
root@ad:/run/systemd/sessions# ps -u ad-sync-prod
  PID TTY TIME CMD
<empty list>

root@ad:/run/systemd/sessions# loginctl user-status ad-sync-prod
ad-sync-prod (1001)
           Since: Fri 2015-02-27 13:40:08 CET; 5 days ago
           State: opening
Failed to query ControlGroup: No such interface 'org.freedesktop.systemd1.Slice'
        Sessions: 35752 35751 35750 35749 35748 35747 35746 35745 35744 35743 35742 35740 35739 35738 35737 35736 35735 35734 35733 35732 35731 35730 35729 35728 35726 35725 35724 35723 35722
            Unit: user-1001.slice

[cybertoast (cybertoast)]

I'm noticing this as an issue as well. My scenario is in using Fabric or Ansible to do deployments. A large number of these dangling SSH files are in the /run/systemd/sessions/ folder and eventually cause the "out of disk space" issue.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Satish Patel (satish-txt)]

We have same issue :(

[David Clarke (a-launchpad-uber-geek-nz)]

This only seems to happen for us, on 14.04 LTS at least, if the package 'systemd' is installed rather than merely libsystemd-{daemon,login}0, and systemd-{services,shim}. It could be that a workaround is simply to remove the 'systemd' package, assuming it's not required for your particular setup.

[Vasya Pupkin (shadowlmd)]

I am having the same issue on Ubuntu 16.04 now. All sessions appear to be for one user that is used for incoming ssh connections for rsync. Here's an example:

$ loginctl show-session 59
Id=59
User=1004
Name=raccess
Timestamp=Чт 2018-03-29 20:20:03 MSK
TimestampMonotonic=1763641802
VTNr=0
Remote=yes
RemoteHost=216.126.59.227
Service=sshd
Scope=session-59.scope
Leader=2814
Audit=59
Type=tty
Class=user
Active=yes
State=closing
IdleHint=no
IdleSinceHint=0
IdleSinceHintMonotonic=0

I don't see any processes running under user raccess apart from system ones:

$ ps aux | grep raccess | grep -v grep
raccess 2822 0.0 1.6 6384 4132 ? Ss 20:20 0:00 /lib/systemd/systemd --user
raccess 2823 0.0 0.4 7316 1096 ? S 20:20 0:00 (sd-pam)

And here's an output of lsof -u raccess: https://pbin.cf/444b2d29

This is happening on two servers.

[Vasya Pupkin (shadowlmd)]

All sessions appear in closing state, here's another one:

$ loginctl show-session 77
Id=77
User=1004
Name=raccess
Timestamp=Чт 2018-03-29 20:30:02 MSK
TimestampMonotonic=2363185101
VTNr=0
Remote=yes
RemoteHost=216.126.59.227
Service=sshd
Scope=session-77.scope
Leader=3389
Audit=77
Type=tty
Class=user
Active=yes
State=closing
IdleHint=no
IdleSinceHint=0
IdleSinceHintMonotonic=0