hardy: openssh-server oom_adj can lead to denial of service

Bug #293000 reported by Karsten Suehring
280
This bug affects 3 people
Affects Status Importance Assigned to Milestone
openssh (Debian)
Fix Released
Unknown
openssh (Ubuntu)
Fix Released
Undecided
Unassigned
Hardy
Won't Fix
Medium
Unassigned

Bug Description

Binary package hint: openssh-server

The ssh init script sets the /proc/$PID/oom_adj value to -17 to avoid being killed by the OOM killer in low memory situations. Unfortunately all child processes of sshd inherit this setting.

So any user with ssh access can easily launch a process which accumulates memory without being killed by the kernel until the system gets to out of memory kernel panic. This will lead to a denial of service.

The bug is already reported in the debian bug tracker under the following location:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=480020

The fix is included in openssh/1:4.7p1-11. Please update Hardy to this package version.

Revision history for this message
Karsten Suehring (suehring) wrote :

As work-around I would suggest setting SSHD_OOM_ADJUST to 0 in /etc/default/ssh. This allows the killing of ssh and child processes by the OOM killer again.

Revision history for this message
Caspar Clemens Mierau (leitmedium) wrote :

Thanks for taking the time to report this bug. Please check if this has been solved on an up to date Ubuntu. If this is an issue on Hardy for you,

Please note: I assume it will not be solved by an update of ssh as packages on a released version are only patched (not updated) and this only when a severe loss of user date or similar things are close to happen.

Changed in openssh:
status: New → Incomplete
Daniel Hahler (blueyed)
Changed in openssh (Ubuntu):
status: Incomplete → Triaged
Revision history for this message
Mathias Gug (mathiaz) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. However, I am closing it because the bug has been fixed in the latest development version of Ubuntu - Lucid Lynx.

This is a significant bug in Ubuntu. If you need a fix for the bug in previous versions of Ubuntu, please do steps 1 and 2 of the SRU Procedure [1] to bring the need to a developer's attention.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates#Procedure

Changed in openssh (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Dave Martin (dave-martin-arm) wrote :

This bug either wasn't fixed or there has been a recent regression.

Ubuntu lucid
openssh-server 1:5.3p1-3ubuntu1

/etc/default/ssh: SSHD_OOM_ADJUST=-17

As well as causing kernel panics, a malicious user can use this technique to kill off trusted root daemons and (if they use a port >= 1024) launch spoofing processes.

Not only is this a DoS risk, it can also lead to takedown of critical system components required for the SSH session to work (NetworkManager etc.)

Changed in openssh (Ubuntu):
status: Fix Released → Confirmed
Revision history for this message
Dave Martin (dave-martin-arm) wrote :

To confirm, sshd's child processes do indeed inherit the oom_adjust setting.

Revision history for this message
Karsten Suehring (suehring) wrote :

I think the debian patch had been applied in releases after Hardy. But the daemon only resets the oom_adj value that it was originally called with. So in Jaunty a DHCP restart script which had the oob_adj value of -17 itself caused the same effect. See bug report #390556.

I have not checked lucid yet. But the issue itself caused me a lot of headaches and it would have been nice if it would have been fixed back in 2008 in a Ubuntu version which is labeled "LTS".

Revision history for this message
Karsten Suehring (suehring) wrote :

OK, I've debugged a bit into the lucid upstart scripts:

First, I can confirm the regression.

The oom_adj patch is still in place, which is the good news. The bad news is, that the problem is now caused by the upstart script /etc/init/ssh.conf

Apparently the author didn't understand how the oom_adj patch works. The config file contains the lines:

  #replaces SSHD_OOM_ADJUST in /etc/default/ssh
  oom never

This comment is false!

"oom never" sets the oom_adj value to -17 before the sshd is started.

The sshd patch now saves that value, sets its own oom_adj value to the one from the environment variable SSHD_OOM_ADJUST (if set) and sets the oom_adj value of all child processes to the saved value, which is -17 in this case.

So all children of sshd cannot be killed which leads to the well known denial of service issue.

Revision history for this message
Colin Watson (cjwatson) wrote :

I understood the oom_adj patch perfectly, having written it. Apparently I misunderstood how Upstart's 'oom' stanza worked though ...

Thanks for the analysis; I'll look into this.

tags: added: regression-potential
Revision history for this message
Karsten Suehring (suehring) wrote :

Colin, thanks for the reply. Maybe I got a wrong impression ;-)

After seeing the issue show up again and again over the last two years, my suggestion would be to change the oom_adj patch itself to set the child oom_adj value always to zero, independent of the value that it was called with.

I understand that the current behavior gives more freedom, but it's not obvious enough, how it works. Basically in the current implementation every caller needs to be aware of it's own oom_adj value which means there is some logic required before starting sshd. We can probably never be sure that the author of every startup script knows what to do. I've seen the problem in Jaunty where a network startup script had oom_adj equal to -17 which was not reset (bug #390556) and now even you made the mistake.

I also cannot imagine any reasons why somebody would need a sshd child oom_adj value different than zero.

Revision history for this message
Colin Watson (cjwatson) wrote : Re: [Bug 293000] Re: hardy: openssh-server oom_adj can lead to denial of service

Yes, I think you're probably right. I was considering a few possible
alternatives and this seems the least bad.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:5.3p1-3ubuntu2

---------------
openssh (1:5.3p1-3ubuntu2) lucid; urgency=low

  * Always set child processes' OOM adjustment to 0, since Upstart will have
    set sshd's OOM adjustment on startup and so simply restoring the startup
    value won't work (LP: #293000). Thanks to Karsten Suehring for
    analysis.
  * Limit automatic respawning to 10 times in 5 seconds (thanks to Fumihito
    YOSHIDA for the suggestion; LP: #533352).
 -- Colin Watson <email address hidden> Sat, 06 Mar 2010 22:00:19 +0000

Changed in openssh (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Nathan Catlow (nat-uncon) wrote :

Is there going to be a back port to Hardy 8.04 LTS?

I have had a serious issue with a Virtual Server where the only access is via SSHD. This resulted in an errant CPAN update downing the entire box due to all services started via SSH being oom_adj == -17 and therefore not being killed when out of control.

I cannot risk setting SSHD_OOM_ADJUST=0 in /etc/default/ssh as it is essential that sshd remains running at all cost.

I have temporarily put echo "0" > /proc/self/oom_adj in /etc/bash.bashrc as a workaround. At least this may stop some potential problems.

This is serious in a Virtual Server environment, where RAM is typically low and remote access requirement high.

tags: added: lucid regression-release
removed: regression-potential
Changed in openssh (Ubuntu Hardy):
status: New → Confirmed
importance: Undecided → Medium
Jack Leigh (leighman)
tags: added: hardy
Revision history for this message
Cal Leeming (sleepycal) wrote :

Hi all,

I have recently came up against this problem myself, and have fixed / submitted a patch to the openssh development team. Here is the original report:

So I modified the code to try and repair this oom_adj problem...

port-linux.c:
line 235: //static int oom_adj_save = INT_MIN;
line 236: static int oom_adj_save = 0;
line 277: verbose("Set %s to %d - sleepycal", OOM_ADJ_PATH, oom_adj_save);

I then ran compiled the package, ran SSHd, and yet we still have -17 in oom_adj_save. Wtf? Now, I'm not much of a C coder, but this is weird even in my books...

May 30 22:18:19 vicky sshd[12825]: Set /proc/self/oom_adj to -17 - sleepycal

So, I went all out crazy, and did the following patch:

        static int sleepycal_oom_adj_save = 0;
        verbose("sleepycal_oom_adj_save=%d", sleepycal_oom_adj_save);

        if (fprintf(fp, "%d\n", sleepycal_oom_adj_save) <= 0)
                verbose("error writing %s: %s", OOM_ADJ_PATH, strerror(errno));
        else
                verbose("Set %s to %d - sleepycal", OOM_ADJ_PATH, sleepycal_oom_adj_save);

And it worked!!! :)

May 30 22:27:12 vicky sshd[2532]: sleepycal_oom_adj_save=0
May 30 22:27:12 vicky sshd[2532]: Set /proc/self/oom_adj to 0 - sleepycal

root@vicky:~/openssh-5.5p1# cat /proc/2532/oom_adj
0

So, it turns out that it is actually OpenSSH which is broken, after almost 3 days of frustrating digging through millions of lines of code lol. Anyways, would appreciate if someone could get this merged into master (obv rename the vars if you want).

Attached is the appropriate patch file as of openssh-5.5p1

Cal

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Thank you for reporting this bug to Ubuntu. hardy has reached EOL
(End of Life) and is no longer supported. As a result, this bug
against hardy is being marked "Won't Fix". Please see
https://wiki.ubuntu.com/Releases for currently supported Ubuntu
releases.

Please feel free to report any other bugs you may find.

Changed in openssh (Ubuntu Hardy):
status: Confirmed → Won't Fix
Mathew Hodson (mhodson)
affects: debian → openssh (Debian)
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.