[FFE] apache2 DoS attack using slowloris

Bug #392759 reported by LiraNuna on 2009-06-27
280
This bug affects 3 people
Affects Status Importance Assigned to Milestone
apache2 (Debian)
Fix Released
Unknown
apache2 (Ubuntu)
High
Chuck Short
Lucid
High
Chuck Short

Bug Description

Ubuntu's apache2 is vulnerable to a DoS attack called "slowloris" (more info at http://ha.ckers.org/slowloris/). I have successfully taken down several ubuntu servers using this method and my server is no less vulnerable.

The answer was supposed to be mod_evasive (package libapache2-mod-evasive or http://www.zdziarski.com/projects/mod_evasive/) but the script have been recently modified to overcome that protection.

The new answer is mod_antiloris (ftp://ftp.monshouwer.eu/pub/linux/mod_antiloris/) which is unpackaged for debian/ubuntu. I urge you to include this mod as default apache2 installation just like open*-blacklist packages are.

LiraNuna (liranuna) wrote :

I forgot to mention my Ubuntu version - 8.04.2 server LTS fully upgraded.

LiraNuna (liranuna) on 2009-06-27
visibility: private → public
Changed in apache2 (Ubuntu):
importance: Undecided → Wishlist
status: New → Confirmed
Dekar (dekar-wc3edit) wrote :

high A real problem, exploitable for many people in a default
  installation. Includes serious remote denial of services,
  local root privilege escalations, or data loss.

Thus it should be changed to HIGH and fixed ASAP!

Jonathan Marsden (jmarsden) wrote :

Dekar: Did you actually test this at all? Please provide some evidence to support your claims.

You have said that you believe this issue is:

> A real problem, exploitable for many people in a default
> installation. Includes serious remote denial of services,
> local root privilege escalations, or data loss.

The default installation, when one installs apache2 using

  sudo apt-get install apache2

uses the apache2-mpm-prefork module, not apache2-mpm-worker. The article by LiraNuna clearly states:

   I assume you are using the threaded version of Apache, else you are not vulnerable to this type of attack.

Please justify your claims about this being a high priority issue, affecting many people in the default installation, in the light of this.

More generally, if you believe this to be a significant issue for many people, rather than making unfounded statements here, please do the community a service and package the module that you wish to see included in Ubuntu :)

Jonathan Marsden (jmarsden) wrote :

Apparently apache2-mpm-worker *is* the default choice of mpm, at least some of the time (though the machine I first checked on had apache-mpm-prefork, I am unsure why). My mistake, it would appear.

Using apache2-mpm-prefork would appear to be a viable workaround, unless it leads to performance issues on a heavily loaded server.

So this comes down to whether slowloris is "a serious remote denial of service" (since it does not cause local root privilege escalation or data loss, as far as I know).

Dekar: What makes you believe slowloris is a "serious remote denial of service"? Is it currently in widespread use? I have no idea what the criteria for "serious" might be!

In some ways, this bug is at its heart a request to package mod-antiloris. There appears to already be a libapache2-mod-antiloris package in Debian unstable and Debian testing; perhaps it can be synced into Ubuntu?

Dekar (dekar-wc3edit) wrote :

It is a serious remote denial of service! It can be used from a single modem line and take down a whole server without generating any logfiles except normal access logs. It's funny how you guys treat it as "Wishlist" for three months even though it's one of the worst remote denial of service attacks that can happen. It just takes your whole Website down! And hey it affects Ubuntu-LTS-Server! I just tried it with my home DSL line against Ubuntu.com and it went down about instantly! What frickin' else do you want? And yeah my page with about 4000 unique daily visitors was affected as well - not sure if you call that big, but at least ubuntu.com should be big enough, isn't it?

Concerning the picture - left one is local slowloris against ubuntu.com - right one is links2 from my server. Guess what? It didn't load anymore till I stopped slowloris! So if you don't care whether ubuntu.com is reachable or not, why do you think they bought the domain in the first place? If that's not serious - what else do you want?

Dekar (dekar-wc3edit) on 2009-09-21
Changed in apache2 (Debian):
importance: Undecided → Unknown
status: New → Unknown
status: Unknown → Confirmed
Changed in apache2 (Debian):
status: Confirmed → New
Dekar (dekar-wc3edit) on 2009-09-21
Changed in apache2 (Debian):
status: New → Confirmed
Stefan Fritsch (sf-sfritsch) wrote :

Some comments:

- All Apache MPMs are affected. The sole exception may be if you use the event MPM without SSL.

- The slowloris attack leaves plenty of error 400 entries in the access log.

- Using iptables connlimit with a reasonable maximum number of connections per IP (like 1/5 or 1/10 of what you server can handle) will give you good protection from single attacking hosts. When the attacker has many hosts (i.e. a botnet) you have lost anyway.

- mod_antiloris has some design issues as discussed on the httpd-dev mailing list. Also, it does not protect against a slightly modified attack. Therefore mod_antiloris is not the general solution.

- I hope that mod_reqtimeout may be a better approach, but the discussion and testing is not finished yet.

For now, the recommendation is to use iptables.

Kees Cook (kees) wrote :

I'm not aware of a reasonable upstream solution to this yet; it is a detectable, direct, denial of service, so there are methods that admins can use to work around (iptables, etc).

Dekar (dekar-wc3edit) wrote :

Can't you at least change the importance level? It is way more important then "wishlist", it is one of the worst things that could happen to a fresh ubuntu server installation! Concerning to the guidelines it should be high!

vhahn (victor-tirm) wrote :

Second this. This is a high priority security bug. You can't seriously classify this as whishlist?!

Kees Cook (kees) wrote :

Changing the priority doesn't really change the availability of a solution. :) But I don't mind classifying it this way.

Changed in apache2 (Ubuntu):
importance: Wishlist → High
Dekar (dekar-wc3edit) on 2009-09-27
Changed in apache2 (Ubuntu):
assignee: nobody → Dekar (dekar-wc3edit)
assignee: Dekar (dekar-wc3edit) → nobody
assignee: nobody → Ubuntu Security Team (ubuntu-security)
Kees Cook (kees) on 2009-09-28
Changed in apache2 (Ubuntu):
assignee: Ubuntu Security Team (ubuntu-security) → nobody
Changed in apache2 (Debian):
status: Confirmed → New
Changed in apache2 (Ubuntu):
status: Confirmed → Triaged
Oliver (oliver341) wrote :

The connlimit module in iptables is an excellent defence against Apache Denial of Service attacks. However, since upgrading to Karmic, iptables is no longer blocking simultaneous connections when requested for me.

I had previously been using:
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 10 -j REJECT

However, worryingly, it no longer works for me. I can establish 20 simultaneous connections with the above firewall rule in place. I believe this should be fixed with some urgency, as my webserver has already been taken offline once by an attack (I stopped the attack by firewalling the attacker's IP address manually).

I've filed a bug report, please check your iptables connlimit and report back either way:
https://bugs.launchpad.net/ubuntu/+source/iptables/+bug/478290

Changed in apache2 (Debian):
status: New → Fix Released
Chuck Short (zulcss) wrote :

This has been fixed in 2.2.16 by enabling the module mod-reqtimeout. It has been enabled by default in the next release in Debian already. I think this is an important fix for lucid that should be necessary. I have attached the debdiff and build log. I have ran ab against it and it has not affected my server. If you have any questions please let me know.

Regards
chuck

summary: - apache2 DoS attack using slowloris
+ [FFE] apache2 DoS attack using slowloris
Chuck Short (zulcss) wrote :
Chuck Short (zulcss) wrote :
Stefan Fritsch (sf-sfritsch) wrote :

You have written "enable it by default" in the changelog, but AFAICS, you have missed the postinst change that actually enables the module. You may want to merge the NEWS.Debian entry, too.

Chuck Short (zulcss) wrote :

Stefan,

Thanks Ill update my debdiff tonight then.

Regards
chuck

Chuck Short (zulcss) wrote :

Updated debdiff

Thierry Carrez (ttx) on 2010-04-09
Changed in apache2 (Ubuntu Lucid):
assignee: nobody → Chuck Short (zulcss)
Steve Langasek (vorlon) wrote :

Please also grab the NEWS.Debian as Stefan suggests.

FFe granted.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apache2 - 2.2.14-5ubuntu7

---------------
apache2 (2.2.14-5ubuntu7) lucid; urgency=low

  * debian/patches/206-fix-potential-memory-leaks.dpatch: Fix potential memory
    leaks by making sure to not destroy bucket brigades that have been created
    by earlier filters. Backported from 2.2.15.
  * debian/patches/206-report-max-client-mpm-worker.dpatch: Don't report server
    has reached MaxClients until it has. Backported from 2.2.15
  * debian/config-dir/apache2.conf: Make the Files ~ "^\.ht" block in apache2.conf
    more secure by adding Satisfy all. (Debian bug: #572075)
  * debian/rules, debian/patches/209-backport-mod-reqtimeout.dpatch,
    debian/config2-dir/mods-available/reqtimeout.load,
    debian/config2-dir/mods-available/reqtimeout.conf debian/NEWS : Backport the
    mod-reqtimeout module from 2.2.15, this will mitigate apache slowloris
    bug in apache. Enable it by default. (LP: #392759)
 -- Chuck Short <email address hidden> Mon, 05 Apr 2010 09:53:35 -0400

Changed in apache2 (Ubuntu Lucid):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.