Apache2 is still Range header DoS vulnerable if gzip compression is enabled

Bug #839569 reported by Pawel Tecza
266
This bug affects 2 people
Affects Status Importance Assigned to Milestone
apache2 (Ubuntu)
Invalid
High
Steve Beattie

Bug Description

Today I've upgraded my Apache2 packages from version 2.2.14-5ubuntu8.4 to 2.2.14-5ubuntu8.6 on Ubuntu Lucid 10.04 LTS boxes we use here. It seems that the WWW server is still Range header DoS (CVE-2011-3192) vulnerable if I have enabled gzip compression-on-the-fly.

This is a result of my testing for disabled compression:

root@server:~# ls /etc/apache2/mods-enabled/ |grep deflate
deflate.conf
deflate.load
root@server:~#

root@server:~# grep gzip /etc/apache2/apache2.conf
SetEnv no-gzip
root@server:~#

ptecza@laptop:~$ perl killapache.pl my.server.pl
Host does not seem vulnerable
ptecza@laptop:~$

ptecza@laptop:~$ telnet my.server.pl www
Trying 11.22.33.44...
Connected to my.server.pl.
Escape character is '^]'.
HEAD / HTTP/1.1
Host: my.server.pl
Range:bytes=0-100
Accept-Encoding: gzip
Connection: close

HTTP/1.1 200 OK
Date: Fri, 02 Sep 2011 12:58:33 GMT
Server: Apache
Set-Cookie: FSESSIONID=m; path=/; domain=my.server.pl; expires=Fri, 02-Sep-2011 13:13:33 GMT
X-Powered-By: PHP/5.3.2-1ubuntu4.9
Set-Cookie: e9231db0fb41e22cabb573ecb7ba8b90=e6mmqpmgrkfiluc6696irp2a11; path=/
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Set-Cookie: lang=deleted; expires=Thu, 02-Sep-2010 12:58:35 GMT; path=/
Set-Cookie: jfcookie=deleted; expires=Thu, 02-Sep-2010 12:58:35 GMT; path=/
Set-Cookie: jfcookie[lang]=deleted; expires=Thu, 02-Sep-2010 12:58:35 GMT; path=/
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified: Fri, 02 Sep 2011 12:58:37 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Content-Type: text/html; charset=utf-8

Connection closed by foreign host.
ptecza@laptop:~$

and below you can see the result for enabled compression:

root@server:~# grep gzip /etc/apache2/apache2.conf
##SetEnv no-gzip
root@server:~#

ptecza@laptop:~$ perl killapache.pl my.server.pl
host seems vuln
ptecza@laptop:~$

ptecza@laptop:~$ telnet my.server.pl www
Trying 11.22.33.44...
Connected to my.server.pl.
Escape character is '^]'.
HEAD / HTTP/1.1
Host: my.server.pl
Range:bytes=0-100
Accept-Encoding: gzip
Connection: close

HTTP/1.1 206 Partial Content
Date: Fri, 02 Sep 2011 13:14:31 GMT
Server: Apache
Set-Cookie: FSESSIONID=m; path=/; domain=my.server.pl; expires=Fri, 02-Sep-2011 13:29:31 GMT
X-Powered-By: PHP/5.3.2-1ubuntu4.9
Set-Cookie: e9231db0fb41e22cabb573ecb7ba8b90=uudo5qjcmlke7egs3sddbr57a0; path=/
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Set-Cookie: lang=deleted; expires=Thu, 02-Sep-2010 13:14:31 GMT; path=/
Set-Cookie: jfcookie=deleted; expires=Thu, 02-Sep-2010 13:14:31 GMT; path=/
Set-Cookie: jfcookie[lang]=deleted; expires=Thu, 02-Sep-2010 13:14:31 GMT; path=/
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified: Fri, 02 Sep 2011 13:14:32 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Range: bytes 0-19/20
Content-Length: 20
Connection: close
Content-Type: text/html; charset=utf-8

Connection closed by foreign host.
ptecza@laptop:~$

CVE References

Changed in apache2 (Ubuntu):
assignee: nobody → Steve Beattie (sbeattie)
Revision history for this message
Steve Beattie (sbeattie) wrote :
Download full text (4.8 KiB)

Hi Paweł,

Thanks for taking the time to report this issue.

First off, killapache.pl's testapache() function for whether the host is vulnerable is not very accurate; all it tests is whether apache returns 206 Partial Content when given the byte range of '0-'. In our update, we included the upstream commit
http://svn.apache.org/viewvc?view=revision&sortby=date&revision=1163833 which returns a 206 on that byte range because debian had a bug report http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=639825 of a client breaking when 200 was returned for that exact byte range.

Second, I'm unable to reproduce the difference in behavior that you're seeing when gzip compression is enabled or disabled; with a stock setup, my attempts look like:

  $ dpkg -l apache2-mpm-prefork | tail -1
  ii apache2-mpm-prefork 2.2.14-5ubuntu8.6

  $ apache2ctl -t -D DUMP_MODULES |grep deflate
  apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
  Syntax OK
   deflate_module (shared)

  # ask for gzip encoding
  $ nc localhost 80
  HEAD / HTTP/1.1
  Host: localhost
  Range:bytes=0-100
  Accept-Encoding: gzip
  Connection: close

  HTTP/1.1 206 Partial Content
  Date: Sat, 03 Sep 2011 00:04:37 GMT
  Server: Apache/2.2.14 (Ubuntu)
  Last-Modified: Wed, 31 Aug 2011 16:56:54 GMT
  ETag: "482db-b1-4abd003eaae6f"
  Accept-Ranges: bytes
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Range: bytes 0-100/146
  Content-Length: 101
  Connection: close
  Content-Type: text/html

  # no gzip encoding
  $ nc localhost 80
  HEAD / HTTP/1.1
  Host: localhost
  Range:bytes=0-100
  Connection: close

  HTTP/1.1 206 Partial Content
  Date: Sat, 03 Sep 2011 00:05:26 GMT
  Server: Apache/2.2.14 (Ubuntu)
  Last-Modified: Wed, 31 Aug 2011 16:56:54 GMT
  ETag: "482db-b1-4abd003eaae6f"
  Accept-Ranges: bytes
  Content-Length: 101
  Vary: Accept-Encoding
  Content-Range: bytes 0-100/177
  Connection: close
  Content-Type: text/html

  # disable gzip encoding
  $ sudo a2dismod deflate
  Module deflate disabled.
  Run '/etc/init.d/apache2 restart' to activate new configuration!
  $ sudo /etc/init.d/apache2 restart
   * Restarting web server apache2
  apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
   ... waiting apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
     ...done.
  $ apache2ctl -t -D DUMP_MODULES |grep deflate
  apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
  Syntax OK

  # ask for gzip encoding; shouldn't get it
  $ nc localhost 80
  HEAD / HTTP/1.1
  Host: localhost
  Range:bytes=0-100
  Accept-Encoding: gzip
  Connection: close

  HTTP/1.1 206 Partial Content
  Date: Sat, 03 Sep 2011 00:07:27 GMT
  Server: Apache/2.2.14 (Ubuntu)
  Last-Modified: Wed, 31 Aug 2011 16:56:54 GMT
  ETag: "482db-b1-4abd003eaae6f"
  Accept-Ranges: bytes
  Content-Length: 101
  Content-Range: bytes 0-100/177
  Connection: close
  Content-Type: text/html

  # don't ask for gzip encoding
  $ nc localhost 80
  HEAD / ...

Read more...

Changed in apache2 (Ubuntu):
status: New → Incomplete
importance: Undecided → High
Revision history for this message
Pawel Tecza (ptecza) wrote :

Hello Steve,

At first, thanks a lot for your response!

When Apache's gzip compression is on here and I enable killapache() function of Perl exploit script, then my web server consumes 100% memory and CPU and it's unusable. So it seems that Apache upgrade hasn't helped me a lot.

I have a little more complex Apache setup (PHP, mod_rewrite, mod_substitute, LDAP auth, etc). This is a back-end server behind revers proxy (Pound), but when I test it, then I go there directly. Of course I was testing upgraded Apache for disabled the mitigation options.

I don't have an idea why Content-Length and Content-Range fields aren't returned, but I'll take a look at my config again and I'll let you know if I find something related.

Revision history for this message
Pawel Tecza (ptecza) wrote :

Today I did a deeper investigation, but still I can't explain why my Apache doesn't send "206 Partial Content" response if a gzip compression is disabled or unset. It sends only "200 OK" and then Content-Range: and Content-Length: fields are missing, of course. Also I noticed that it doesn't reply "Accept-Ranges: bytes" at all, even when mod_deflate is enabled.

But probably the reason of my issue with killapache.pl script is not a buggy Apache version. Simply my virtual server is too weak to handle 50 connections to heavy Joomla page in the same moment :) When I decrease "numforks" argument of the exploit script from 50 by default to 10, then the server is still working. Yes, it consumes 100% CPU, but it still has some free memory to run. Please note that it's a true both for old and buggy Apache and for new and fixed version.

Fortunately I have the antiloris.sh script for protecting me against the slowloris attack, so it can drop all script kiddies which use killapache.pl :)

So probably it's a good idea to close my bug report.

Revision history for this message
Steve Beattie (sbeattie) wrote :

Paweł,

Can you confirm that sending a request with an overlapping byte range e.g.:

  HEAD / HTTP/1.1
  Host: localhost
  Range:bytes=1-15,10-35,8-9,14-22,0-5,23-
  Accept-Encoding: gzip
  Connection: close

returns "200 OK"?

Perhaps you could report what modules you have loaded? "apache2ctl -t -D DUMP_MODULES" will do it.

I'm going to leave this bug open and make it public, as I've received another report via email of a lucid user claiming that the update didn't help their system, either, and if possible, I'd like them to chime in here, too.

visibility: private → public
Revision history for this message
Pawel Tecza (ptecza) wrote :
Download full text (3.2 KiB)

Hi Steve,

Yes, I can confirm that my Apache returns "200 OK" for that request:

root@server:~# nc localhost 80
HEAD / HTTP/1.1
Host: localhost
Range:bytes=1-15,10-35,8-9,14-22,0-5,23-
Accept-Encoding: gzip
Connection: close

HTTP/1.1 200 OK
Date: Wed, 07 Sep 2011 08:51:43 GMT
Server: Apache
Set-Cookie: FSESSIONID=m; path=/; domain=my.server.pl; expires=Wed, 07-Sep-2011 09:06:43 GMT
X-Powered-By: PHP/5.3.2-1ubuntu4.9
Set-Cookie: e9231db0fb41e22cabb573ecb7ba8b90=br5glf7jvticj4gs4kiglb5d13; path=/
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Set-Cookie: lang=deleted; expires=Tue, 07-Sep-2010 08:51:43 GMT; path=/
Set-Cookie: jfcookie=deleted; expires=Tue, 07-Sep-2010 08:51:43 GMT; path=/
Set-Cookie: jfcookie[lang]=deleted; expires=Tue, 07-Sep-2010 08:51:43 GMT; path=/
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified: Wed, 07 Sep 2011 08:51:45 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 20
Connection: close
Content-Type: text/html; charset=utf-8

root@server:~#

but it still returns "206 Partial Content" for non overlapping byte range:

root@server:~# nc localhost 80
HEAD / HTTP/1.1
Host: localhost
Range:bytes=0-100
Accept-Encoding: gzip
Connection: close

HTTP/1.1 206 Partial Content
Date: Wed, 07 Sep 2011 08:58:34 GMT
Server: Apache
Set-Cookie: FSESSIONID=m; path=/; domain=my.server.pl; expires=Wed, 07-Sep-2011 09:13:34 GMT
X-Powered-By: PHP/5.3.2-1ubuntu4.9
Set-Cookie: e9231db0fb41e22cabb573ecb7ba8b90=3pv0n50sf008la91se85acne22; path=/
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Set-Cookie: lang=deleted; expires=Tue, 07-Sep-2010 08:58:34 GMT; path=/
Set-Cookie: jfcookie=deleted; expires=Tue, 07-Sep-2010 08:58:34 GMT; path=/
Set-Cookie: jfcookie[lang]=deleted; expires=Tue, 07-Sep-2010 08:58:34 GMT; path=/
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified: Wed, 07 Sep 2011 08:58:35 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Range: bytes 0-19/20
Content-Length: 20
Connection: close
Content-Type: text/html; charset=utf-8

root@server:~#

And here you can see my enabled Apache's modules:

root@server:~# apache2ctl -t -D DUMP_MODULES
Loaded Modules:
 core_module (static)
 log_config_module (static)
 logio_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authnz_ldap_module (shared)
 authz_default_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cgi_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 expires_module (shared)
 headers_module (shared)
 ldap_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 pagespeed_module (shared)
 php5_module (shared)
 proxy_module (shared)
 proxy_connect_module (shared)
 proxy_html_module (shared)
 proxy_http_module (shared)
 reqtimeout_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 ssl_module (shared)...

Read more...

Revision history for this message
Upen (upen) wrote :

Hi,

I am the other user who reported that even after fully patching 10.04 LTS, I see that my virtual Lucid is still vulnerable. I am not sure where exactly is the problem, system resources or apache bug or my configuration.

OS: Ubuntu 10.04.3 LTS
Memory = 512 MB
1 CPU : model name : Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz

nc www.server.name 80
HEAD / HTTP/1.1
Host: www.server.name
Range:bytes=1-15,10-35,8-9,14-22,0-5,23-
Accept-Encoding: gzip
Connection: close

HTTP/1.1 200 OK
Date: Wed, 07 Sep 2011 15:05:30 GMT
Server: Apache/2.2.14 (Ubuntu)
Last-Modified: Mon, 02 Aug 2010 21:42:40 GMT
ETag: "e51e-b1-48cde146fd1b1"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 146
Connection: close
Content-Type: text/html

apache2ctl -t -D DUMP_MODULES
Loaded Modules:
 core_module (static)
 log_config_module (static)
 logio_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 actions_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_default_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cgi_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 headers_module (shared)
 mime_module (shared)
 security2_module (shared)
 negotiation_module (shared)
 php5_module (shared)
 reqtimeout_module (shared)
 setenvif_module (shared)
 status_module (shared)
 unique_id_module (shared)
Syntax OK

Revision history for this message
Stefan Fritsch (sf-sfritsch) wrote :

Note that it is impossible to determine from apache's response alone if apache is vulnerable or not. The only way to check it is to do a request with lots of overlapping ranges (like killapache does) and check if the process size increases a lot or not. On 32bit machines, the original vulnerability caused an increase by 70-90MB per process/thread. On 64bit machines, the increase was even larger.

Revision history for this message
Pawel Tecza (ptecza) wrote :

Stefen,

Yes, you're absolutely right! We can only check in that way if a server supports byte Range headers.

killapache.pl causes that even my upgraded server is DoS'ed, but it's rather related to my Apache's config. Probably I need to decrease a value of MaxClients and MaxKeepAliveRequests, because I have too less resources to handle the request of 50 forks of killapache.pl.

Revision history for this message
Upen (upen) wrote :

In addition to my earlier comment and after reading other comments, I'd like to add now that I changed MaxClients for apache2( In Pre fork MPM ) on my virtual Lucid and restarted apache2. Ran killapache but this time the load on system didn't go high, and apache2 processes were limited to 3(observed using top command).

Before setting MaxClients I ran a guideline script from http://rimuhosting.com/howto/memory.jsp to find MaxClient number that I can use on my lucid. Script showed me number 3 and was surprised why so low as the default is 150 :). Anyways looking at free physical memory may be it makes sense to use 3 on system.

So I am now certain that in my case it was unhealthy configuration of apache2 that was the cause that made fully patched Lucid system die after running against kill apache script. And, even after using 1000 forks in killapache script my Lucid continued to stay in good health.

Revision history for this message
Steve Beattie (sbeattie) wrote :

Paweł and Upen, thanks for following up. Based on your comments, I'm going to close this bug report; please re-open it if you find any evidence that suggests the fix for CVE-2011-3192 is incomplete.

Stefan, thanks for chiming in.

Changed in apache2 (Ubuntu):
status: Incomplete → Invalid
Revision history for this message
Pawel Tecza (ptecza) wrote :

Upen, thank you for the info about the Apache's memory guideline script. I'll try it too.

Revision history for this message
Upen (upen) wrote :

You are welcome. Thanks to you for creating bug report and Steve for your immediate response. Appreciate it.

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.