Ubuntu

[SRU] memory leaks in apache2 when running mod_ssl

Reported by Jamie Strandboge on 2008-04-30
30
Affects Status Importance Assigned to Milestone
apache2 (Ubuntu)
High
Martin Pitt
Hardy
High
Unassigned
Intrepid
High
Martin Pitt
openssl (Ubuntu)
Undecided
Unassigned
Hardy
Undecided
Unassigned
Intrepid
Undecided
Unassigned

Bug Description

Binary package hint: apache2

The following came in an email to <email address hidden>. I asked the sender to file a report, but it hasn't happened yet, so I am filing it on his behalf (essentially pasting the email here).

--- EMAIL FROM USER ---
After upgrading our servers from Ubuntu 6.06 to Ubuntu 8.04 we started seeing MASSIVE memory leaks in Apache 2.2 (mpm-worker). Before decreasing MaxRequestsPerChild we actually got kernel panic OOMs so in our view this is a serious DenialOfSerivce vulnerability.

I have spent some time debugging the issue using valgrind and some custom debugging printf's and I have so far concluded that it is related to SSLv3/TLSv1 zlib compression.

How to reproduce the leak:
(1) Set up a SSL-enabled host in Apache2.2. Session cache and the like does not seem to matter, but make sure that the childs run long enough to notice the leak.

(2) Verify that zlib compression is enabled:
$ openssl s_client -tls1 -connect host:port

(3) Flood the host with compression enabled requests (no SSLv2):
$ ab -n x -c y -f tls1 https://XXX

Valgrind indicates that the leak occurs inside crypto/comp/c_zlib.c in libssl0.9.8g:

static int zlib_stateful_init(COMP_CTX *ctx)
        {
        int err;
        struct zlib_state *state =
-> (struct zlib_state *)OPENSSL_malloc(sizeof(struct
zlib_state));

My debugging printf's seem to indicate that (in the same file):

static void zlib_stateful_finish(COMP_CTX *ctx)

is called correctly, but

static void zlib_stateful_free_ex_data(...)

which is supposed to free the zlib_state allocation is never called.

The zlib_stateful_free_ex_data function seems to be called when I use openssl s_server instead of Apache as the SSL server. I am therefore not completely sure whether the root of this bug is in apache or openssl.

BTW, bug #186339 looks like it is the same issue.

CVE References

Changed in apache2:
importance: Undecided → High

Hi,

I've got the same bug. I've tried disabling mod_ssl without any success, the memory leak is still happening... Do you have an idea ? Maybe it's not/not only SSL after all ?

Léobaillard (leobaillard) wrote :
Download full text (4.4 KiB)

BTW, here's some debug information from /var/log/kern.log :

May 3 19:51:22 yoda kernel: [10632.509889] Out of memory: kill process 9047 (apache2) score 19040 or a child
May 3 19:51:22 yoda kernel: [10632.510045] Killed process 9047 (apache2)
May 3 19:51:22 yoda kernel: [10634.310259] named invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
May 3 19:51:22 yoda kernel: [10634.310291] Pid: 4539, comm: named Not tainted 2.6.24-16-server #1
May 3 19:51:22 yoda kernel: [10634.310370] [oom_kill_process+0x10a/0x120] oom_kill_process+0x10a/0x120
May 3 19:51:22 yoda kernel: [10634.310460] [out_of_memory+0x167/0x1a0] out_of_memory+0x167/0x1a0
May 3 19:51:22 yoda kernel: [10634.310510] [agpgart:__alloc_pages+0x34c/0x380] __alloc_pages+0x34c/0x380
May 3 19:51:22 yoda kernel: [10634.310573] [filemap_fault+0x10a/0x420] filemap_fault+0x10a/0x420
May 3 19:51:22 yoda kernel: [10634.310638] [__do_fault+0x83/0x4c0] __do_fault+0x83/0x4c0
May 3 19:51:22 yoda kernel: [10634.310747] [handle_mm_fault+0x21b/0xb80] handle_mm_fault+0x21b/0xb80
May 3 19:51:22 yoda kernel: [10634.310778] [<c012b0c0>] default_wake_function+0x0/0x10
May 3 19:51:22 yoda kernel: [10634.310875] [sched_clock+0x13/0x30] sched_clock+0x13/0x30
May 3 19:51:22 yoda kernel: [10634.310914] [snd_pcm:getnstimeofday+0x34/0x9790] getnstimeofday+0x34/0xf0
May 3 19:51:22 yoda kernel: [10634.310966] [snd_pcm:ktime_get_ts+0x1e/0x4f0] ktime_get_ts+0x1e/0x60
May 3 19:51:22 yoda kernel: [10634.310996] [do_page_fault+0x143/0x900] do_page_fault+0x143/0x900
May 3 19:51:22 yoda kernel: [10634.311025] [sys_futex+0x98/0x120] sys_futex+0x98/0x120
May 3 19:51:22 yoda kernel: [10634.311054] [snd_seq:do_gettimeofday+0x34/0xa450] do_gettimeofday+0x34/0xe0
May 3 19:51:22 yoda kernel: [10634.311090] [sys_gettimeofday+0x28/0x80] sys_gettimeofday+0x28/0x80
May 3 19:51:22 yoda kernel: [10634.311118] [do_page_fault+0x0/0x900] do_page_fault+0x0/0x900
May 3 19:51:22 yoda kernel: [10634.311134] [error_code+0x72/0x78] error_code+0x72/0x78
May 3 19:51:22 yoda kernel: [10634.311173] [unix_create1+0x50/0x120] unix_create1+0x50/0x120
May 3 19:51:22 yoda kernel: [10634.311229] =======================
May 3 19:51:22 yoda kernel: [10634.311236] Mem-info:
May 3 19:51:22 yoda kernel: [10634.311243] DMA per-cpu:
May 3 19:51:22 yoda kernel: [10634.311253] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
May 3 19:51:22 yoda kernel: [10634.311264] Normal per-cpu:
May 3 19:51:22 yoda kernel: [10634.311273] CPU 0: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 43
May 3 19:51:22 yoda kernel: [10634.311291] Active:87819 inactive:295 dirty:0 writeback:0 unstable:0
May 3 19:51:22 yoda kernel: [10634.311297] free:976 slab:3176 mapped:2 pagetables:1570 bounce:0
May 3 19:51:22 yoda kernel: [10634.311314] DMA free:1552kB min:104kB low:128kB high:156kB active:10208kB inactive:16kB present:16256kB pages_scanned:127731 all_unreclaimable? yes
May 3 19:51:22 yoda kernel: [10634.311327] lowmem_reserve[]: 0 364 364 364
May 3 19:51:22 yoda kernel: [10634.311347] Normal free:2352kB min:2384kB low:2980kB high:3576kB active:341068kB inactive:1164kB p...

Read more...

Chuck Short (zulcss) wrote :

Hello,

Can you provide your ssl configuration so I can reproduce this?
Please include the output of dpkg -l | grep apache.

Thanks
chuck

Changed in apache2:
status: New → Incomplete
Dustin Kirkland  (kirkland) wrote :

I've reproduced this in a Hardy VM with 128MB of memory.

Beyond the default server install, I installed:
apache2-mpm-worker ssl-cert

I enabled ssl with a2enmod.

I created a very simple ssl config for apache and enabled it with a2ensite.

Then, from the VM host, I ran:
time ab -n 100000 -c 20 -f tls1 https://192.168.122.74:443/

It locked up the VM. I could see apache2 consuming all virtual memory, and the load pegged at 20.87.

Interestingly, I couldn't shutdown or ctrl-alt-delete the VM to gracefully reset it to a sane state.

:-Dustin

Changed in apache2:
assignee: nobody → kirkland
status: Incomplete → Confirmed
Dustin Kirkland  (kirkland) wrote :

Chuck-

SSL configuration:
NameVirtualHost *:443
<VirtualHost *:443>
 SSLEngine On
 SSLCertificateFile /etc/ssl/certs/ssl.pem
 DocumentRoot /var/www
 <Directory />
  Options Indexes
 </Directory>
</VirtualHost>

:-Dustin

Léobaillard (leobaillard) wrote :

The only way to reboot the machine safely is to use the Magic SysRq keys. For instance, I used : AltGr + SysRq + s, AltGr + SysRq + u and AltGr + SysRq + b which syncs the mounted file systems, remounts all the mounted file systems in Read Only mode and finally reboots the computer.

But I still can't prevent Apache from crashing the system even with SSL disabled.

Dustin Kirkland  (kirkland) wrote :

Léobaillard-

I cannot reproduce this with SSL disabled.

Can you give very detailed steps for reproducing this problem without SSL?

:-Dustin

Dustin Kirkland  (kirkland) wrote :

FWIW, I have also reproduced this problem in Gutsy as well, which uses openssl-0.9.8e-5ubuntu3.

However, Dapper-Feisty do not exhibit this problem. Feisty uses openssl-0.9.8c-4ubuntu0.2.

It seems that the regression was introduced sometime between openssl-0.9.8c and 0.9.8e.

:-Dustin

Changed in openssl:
assignee: nobody → kirkland
importance: Undecided → High
status: New → Triaged
Dustin Kirkland  (kirkland) wrote :

Upstream bug filed with Apache:
https://issues.apache.org/bugzilla/show_bug.cgi?id=44975

They suggested applying this patch:
http://svn.apache.org/viewvc?view=rev&revision=654119

Test packages available in my PPA.

This patch fixes half of the problem. It solves the out-of-memory errors, however, SSL connections are simply dropped when the server is flooded. Note that Feisty's apache2+mod_ssl is able to handle the flood on both the client and server without errors.

:-Dustin

Dustin Kirkland  (kirkland) wrote :

Please disregard my last comment. Those packages did not actually have the patch applied due to some nuances with dpatch in the build. As a result those tests are invalid.

I have since cleanly applied the patch, built test packages, and tested.

These test have succeeded, with 100,000 SSL request being issued and completed successfully, without loss of SSL connection, and without out-of-memory errors.

I'm submitting the attached debdiff as the fix for this issue, and Chuck will be bringing for the SRU. The patch is the one suggested by Apache ( http://svn.apache.org/viewvc?view=rev&revision=654119 )

:-Dustin

Dustin Kirkland  (kirkland) wrote :

The bug is in Apache, not OpenSSL.

Changed in openssl:
assignee: kirkland → nobody
importance: High → Undecided
status: Triaged → Invalid
Dustin Kirkland  (kirkland) wrote :

Patch submitted.

Changed in apache2:
milestone: none → ubuntu-8.04.1
status: Confirmed → In Progress
Chuck Short (zulcss) wrote :

Depending on the system load apache with apache-mpm-worker and mod_ssl enabled will cause ssl to run out of memory and crash. The following patch resolves this issue. It will be needed to be ported to intrepid since it is also vulnerable to this condition.

Steps to reproduce: (TEST CASE)

1. Install apache-mpm-worker and ssl-cert
2. Confgure the SSL cert according to https://help.ubuntu.com/8.04/serverguide/C/certificates-and-security.html.
3. Use the following config in your /etc/apache2/sites-enabled/default.

NameVirtualHost *:443
<VirtualHost *:443>
 SSLEngine On
 SSLCertificateFile /etc/ssl/certs/ssl.pem
 DocumentRoot /var/www
 <Directory />
  Options Indexes
 </Directory>
</VirtualHost>

4. Run the following command:

ab -n 100000 -c 20 -f tls1 https://<ip address>:443/

You should get OOM errors in a couple of minutes of running the test.

If you have any questions let me know.

Regards
chuck

Changed in openssl:
status: New → Invalid
Changed in apache2:
importance: Undecided → High
Stefan Fritsch (sf-sfritsch) wrote :

2.2.8-4 with the patch has just been uploaded to Debian

Martin Pitt (pitti) on 2008-05-14
Changed in apache2:
milestone: none → ubuntu-8.04.1
milestone: ubuntu-8.04.1 → none
Martin Pitt (pitti) wrote :

Accepted into -proposed, please test and give feedback here

Changed in apache2:
milestone: ubuntu-8.04.1 → none
status: New → Fix Committed
Martin Pitt (pitti) wrote :

Since Intrepid is a sync now, I take the intrepid task.

Changed in apache2:
assignee: kirkland → pitti
Steve Langasek (vorlon) on 2008-05-14
Changed in apache2:
milestone: none → ubuntu-8.04.1
Martin Pitt (pitti) wrote :

Verification step 1: I set up apache2 in hardy final and successfully reproduced the OOM condition. I will test the updated package with that setup, too.

Changed in apache2:
assignee: nobody → pitti
Martin Pitt (pitti) wrote :

I upgraded to the hardy-proposed version and ran the hammering benchmark again. Memory was constantly acquired and released now and never got exhausted.

I did both tests on a 256 MB VM.

Martin Pitt (pitti) wrote :

I synced the fixed Debian version to intrepid.

Changed in apache2:
status: In Progress → Fix Released
Martin Pitt (pitti) wrote :

Tests done, unassigning.

Changed in apache2:
assignee: pitti → nobody
Martin Pitt (pitti) wrote :

Copied to hardy-updates.

Changed in apache2:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.