Bug #230878 “Apache 2 produces an OOM after 4 hours using” : Bugs : apache2 package : Ubuntu

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-16:

#1

I've also added some restrictions for apache which seems not to change the problem :

<IfModule prefork.c>
        MaxClients 40
        MaxSpareServers 8
        MaxRequestsPerChild 2000
</IfModule>

RLimitNPROC 10
RLimitMEM 67108864
RLimitCPU 30

Revision history for this message

Dustin Kirkland  (kirkland) wrote on 2008-05-16:

#2

Hi Léobaillard-

I at first marked this a duplicate of Bug #224945, but I see now that you're not dealing with SSL. In which case I'm interested in reproducing this bug (and I un-marked it duplicate).

Would it be possible for you to determine an Apache bench (ab) command that you can run to reproduce this problem reliably, and quicker than within 4 hours?

Thanks,
:-Dustin

Revision history for this message

Dustin Kirkland  (kirkland) wrote on 2008-05-16:

#3

Marking incomplete until we have full instructions for reproducing this bug (ideally using ab).

Please change back to "new" once you've provided these.

Thanks!
:-Dustin

Changed in apache2:
assignee:	nobody → kirkland
status:	New → Incomplete

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-16:

#4

Ok, I'll do this as soon as I go back home and post you the results.

Revision history for this message

Stefan Fritsch (sf-sfritsch) wrote on 2008-05-17:

#5

It would also be interesting to know what apache modules you have enabled and how the output of top looks just before the OOM situation (i.e. are there many large apache2 processes or only a few very large ones?).

You could also try tweaking "memory_limit" in your php.ini.

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-19:

#6

htop-apache-loading.png Edit (410.8 KiB, image/png)

I ran some ab tests (it's my first time so maybe the settings were too low, you'll tell me) but it didn't produce an OOM. I couldn't get a fresh htop when the server was loading but I've taken a screenshot just before the SSH has frozen.

Here are the ab results

$ ab -c 50 -n 50000 http://<server>/

Benchmarking <server> (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Finished 50000 requests

Server Software: Apache
Server Hostname: <server>
Server Port: 80

Document Path: /
Document Length: 224 bytes

Concurrency Level: 50
Time taken for tests: 172.678631 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Non-2xx responses: 50005
Total transferred: 21802180 bytes
HTML transferred: 11201120 bytes
Requests per second: 289.56 [#/sec] (mean)
Time per request: 172.679 [ms] (mean)
Time per request: 3.454 [ms] (mean, across all concurrent requests)
Transfer rate: 123.30 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 58 414.4 17 21013
Processing: 2 82 1320.6 18 90756
Waiting: 1 55 1264.2 17 90741
Total: 5 140 1397.8 36 90774

Percentage of the requests served within a certain time (ms)
  50% 36
  66% 36
  75% 38
  80% 39
  90% 40
  95% 44
  98% 1887
  99% 3036
100% 90774 (longest request)

$ ab -c 500 -n 50000 http://<server>/

Benchmarking <server> (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Finished 50000 requests

Server Software: Apache
Server Hostname: <server>
Server Port: 80

Document Path: /
Document Length: 224 bytes

Concurrency Level: 500
Time taken for tests: 173.445502 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Non-2xx responses: 50006
Total transferred: 21802616 bytes
HTML transferred: 11201344 bytes
Requests per second: 288.27 [#/sec] (mean)
Time per request: 1734.455 [ms] (mean)
Time per request: 3.469 [ms] (mean, across all concurrent requests)
Transfer rate: 122.75 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 82 920.7 18 93023
Processing: 2 142 2040.2 20 146381
Waiting: 1 92 1933.3 20 146363
Total: 3 224 2265.0 39 146408

Percentage of the requests served within a certain time (ms)
  50% 39
  66% 45
  75% 58
  80% 63
  90% 77
  95% 85
  98% 3036
  99% 4119
100% 146408 (longest request)

I ran some ab tests (it's my first time so maybe the settings were too low, you'll tell me) but it didn't produce an OOM. I couldn't get a fresh htop when the server was loading but I've taken a screenshot just before the SSH has frozen.

Here are the ab results

$ ab -c 50 -n 50000 http://<server>/

Benchmarking <server> (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Finished 50000 requests

Server Software:        Apache
Server Hostname:        <server>
Server Port:            80

Document Path:          /
Document Length:        224 bytes

Concurrency Level:      50
Time taken for tests:   172.678631 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Non-2xx responses:      50005
Total transferred:      21802180 bytes
HTML transferred:       11201120 bytes
Requests per second:    289.56 [#/sec] (mean)
Time per request:       172.679 [ms] (mean)
Time per request:       3.454 [ms] (mean, across all concurrent requests)
Transfer rate:          123.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1   58 414.4     17   21013
Processing:     2   82 1320.6     18   90756
Waiting:        1   55 1264.2     17   90741
Total:          5  140 1397.8     36   90774

Percentage of the requests served within a certain time (ms)
  50%     36
  66%     36
  75%     38
  80%     39
  90%     40
  95%     44
  98%   1887
  99%   3036
 100%  90774 (longest request)

$ ab -c 500 -n 50000 http://<server>/
 
 Benchmarking <server> (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Finished 50000 requests

Server Software:        Apache
Server Hostname:        <server>
Server Port:            80

Document Path:          /
Document Length:        224 bytes

Concurrency Level:      500
Time taken for tests:   173.445502 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Non-2xx responses:      50006
Total transferred:      21802616 bytes
HTML transferred:       11201344 bytes
Requests per second:    288.27 [#/sec] (mean)
Time per request:       1734.455 [ms] (mean)
Time per request:       3.469 [ms] (mean, across all concurrent requests)
Transfer rate:          122.75 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1   82 920.7     18   93023
Processing:     2  142 2040.2     20  146381
Waiting:        1   92 1933.3     20  146363
Total:          3  224 2265.0     39  146408

Percentage of the requests served within a certain time (ms)
  50%     39
  66%     45
  75%     58
  80%     63
  90%     77
  95%     85
  98%   3036
  99%   4119
 100%  146408 (longest request)

Léobaillard (leobaillard) on 2008-05-19

Changed in apache2:
status:	Incomplete → New

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-19:

#7

I've just seen that the responses werent 2xx, I got a 404, I've fixed this and I'll run more ab tests with higher values.

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-20:

#8

htop-apache-loading-overload.png Edit (419.5 KiB, image/png)

Download full text (5.9 KiB)

I've launched the same ab test but the machine has frozen before I could see the results, nevertheless, when it was frozen, it didn't produce an OOM, the OOM has come later but not within the 4 hours limit.

Here is the initial OOM of apache2 and it seems to have taken down other processes before locking completely the machine :

May 20 01:30:00 yoda kernel: [347664.832222] apache2 invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
May 20 01:30:00 yoda kernel: [347664.832229] Pid: 6899, comm: apache2 Not tainted 2.6.24-16-server #1
May 20 01:30:00 yoda kernel: [347664.832246] [oom_kill_process+0x10a/0x120] oom_kill_process+0x10a/0x120
May 20 01:30:00 yoda kernel: [347664.832274] [out_of_memory+0x167/0x1a0] out_of_memory+0x167/0x1a0
May 20 01:30:01 yoda kernel: [347664.832299] [agpgart:__alloc_pages+0x34c/0x380] __alloc_pages+0x34c/0x380
May 20 01:30:03 yoda kernel: [347664.832334] [__do_page_cache_readahead+0x11d/0x240] __do_page_cache_readahead+0x11d/0x240
May 20 01:30:03 yoda kernel: [347664.832337] [sync_page+0x0/0x40] sync_page+0x0/0x40
May 20 01:30:03 yoda kernel: [347664.832368] [do_page_cache_readahead+0x4c/0x70] do_page_cache_readahead+0x4c/0x70
May 20 01:30:03 yoda kernel: [347664.832380] [filemap_fault+0x2f2/0x420] filemap_fault+0x2f2/0x420
May 20 01:30:05 yoda kernel: [347664.832387] [kmap_atomic_prot+0xfc/0x130] kmap_atomic_prot+0xfc/0x130
May 20 01:30:06 yoda kernel: [347664.832417] [__do_fault+0x83/0x4c0] __do_fault+0x83/0x4c0
May 20 01:30:06 yoda kernel: [347664.832445] [kmap_atomic_prot+0xfc/0x130] kmap_atomic_prot+0xfc/0x130
May 20 01:30:06 yoda kernel: [347664.832474] [handle_mm_fault+0x21b/0xb80] handle_mm_fault+0x21b/0xb80
May 20 01:30:06 yoda kernel: [347664.832543] [set_process_cpu_timer+0xa6/0xc0] set_process_cpu_timer+0xa6/0xc0
May 20 01:30:31 yoda kernel: [347664.832574] [do_page_fault+0x143/0x900] do_page_fault+0x143/0x900
May 20 01:30:32 yoda kernel: [347664.832580] [do_sigaction+0x65/0x170] do_sigaction+0x65/0x170
May 20 01:30:33 yoda kernel: [347664.832598] [recalc_sigpending+0xb/0x40] recalc_sigpending+0xb/0x40
May 20 01:30:33 yoda kernel: [347664.832609] [sys_rt_sigprocmask+0xed/0x110] sys_rt_sigprocmask+0xed/0x110
May 20 01:30:33 yoda kernel: [347664.832620] [do_page_fault+0x0/0x900] do_page_fault+0x0/0x900
May 20 01:30:33 yoda kernel: [347664.832624] [error_code+0x72/0x78] error_code+0x72/0x78
May 20 01:30:33 yoda kernel: [347664.832662] =======================
May 20 01:30:33 yoda kernel: [347664.832663] Mem-info:
May 20 01:30:33 yoda kernel: [347664.832665] DMA per-cpu:
May 20 01:30:33 yoda kernel: [347664.832667] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
May 20 01:30:33 yoda kernel: [347664.832669] Normal per-cpu:
May 20 01:30:33 yoda kernel: [347664.832670] CPU 0: Hot: hi: 186, btch: 31 usd: 177 Cold: hi: 62, btch: 15 usd: 58
May 20 01:30:33 yoda kernel: [347664.832672] HighMem per-cpu:
May 20 01:30:33 yoda kernel: [347664.832674] CPU 0: Hot: hi: 42, btch: 7 usd: 3 Cold: hi: 14, btch: 3 usd: 2
May 20 01:30:33 yoda kernel: [347664.832677] Active:195460 inactive:49533 dirty:0 writeback:0 unstable:0
May 20 01:30:...

I've launched the same ab test but the machine has frozen before I could see the results, nevertheless, when it was frozen, it didn't produce an OOM, the OOM has come later but not within the 4 hours limit.

Here is the initial OOM of apache2 and it seems to have taken down other processes before locking completely the machine :

May 20 01:30:00 yoda kernel: [347664.832222] apache2 invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
May 20 01:30:00 yoda kernel: [347664.832229] Pid: 6899, comm: apache2 Not tainted 2.6.24-16-server #1
May 20 01:30:00 yoda kernel: [347664.832246]  [oom_kill_process+0x10a/0x120] oom_kill_process+0x10a/0x120
May 20 01:30:00 yoda kernel: [347664.832274]  [out_of_memory+0x167/0x1a0] out_of_memory+0x167/0x1a0
May 20 01:30:01 yoda kernel: [347664.832299]  [agpgart:__alloc_pages+0x34c/0x380] __alloc_pages+0x34c/0x380
May 20 01:30:03 yoda kernel: [347664.832334]  [__do_page_cache_readahead+0x11d/0x240] __do_page_cache_readahead+0x11d/0x240
May 20 01:30:03 yoda kernel: [347664.832337]  [sync_page+0x0/0x40] sync_page+0x0/0x40
May 20 01:30:03 yoda kernel: [347664.832368]  [do_page_cache_readahead+0x4c/0x70] do_page_cache_readahead+0x4c/0x70
May 20 01:30:03 yoda kernel: [347664.832380]  [filemap_fault+0x2f2/0x420] filemap_fault+0x2f2/0x420
May 20 01:30:05 yoda kernel: [347664.832387]  [kmap_atomic_prot+0xfc/0x130] kmap_atomic_prot+0xfc/0x130
May 20 01:30:06 yoda kernel: [347664.832417]  [__do_fault+0x83/0x4c0] __do_fault+0x83/0x4c0
May 20 01:30:06 yoda kernel: [347664.832445]  [kmap_atomic_prot+0xfc/0x130] kmap_atomic_prot+0xfc/0x130
May 20 01:30:06 yoda kernel: [347664.832474]  [handle_mm_fault+0x21b/0xb80] handle_mm_fault+0x21b/0xb80
May 20 01:30:06 yoda kernel: [347664.832543]  [set_process_cpu_timer+0xa6/0xc0] set_process_cpu_timer+0xa6/0xc0
May 20 01:30:31 yoda kernel: [347664.832574]  [do_page_fault+0x143/0x900] do_page_fault+0x143/0x900
May 20 01:30:32 yoda kernel: [347664.832580]  [do_sigaction+0x65/0x170] do_sigaction+0x65/0x170
May 20 01:30:33 yoda kernel: [347664.832598]  [recalc_sigpending+0xb/0x40] recalc_sigpending+0xb/0x40
May 20 01:30:33 yoda kernel: [347664.832609]  [sys_rt_sigprocmask+0xed/0x110] sys_rt_sigprocmask+0xed/0x110
May 20 01:30:33 yoda kernel: [347664.832620]  [do_page_fault+0x0/0x900] do_page_fault+0x0/0x900
May 20 01:30:33 yoda kernel: [347664.832624]  [error_code+0x72/0x78] error_code+0x72/0x78
May 20 01:30:33 yoda kernel: [347664.832662]  =======================
May 20 01:30:33 yoda kernel: [347664.832663] Mem-info:
May 20 01:30:33 yoda kernel: [347664.832665] DMA per-cpu:
May 20 01:30:33 yoda kernel: [347664.832667] CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
May 20 01:30:33 yoda kernel: [347664.832669] Normal per-cpu:
May 20 01:30:33 yoda kernel: [347664.832670] CPU    0: Hot: hi:  186, btch:  31 usd: 177   Cold: hi:   62, btch:  15 usd:  58
May 20 01:30:33 yoda kernel: [347664.832672] HighMem per-cpu:
May 20 01:30:33 yoda kernel: [347664.832674] CPU    0: Hot: hi:   42, btch:   7 usd:   3   Cold: hi:   14, btch:   3 usd:   2
May 20 01:30:33 yoda kernel: [347664.832677] Active:195460 inactive:49533 dirty:0 writeback:0 unstable:0
May 20 01:30:33 yoda kernel: [347664.832678]  free:2987 slab:3693 mapped:11 pagetables:2018 bounce:0
May 20 01:30:33 yoda kernel: [347664.832681] DMA free:4056kB min:68kB low:84kB high:100kB active:4588kB inactive:3344kB present:16256kB pages_scanned:13377 all_unreclaimable? yes
May 20 01:30:33 yoda kernel: [347664.832683] lowmem_reserve[]: 0 873 999 999
May 20 01:30:33 yoda kernel: [347664.832686] Normal free:7772kB min:3744kB low:4680kB high:5616kB active:724200kB inactive:126740kB present:894080kB pages_scanned:2424320 all_unreclaimable? yes
May 20 01:30:33 yoda kernel: [347664.832688] lowmem_reserve[]: 0 0 1014 1014
May 20 01:30:33 yoda kernel: [347664.832691] HighMem free:120kB min:128kB low:264kB high:400kB active:53052kB inactive:68048kB present:129796kB pages_scanned:604459 all_unreclaimable? yes
May 20 01:30:33 yoda kernel: [347664.832694] lowmem_reserve[]: 0 0 0 0
May 20 01:30:33 yoda kernel: [347664.832696] DMA: 2*4kB 2*8kB 0*16kB 0*32kB 3*64kB 0*128kB 3*256kB 2*512kB 0*1024kB 1*2048kB 0*4096kB = 4056kB
May 20 01:30:33 yoda kernel: [347664.832701] Normal: 485*4kB 1*8kB 8*16kB 4*32kB 3*64kB 6*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 7772kB
May 20 01:30:33 yoda kernel: [347664.832706] HighMem: 2*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 120kB
May 20 01:30:33 yoda kernel: [347664.832712] Swap cache: add 253027, delete 253022, find 609448/615602, race 0+12
May 20 01:30:33 yoda kernel: [347664.832714] Free swap  = 0kB
May 20 01:30:33 yoda kernel: [347664.832715] Total swap = 779144kB
May 20 01:30:33 yoda kernel: [347664.832716] Free swap:            0kB
May 20 01:30:33 yoda kernel: [347664.836648] 262080 pages of RAM
May 20 01:30:33 yoda kernel: [347664.836650] 32704 pages of HIGHMEM
May 20 01:30:33 yoda kernel: [347664.836651] 3318 reserved pages
May 20 01:30:33 yoda kernel: [347664.836652] 12389 pages shared
May 20 01:30:33 yoda kernel: [347664.836653] 5 pages swap cached
May 20 01:30:33 yoda kernel: [347664.836654] 0 pages dirty
May 20 01:30:33 yoda kernel: [347664.836655] 0 pages writeback
May 20 01:30:33 yoda kernel: [347664.836656] 11 pages mapped
May 20 01:30:33 yoda kernel: [347664.836657] 3693 pages slab
May 20 01:30:33 yoda kernel: [347664.836658] 2018 pages pagetables

We can see here that it is 1:30AM, I've launched the last ab test at approximately 00:10, so it's 1h20 after... This problem is getting very weird... I'm so desperate not to find the solution because it's becoming urgent. Indeed I'm hosting about 100 websites and I'm forced to launch an other httpd to display an apology message...

I join to this message the last screenshot of htop before the ssh connection got stuck which was taken at 00:12.

Good luck in finding out what the problem is, and thanks a lot, I really need you :)

Revision history for this message

Chuck Short (zulcss) wrote on 2008-05-20:

#9

Léobaillard,

Can you test the apache that is in my ppa archive?

http://launchpad.net/~zulcss/+archive

It should be available shortly.

Thanks
chuck

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-20:

#10

Ok, no problem, but what are the differences with my version ? Were do you think the problem is ? I'll try this this afternoon when I come back home.

Revision history for this message

Dustin Kirkland  (kirkland) wrote on 2008-05-20:

#11

Chuck has confirmed this, and triaged it to Apache2. He currently has a fix in progress in his PPA.

:-Dustin

Changed in apache2:
status:	New → In Progress
assignee:	kirkland → zulcss

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-20:

#12

Great ! I install it right now !

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-20:

#13

Unfortunately, it didn't changed anything. The OOM has occured about 20 minutes ago... But the first OOM I saw on the server's screen was triggered by mysqld, usually it's apache and then other processes but this time, it was mysqld... I don't know why but I'm nearly sure that apache is the bad one because, as I already said it in my first message, when apache isn't started, the OOM doesn't occur.

Do you have an idea ?

Revision history for this message

Chuck Short (zulcss) wrote on 2008-05-21:

#14

Well if mysql is causing you an OOM error then have you tried considering running a memory test on your machine then?

chuck

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-21:

#15

Memory test ? If you mean by this checking th integrity of the physical memory, I don't think this is where the problem is because I tryied to change my machine keeping the hard drives and the data and it didn't changed anything... :( So maybe it's related with apache but not apache itself ? Maybe it's my configuration ? Maybe it's php ? I don't know where to start, I don't have any clue appart from the fact that it's happening only when apache is running... :( I can run a memory test, just to see, but I don't think it'll be relevant. Other ideas / tracks ? Again, thank you so much for the interest you give to my problem, I'm very greatful.

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-21:

#16

Maybe I can trace Apache to see where it's going wrong ? What do you think ?

Revision history for this message

Chuck Short (zulcss) wrote on 2008-05-21:

#17

Can you please add your apache configuration files, your php.ini files and anything relevant in your apache2 error logs?

Thanks
chuck

Changed in apache2:
status:	In Progress → Incomplete

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-25:

#18

apache2.conf Edit (11.6 KiB, text/html)

Hi,

Here's my apache2.conf. I didn't notice anything in the apache error logs...

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-25:

#19

php.ini Edit (39.8 KiB, text/html)

And here's my php.ini file.

Changed in apache2:
status:	Incomplete → New

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-25:

#20

I also have a bunch of these :

127.0.0.1 - - [24/May/2008:09:05:59 +0200] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"

in my access.log.

Revision history for this message

Chuck Short (zulcss) wrote on 2008-05-27:

#21

What kernel are you using? This seems to be more like a hardware issue than anything else.

Thanks
chuck

Revision history for this message

Léobaillard (leobaillard) wrote on 2008-05-27:

#22

My kernel is : 2.6.24-16-server. Do you think that a kernel update would do the trick ?

Revision history for this message

Chuck Short (zulcss) wrote on 2008-05-27:

#23

Hi,

I really dont think its a kernel problem either basically you have a process on your machine that is chewing up all of your memory and you really have to nail it down to see why that is. The reason why you are getting OOM because its not that smart and it killed the first. You might want to consider try a different version of apache (other than mpm-worker) as well. From what you have given us so far I cant really tell which is eating up all of your memory.

Regards
chuck

Changed in apache2:
status:	New → Triaged

Chuck Short (zulcss) on 2009-10-06

Changed in apache2 (Ubuntu):
importance:	Undecided → Low

Revision history for this message

Dave Walker (davewalker) wrote on 2010-05-17:

#24

Marking incomplete as waiting on reporter to provide some further information to help triage.

Changed in apache2 (Ubuntu):
status:	Triaged → Incomplete

Revision history for this message

Chuck Short (zulcss) wrote on 2010-08-25:

#25

We'd like to figure out what's causing this bug for you, but we haven't heard back from you in a while. Could you please provide the requested information? Thanks!

Revision history for this message

Dave Walker (davewalker) wrote on 2011-03-07:

#26

Marking Invalid as a reasonable assumption that it might have been hardware memory issues. If this is not the case, please provide further information as requested.

Thanks again!

Changed in apache2 (Ubuntu):
status:	Incomplete → Opinion
status:	Opinion → Invalid

Revision history for this message

GrayMatter (cgray) wrote on 2011-09-22:

#27

Download full text (7.4 KiB)

We are experiencing the same issue. Fresh install of Ubuntu 10.10LTS with default apache2, mysql, php. I can say for certain that it's not a hardware issue. This is a virtual server running on an ESX host. Let me know if there are any other details I can pull to help with this issue. This is a production server and it crashes with this OOM about once a week.

Apache2 Loaded Modules:
core mod_log_config mod_logio prefork http_core mod_so mod_alias mod_auth_basic mod_authn_file mod_authz_default mod_authz_groupfile mod_authz_host mod_authz_user mod_autoindex mod_cgi mod_deflate mod_dir mod_env mod_mime mod_negotiation mod_php5 mod_reqtimeout mod_rewrite mod_setenvif mod_status

PHP Version: 5.3.3-1ubuntu9.5

The default also came with Suhosin Patch 0.9.9.1.

root@public-lamp:/var/log# mysql --version
mysql Ver 14.14 Distrib 5.1.49, for debian-linux-gnu (x86_64) using readline 6.1

2.6.35-30-server #59-Ubuntu SMP 64-bit

Sep 21 12:31:32 public-lamp kernel: [1707807.719708] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Sep 21 12:31:32 public-lamp kernel: [1707807.797392] apache2 cpuset=/ mems_allowed=0
Sep 21 12:31:32 public-lamp kernel: [1707807.824987] Pid: 16345, comm: apache2 Not tainted 2.6.35-30-server #56-Ubuntu
Sep 21 12:31:32 public-lamp kernel: [1707807.824991] Call Trace:
Sep 21 12:31:32 public-lamp kernel: [1707807.877827] [<ffffffff81105331>] dump_header+0x81/0xc0
Sep 21 12:31:32 public-lamp kernel: [1707807.995049] [<ffffffff811053f1>] oom_kill_process+0x81/0x180
Sep 21 12:31:32 public-lamp kernel: [1707807.995053] [<ffffffff81105928>] __out_of_memory+0x58/0xd0
Sep 21 12:31:32 public-lamp kernel: [1707807.995057] [<ffffffff81105a26>] out_of_memory+0x86/0x1c0
Sep 21 12:31:32 public-lamp kernel: [1707807.995061] [<ffffffff8110948e>] __alloc_pages_slowpath+0x58e/0x5a0
Sep 21 12:31:32 public-lamp kernel: [1707807.995065] [<ffffffff8110960c>] __alloc_pages_nodemask+0x16c/0x1d0
Sep 21 12:31:32 public-lamp kernel: [1707807.995075] [<ffffffff8113b9ca>] alloc_pages_current+0x9a/0x100
Sep 21 12:31:32 public-lamp kernel: [1707807.995079] [<ffffffff81102a37>] __page_cache_alloc+0x87/0x90
Sep 21 12:31:32 public-lamp kernel: [1707807.995090] [<ffffffff8110cc09>] __do_page_cache_readahead+0xc9/0x210
Sep 21 12:31:32 public-lamp kernel: [1707807.995095] [<ffffffff8110cd71>] ra_submit+0x21/0x30
Sep 21 12:31:32 public-lamp kernel: [1707807.995098] [<ffffffff8110d125>] ondemand_readahead+0x115/0x240
Sep 21 12:31:32 public-lamp kernel: [1707808.004712] [<ffffffff81049219>] ? __wake_up_common+0x59/0x90
Sep 21 12:31:32 public-lamp kernel: [1707808.005747] [<ffffffff8110d351>] page_cache_sync_readahead+0x31/0x50
Sep 21 12:31:32 public-lamp kernel: [1707808.005751] [<ffffffff81104259>] filemap_fault+0x419/0x450
Sep 21 12:31:32 public-lamp kernel: [1707808.005756] [<ffffffff8111f304>] __do_fault+0x54/0x560
Sep 21 12:31:32 public-lamp kernel: [1707808.005760] [<ffffffff81122b69>] handle_mm_fault+0x1b9/0x440
Sep 21 12:31:32 public-lamp kernel: [1707808.012723] [<ffffffff81170d20>] ? mntput_no_expire+0x30/0x110
Sep 21 12:31:32 public-lamp kernel: [1707808.022743] [<ffffffff815a6a75>] do_page_fault+0x125/0x350
Sep 21 ...

We are experiencing the same issue.  Fresh install of Ubuntu 10.10LTS with default apache2, mysql, php. I can say for certain that it's not a hardware issue.  This is a virtual server running on an ESX host.  Let me know if there are any other details I can pull to help with this issue.  This is a production server and it crashes with this OOM about once a week.

Apache2 Loaded Modules:
core mod_log_config mod_logio prefork http_core mod_so mod_alias mod_auth_basic mod_authn_file mod_authz_default mod_authz_groupfile mod_authz_host mod_authz_user mod_autoindex mod_cgi mod_deflate mod_dir mod_env mod_mime mod_negotiation mod_php5 mod_reqtimeout mod_rewrite mod_setenvif mod_status

PHP Version: 5.3.3-1ubuntu9.5

The default also came with Suhosin Patch 0.9.9.1.

root@public-lamp:/var/log# mysql --version
mysql  Ver 14.14 Distrib 5.1.49, for debian-linux-gnu (x86_64) using readline 6.1

2.6.35-30-server #59-Ubuntu SMP 64-bit

Sep 21 12:31:32 public-lamp kernel: [1707807.719708] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Sep 21 12:31:32 public-lamp kernel: [1707807.797392] apache2 cpuset=/ mems_allowed=0
Sep 21 12:31:32 public-lamp kernel: [1707807.824987] Pid: 16345, comm: apache2 Not tainted 2.6.35-30-server #56-Ubuntu
Sep 21 12:31:32 public-lamp kernel: [1707807.824991] Call Trace:
Sep 21 12:31:32 public-lamp kernel: [1707807.877827]  [<ffffffff81105331>] dump_header+0x81/0xc0
Sep 21 12:31:32 public-lamp kernel: [1707807.995049]  [<ffffffff811053f1>] oom_kill_process+0x81/0x180
Sep 21 12:31:32 public-lamp kernel: [1707807.995053]  [<ffffffff81105928>] __out_of_memory+0x58/0xd0
Sep 21 12:31:32 public-lamp kernel: [1707807.995057]  [<ffffffff81105a26>] out_of_memory+0x86/0x1c0
Sep 21 12:31:32 public-lamp kernel: [1707807.995061]  [<ffffffff8110948e>] __alloc_pages_slowpath+0x58e/0x5a0
Sep 21 12:31:32 public-lamp kernel: [1707807.995065]  [<ffffffff8110960c>] __alloc_pages_nodemask+0x16c/0x1d0
Sep 21 12:31:32 public-lamp kernel: [1707807.995075]  [<ffffffff8113b9ca>] alloc_pages_current+0x9a/0x100
Sep 21 12:31:32 public-lamp kernel: [1707807.995079]  [<ffffffff81102a37>] __page_cache_alloc+0x87/0x90
Sep 21 12:31:32 public-lamp kernel: [1707807.995090]  [<ffffffff8110cc09>] __do_page_cache_readahead+0xc9/0x210
Sep 21 12:31:32 public-lamp kernel: [1707807.995095]  [<ffffffff8110cd71>] ra_submit+0x21/0x30
Sep 21 12:31:32 public-lamp kernel: [1707807.995098]  [<ffffffff8110d125>] ondemand_readahead+0x115/0x240
Sep 21 12:31:32 public-lamp kernel: [1707808.004712]  [<ffffffff81049219>] ? __wake_up_common+0x59/0x90
Sep 21 12:31:32 public-lamp kernel: [1707808.005747]  [<ffffffff8110d351>] page_cache_sync_readahead+0x31/0x50
Sep 21 12:31:32 public-lamp kernel: [1707808.005751]  [<ffffffff81104259>] filemap_fault+0x419/0x450
Sep 21 12:31:32 public-lamp kernel: [1707808.005756]  [<ffffffff8111f304>] __do_fault+0x54/0x560
Sep 21 12:31:32 public-lamp kernel: [1707808.005760]  [<ffffffff81122b69>] handle_mm_fault+0x1b9/0x440
Sep 21 12:31:32 public-lamp kernel: [1707808.012723]  [<ffffffff81170d20>] ? mntput_no_expire+0x30/0x110
Sep 21 12:31:32 public-lamp kernel: [1707808.022743]  [<ffffffff815a6a75>] do_page_fault+0x125/0x350
Sep 21 12:31:32 public-lamp kernel: [1707808.033163]  [<ffffffff815a35f5>] page_fault+0x25/0x30
Sep 21 12:31:32 public-lamp kernel: [1707808.033168] Mem-Info:
Sep 21 12:31:32 public-lamp kernel: [1707808.033171] Node 0 DMA per-cpu:
Sep 21 12:31:32 public-lamp kernel: [1707808.033174] CPU    0: hi:    0, btch:   1 usd:   0
Sep 21 12:31:32 public-lamp kernel: [1707808.033177] CPU    1: hi:    0, btch:   1 usd:   0
Sep 21 12:31:32 public-lamp kernel: [1707808.033179] Node 0 DMA32 per-cpu:
Sep 21 12:31:32 public-lamp kernel: [1707808.033182] CPU    0: hi:  186, btch:  31 usd:  32
Sep 21 12:31:32 public-lamp kernel: [1707808.033185] CPU    1: hi:  186, btch:  31 usd:   0
Sep 21 12:31:32 public-lamp kernel: [1707808.033187] Node 0 Normal per-cpu:
Sep 21 12:31:32 public-lamp kernel: [1707808.033189] CPU    0: hi:  186, btch:  31 usd:  18
Sep 21 12:31:32 public-lamp kernel: [1707808.033192] CPU    1: hi:  186, btch:  31 usd:   6
Sep 21 12:31:32 public-lamp kernel: [1707808.033246] active_anon:947260 inactive_anon:236889 isolated_anon:0
Sep 21 12:31:32 public-lamp kernel: [1707808.033248]  active_file:334 inactive_file:562 isolated_file:0
Sep 21 12:31:33 public-lamp kernel: [1707808.033249]  unevictable:0 dirty:0 writeback:65 unstable:0
Sep 21 12:31:33 public-lamp kernel: [1707808.033250]  free:9539 slab_reclaimable:2483 slab_unreclaimable:5104
Sep 21 12:31:33 public-lamp kernel: [1707808.033251]  mapped:322 shmem:10 pagetables:12402 bounce:0
Sep 21 12:31:33 public-lamp kernel: [1707808.033254] Node 0 DMA free:15836kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15708kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Sep 21 12:31:33 public-lamp kernel: [1707808.040340] lowmem_reserve[]: 0 3000 6030 6030
Sep 21 12:31:33 public-lamp kernel: [1707808.040346] Node 0 DMA32 free:17168kB min:4936kB low:6168kB high:7404kB active_anon:2399240kB inactive_anon:599996kB active_file:212kB inactive_file:688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:168kB mapped:156kB shmem:4kB slab_reclaimable:652kB slab_unreclaimable:5820kB kernel_stack:1240kB pagetables:25256kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
Sep 21 12:31:33 public-lamp kernel: [1707808.040358] lowmem_reserve[]: 0 0 3030 3030
Sep 21 12:31:33 public-lamp kernel: [1707808.040983] Node 0 Normal free:5152kB min:4984kB low:6228kB high:7476kB active_anon:1389800kB inactive_anon:347560kB active_file:1124kB inactive_file:1708kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3102720kB mlocked:0kB dirty:0kB writeback:92kB mapped:1132kB shmem:36kB slab_reclaimable:9280kB slab_unreclaimable:14580kB kernel_stack:1224kB pagetables:24352kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
Sep 21 12:31:33 public-lamp kernel: [1707808.040996] lowmem_reserve[]: 0 0 0 0
Sep 21 12:31:33 public-lamp kernel: [1707808.041001] Node 0 DMA: 3*4kB 0*8kB 1*16kB 2*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15836kB
Sep 21 12:31:33 public-lamp kernel: [1707808.041021] Node 0 DMA32: 28*4kB 6*8kB 8*16kB 10*32kB 5*64kB 1*128kB 13*256kB 15*512kB 1*1024kB 0*2048kB 1*4096kB = 17184kB
Sep 21 12:31:33 public-lamp kernel: [1707808.041044] Node 0 Normal: 1236*4kB 14*8kB 6*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5152kB
Sep 21 12:31:33 public-lamp kernel: [1707808.041055] 40476 total pagecache pages
Sep 21 12:31:33 public-lamp kernel: [1707808.041057] 39519 pages in swap cache
Sep 21 12:31:33 public-lamp kernel: [1707808.041063] Swap cache stats: add 465967, delete 426448, find 30615/43793
Sep 21 12:31:33 public-lamp kernel: [1707808.041066] Free swap  = 40kB
Sep 21 12:31:33 public-lamp kernel: [1707808.041067] Total swap = 1331196kB
Sep 21 12:31:33 public-lamp kernel: [1707813.575934] 1572848 pages RAM
Sep 21 12:31:33 public-lamp kernel: [1707813.575938] 41073 pages reserved
Sep 21 12:31:33 public-lamp kernel: [1707813.575940] 79247 pages shared
Sep 21 12:31:33 public-lamp kernel: [1707813.575942] 1520465 pages non-shared

Changed in apache2 (Ubuntu):
status:	Invalid → Incomplete

Revision history for this message

Clint Byrum (clint-fewbar) wrote on 2012-08-10:

#28

I notice that both of the reporters are using libapache-mod-php5. It might be that this is a PHP memory leak issue and has nothing to do with apache. OOM killed processes are often not the actual problem.

I don't see anything we can actually act on to reproduce in this bug report. Unless a user is willing to share their PHP configs and possibly code that reproduces the issue, I don't see how we can do anything.

Because of that, I'm going to go ahead and close this issue as Invalid. If anyone has information suggesting that this is still a bug we can respond to and debug in Ubuntu, please feel free to re-open it or to open a new bug from a recent release of Ubuntu.

Changed in apache2 (Ubuntu):
status:	Incomplete → Invalid

Ubuntu
apache2 package

Apache 2 produces an OOM after 4 hours using

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntuapache2 package

Apache 2 produces an OOM after 4 hours using

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
apache2 package