Apache Server Dumpos Frequently - "*** glibc detected *** /usr/sbin/apache2: double free or corruption"

[Allan-9]

Apache Sewrvices crashes several times a day with memory dumps in the error.log file - usually it will restart itself though occasionally it will need me to restart the service manually.

Ubuntu 10.10 maverick (from lsb_release)
Apache/2.2.16 (Ubuntu)
PHP Version 5.3.3-1ubuntu9.5

*** glibc detected *** /usr/sbin/apache2: double free or corruption (fasttop): 0x00007f4fe4028600 ***
======= Backtrace: =========
/lib/libc.so.6(+0x774b6)[0x7f4fe1bfe4b6]
/lib/libc.so.6(cfree+0x73)[0x7f4fe1c04c83]
/usr/lib/apache2/modules/libphp5.so(+0x2d3eb1)[0x7f4fde527eb1]
/usr/lib/apache2/modules/libphp5.so(+0x1f038a)[0x7f4fde44438a]
/usr/lib/apache2/modules/libphp5.so(zend_error+0x60c)[0x7f4fde57a13c]
/lib/libc.so.6(+0x33c20)[0x7f4fe1bbac20]
/lib/libc.so.6(memcpy+0x11)[0x7f4fe1c0daa1]
/usr/lib/apache2/modules/libphp5.so(+0x2d3ed5)[0x7f4fde527ed5]
/usr/lib/apache2/modules/libphp5.so(+0x1f038a)[0x7f4fde44438a]
/usr/lib/apache2/modules/libphp5.so(zend_error+0x60c)[0x7f4fde57a13c]
/usr/lib/apache2/modules/libphp5.so(+0x32fee6)[0x7f4fde583ee6]
/usr/lib/apache2/modules/libphp5.so(zend_parse_parameters+0x95)[0x7f4fde5846a5]
/usr/lib/apache2/modules/libphp5.so(zif_feof+0x2c)[0x7f4fde4b888c]
/usr/lib/apache2/modules/libphp5.so(+0x3770fa)[0x7f4fde5cb0fa]
/usr/lib/apache2/modules/libphp5.so(execute+0x210)[0x7f4fde5a21a0]
/usr/lib/apache2/modules/libphp5.so(zend_execute_scripts+0x15d)[0x7f4fde57999d]
/usr/lib/apache2/modules/libphp5.so(php_execute_script+0x1d8)[0x7f4fde5252a8]
/usr/lib/apache2/modules/libphp5.so(+0x3b359d)[0x7f4fde60759d]
/usr/sbin/apache2(ap_run_handler+0x70)[0x7f4fe2a14310]
/usr/sbin/apache2(ap_invoke_handler+0xce)[0x7f4fe2a17c9e]
/usr/sbin/apache2(ap_process_request+0x1c8)[0x7f4fe2a257b8]
/usr/sbin/apache2(+0x4e748)[0x7f4fe2a22748]
/usr/sbin/apache2(ap_run_process_connection+0x68)[0x7f4fe2a1bef8]
/usr/sbin/apache2(+0x56177)[0x7f4fe2a2a177]
/usr/sbin/apache2(+0x5648a)[0x7f4fe2a2a48a]
/usr/sbin/apache2(ap_mpm_run+0xc24)[0x7f4fe2a2b114]
/usr/sbin/apache2(main+0xb40)[0x7f4fe2a00530]
/lib/libc.so.6(__libc_start_main+0xfe)[0x7f4fe1ba5d8e]
/usr/sbin/apache2(+0x2b4c9)[0x7f4fe29ff4c9]
======= Memory map: ========
(I left this part off let me know if its needed)

[Allan-9 (allan-9)]

I created a brand new server last night, using EC2 (Amazon) - used the official AMI of Ubuntu 10.10 from Ubuntu site ami-cef405a7
Added Apache, PHP, MySQL (apt-get install)

Moved one domain which uses VBulletin to this server
This morning I have the same crash on this new server int he Apache error.log

*** glibc detected *** /usr/sbin/apache2: double free or corruption (fasttop): 0x00007f5c084c7950 ***
======= Backtrace: =========
/lib/libc.so.6(+0x774b6)[0x7f5c069e74b6]
/lib/libc.so.6(cfree+0x73)[0x7f5c069edc83]
/usr/lib/apache2/modules/libphp5.so(+0x2d40eb)[0x7f5c035200eb]
/usr/lib/apache2/modules/libphp5.so(+0x1f038a)[0x7f5c0343c38a]
/usr/lib/apache2/modules/libphp5.so(zend_error+0x60c)[0x7f5c0357213c]
/lib/libc.so.6(+0x33c20)[0x7f5c069a3c20]
/lib/libc.so.6(__libc_malloc+0xd)[0x7f5c069ec32d]
/lib/libc.so.6(__strdup+0x22)[0x7f5c069f2b82]
/usr/lib/apache2/modules/libphp5.so(+0x2d3ed5)[0x7f5c0351fed5]
/usr/lib/apache2/modules/libphp5.so(+0x1f038a)[0x7f5c0343c38a]
/usr/lib/apache2/modules/libphp5.so(zend_error+0x60c)[0x7f5c0357213c]
/usr/lib/apache2/modules/libphp5.so(+0x32fee6)[0x7f5c0357bee6]
/usr/lib/apache2/modules/libphp5.so(zend_parse_parameters+0x95)[0x7f5c0357c6a5]
/usr/lib/apache2/modules/libphp5.so(zif_feof+0x2c)[0x7f5c034b088c]
/usr/lib/apache2/modules/libphp5.so(+0x3770fa)[0x7f5c035c30fa]
/usr/lib/apache2/modules/libphp5.so(execute+0x210)[0x7f5c0359a1a0]
/usr/lib/apache2/modules/libphp5.so(zend_execute_scripts+0x15d)[0x7f5c0357199d]
/usr/lib/apache2/modules/libphp5.so(php_execute_script+0x1d8)[0x7f5c0351d2a8]
/usr/lib/apache2/modules/libphp5.so(+0x3b359d)[0x7f5c035ff59d]
/usr/sbin/apache2(ap_run_handler+0x70)[0x7f5c077fe310]
/usr/sbin/apache2(ap_invoke_handler+0xce)[0x7f5c07801c9e]
/usr/sbin/apache2(ap_process_request+0x1c8)[0x7f5c0780f7b8]
/usr/sbin/apache2(+0x4e748)[0x7f5c0780c748]
/usr/sbin/apache2(ap_run_process_connection+0x68)[0x7f5c07805ef8]
/usr/sbin/apache2(+0x56177)[0x7f5c07814177]
/usr/sbin/apache2(+0x5648a)[0x7f5c0781448a]
/usr/sbin/apache2(ap_mpm_run+0xc24)[0x7f5c07815114]
/usr/sbin/apache2(main+0xb40)[0x7f5c077ea530]
/lib/libc.so.6(__libc_start_main+0xfe)[0x7f5c0698ed8e]
/usr/sbin/apache2(+0x2b4c9)[0x7f5c077e94c9]

[Dave Walker (davewalker)]

Thanks for the bug report. Are you able to provide or comment on the work load you doing, to allow us to reproduce this?

Thanks.

Marking Low, may be readjusted depending on work load.

[Allan-9 (allan-9)]

Hi, thanks for your attention

As far as work - Ive built a new Amazone server from scratch, using Ubuntos official AMI not a community one

I managed by sheer luck to move one domain to the new server and the crashes happen on the new server but for two days the original server hasnt had one crash. Which is a huge step to isolating the issue.
The original server is a Drupal/Moodle setup - no crashes in two days

The failing server is a WprdPerfect/Vbulletin/HTML set up. I upgraded the WP software yestrday but the problem remains. VBulletin is at latest release.

I turned on PHP error logging but have no entries, not sure if ther eis something I did wrong or PHP just doesnt error out

My biggest issue now is that the crash/dump doesnt do a time stamp, so the I have to guess and I've not been able to find any smoking gun from the logs about what was going on at the time of the crash.

Is there any way to force entries intot he error log? Perhaps Some wget call I can put in a cron job that will cause and error entry?

When you say Work 'load' I am not sure what you mean, you want to know the server loads?

Allan

[Allan-9 (allan-9)]

I added Dumpio to the apache config, I was able to catch a apache dump in a 45s window.
I went through ALL the logs, not just apache logs, and looked at entries in that time range,
I went through access.log and executed all the GET statements, called the same pages, no errors occured.

From a load perspective there were about 20 entries in the access.log during that time frame, not very loaded.
I have a program called loadavg running and i dont see any spikes or unusual loads, trasnmissions or memory usage when the apache resets occur

Any help or ideas on what to look for would be greatly appreciated

Allan

[Launchpad Janitor (janitor)]

[Expired for apache2 (Ubuntu) because there has been no activity for 60 days.]