Apache2 crashes on reload

Bug #669005 reported by Guy Van Sanden
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
apache2 (Ubuntu)
Won't Fix
Undecided
Unassigned

Bug Description

Binary package hint: apache2

Apache2 recently started crashing on reload, a full restart of apache still works.

The error log shows:
Sun Oct 31 08:14:03 2010] [notice] Graceful restart requested, doing restart
[Sun Oct 31 08:14:03 2010] [notice] seg fault or similar nasty error detected in the parent process

Because it no longer reloads, apache is down after each logrotate.

System information:

* lsb_release -a (10-31 08:36)
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 8.04.4 LTS
Release: 8.04
Codename: hardy

* dpkg -l |grep apache
ii apache2 2.2.8-1ubuntu0.18 Next generation, scalable, extendable web se
ii apache2-mpm-prefork 2.2.8-1ubuntu0.18 Traditional model for Apache HTTPD
ii apache2-utils 2.2.8-1ubuntu0.18 utility programs for webservers
ii apache2.2-common 2.2.8-1ubuntu0.18 Next generation, scalable, extendable web se
ii libapache2-mod-php5 5.2.4-2ubuntu5.12 server-side, HTML-embedded scripting languag

* dpkg -l |grep php

ii libapache2-mod-php5 5.2.4-2ubuntu5.12 server-side, HTML-embedded scripting languag
ii php-pear 5.2.4-2ubuntu5.12 PEAR - PHP Extension and Application Reposit
ii php5 5.2.4-2ubuntu5.12 server-side, HTML-embedded scripting languag
ii php5-cgi 5.2.4-2ubuntu5.12 server-side, HTML-embedded scripting languag
ii php5-cli 5.2.4-2ubuntu5.12 command-line interpreter for the php5 script
ii php5-common 5.2.4-2ubuntu5.12 Common files for packages built from the php
ii php5-curl 5.2.4-2ubuntu5.12 CURL module for php5
ii php5-gd 5.2.4-2ubuntu5.12 GD module for php5
ii php5-gmp 5.2.4-2ubuntu5.12 GMP module for php5
ii php5-ldap 5.2.4-2ubuntu5.12 LDAP module for php5
ii php5-mysql 5.2.4-2ubuntu5.12 MySQL module for php5
ii phpldapadmin 1.1.0.4-2ubuntu1 web based interface for administering LDAP s

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Hello Guy, thank you for taking the time to file this bug report and help us make Ubuntu better.

Can you enable core dumps for apache so we get a core file when it segfaults?

This *should* do the trick:

service apache2 stop
ulimit -c unlimited
service apache2 start

After it has crashed, you can backtrace it yourself:

gdb /usr/sbin/apache2 /path/to/core/file (probably /tmp/something, look in /var/log/error.log)

then run

'bt'

and paste the output here.

Marking Incomplete pending response from Guy.

Changed in apache2 (Ubuntu):
status: New → Incomplete
tags: added: hardy
Revision history for this message
Guy Van Sanden (gvs) wrote :

Hi Clint

I did exactly those steps:

root@chaos:/tmp # ulimit -c unlimited (11-05 13:25)
root@chaos:/tmp # service apache2 start (11-05 13:25)
 * Starting web server apache2
   ...done.
root@chaos:/tmp # /etc/init.d/apache2 reload (11-05 13:25)
 * Reloading web server config apache2
   ...done.

And get the same error in the log, but no core file.

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Guy, thanks for trying that out.

I just simulated it here on a maverick system and the core was dumped in /etc/apache2/core

Can you look in /etc/apache2 for the core file?

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Oh also Guy, you'll get a better backtrace output if you also install the 'libc6-dbg' package to get full debug symbols.

Revision history for this message
Guy Van Sanden (gvs) wrote :

Thanks, that worked

(gdb) bt
#0 0x00007f18631324f0 in ?? ()
#1 0x00007f186d18f055 in ERR_get_state () from /usr/lib/libcrypto.so.0.9.8
#2 0x00007f186d18fa05 in ERR_put_error () from /usr/lib/libcrypto.so.0.9.8
#3 0x00007f186d17edcc in ENGINE_add () from /usr/lib/libcrypto.so.0.9.8
#4 0x00007f186d1818c8 in ENGINE_load_dynamic () from /usr/lib/libcrypto.so.0.9.8
#5 0x00007f1867d06859 in ?? () from /usr/lib/libcurl.so.4
#6 0x00007f1867d0ed10 in curl_global_init () from /usr/lib/libcurl.so.4
#7 0x00007f1867f2fcee in zm_startup_curl () from /usr/lib/php5/20060613/curl.so
#8 0x00007f1869a28da3 in zend_startup_module_ex () from /usr/lib/apache2/modules/libphp5.so
#9 0x00007f1869a2e195 in zend_hash_apply () from /usr/lib/apache2/modules/libphp5.so
#10 0x00007f1869a275da in zend_startup_modules () from /usr/lib/apache2/modules/libphp5.so
#11 0x00007f18699e0b0b in php_module_startup () from /usr/lib/apache2/modules/libphp5.so
#12 0x00007f1869aa2e45 in ?? () from /usr/lib/apache2/modules/libphp5.so
#13 0x00007f1869aa38e3 in ?? () from /usr/lib/apache2/modules/libphp5.so
#14 0x0000000000437bcc in ap_run_post_config ()
#15 0x0000000000425a21 in main ()

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Guy, awesome, this narrows things down a bit. Can you also install these debug symbols and repeat the trace?:

php5-dbg
libssl0.9.8-dbg
libcurl3-dbg

I'm guessing that this is a bug in PHP or curl that isn't re-initializing some kind of global state for openssl.

Are you definitely using the curl module for php? If not, you can try

apt-get remove php5-curl

And see if that resolves the issue. Please report back if that does (as it will help us to isolate the issue).

Changed in apache2 (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Marking Confirmed as this is definitely a bug.

Revision history for this message
Guy Van Sanden (gvs) wrote :

Hi Clint

It gives me this:

(gdb) bt
#0 0x00007fb3b834a4f0 in ?? ()
#1 0x00007fb3c23a7055 in ERR_get_state () at err.c:1005
#2 0x00007fb3c23a7a05 in ERR_put_error (lib=38, func=120, reason=103, file=0x7fb3c240f0e3 "eng_list.c",
    line=116) at err.c:701
#3 0x00007fb3c2396dcc in ENGINE_add (e=0x98a6f0) at eng_list.c:115
#4 0x00007fb3c23998c8 in ENGINE_load_dynamic () at eng_dyn.c:291
#5 0x00007fb3bcf1e859 in Curl_ossl_init () at ssluse.c:560
#6 0x00007fb3bcf26d10 in curl_global_init (flags=<value optimized out>) at easy.c:226
#7 0x00007fb3bd147cee in zm_startup_curl () from /usr/lib/php5/20060613/curl.so
#8 0x00007fb3bec40da3 in zend_startup_module_ex () from /usr/lib/apache2/modules/libphp5.so
#9 0x00007fb3bec46195 in zend_hash_apply () from /usr/lib/apache2/modules/libphp5.so
#10 0x00007fb3bec3f5da in zend_startup_modules () from /usr/lib/apache2/modules/libphp5.so
#11 0x00007fb3bebf8b0b in php_module_startup () from /usr/lib/apache2/modules/libphp5.so
#12 0x00007fb3becbae45 in ?? () from /usr/lib/apache2/modules/libphp5.so
#13 0x00007fb3becbb8e3 in ?? () from /usr/lib/apache2/modules/libphp5.so
#14 0x0000000000437bcc in ap_run_post_config ()
#15 0x0000000000425a21 in main ()

I doubt it's SSL related, the server is behind a reverse proxy that does SSL termination and that part of the config didn't change. But if I am correct, it started doing this after I installed Community-ID, which is using Zend and php-curl...

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Guy, thanks. I've been digging through the code a bit, and it appears that we're probably dying because the id_callback that PHP has setup is invalid in some way. The spot where we get the "in ??" is this:

    pid=(unsigned long)CRYPTO_thread_id();

CRYPTO_thread_id() returns the result of a dynamic function pointer which is in fact set during the php curl extension's initialization step to tsrm_thread_id(), but its possible this pointer has been invalidated in some way.

I'm going to go ahead and try to install Community-ID and see if I can reproduce this on Hardy.

Revision history for this message
Guy Van Sanden (gvs) wrote :

Ok, I'm running 2.0 RC3 already.

Changed in apache2 (Ubuntu):
assignee: nobody → Clint Byrum (clint-fewbar)
tags: added: work-intensive
Changed in apache2 (Ubuntu):
assignee: Clint Byrum (clint-fewbar) → farhan saleh robleh (farhn)
Colin Watson (cjwatson)
Changed in apache2 (Ubuntu):
assignee: farhan saleh robleh (farhn) → nobody
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

We are sorry, but it looks like this bug fell through the cracks and was forgotten. Since Ubuntu 8.04 is no longer supported, I'm going to mark this issue as "won't fix".

If it's still happening in a supported version of Ubuntu, then please reopen this bug stating the new information. Thanks.

Changed in apache2 (Ubuntu):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.