Apache2 crashes on reload

Bug #669005 reported by Guy Van Sanden on 2010-10-31
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
apache2 (Ubuntu)
Undecided
Clint Byrum

Bug Description

Binary package hint: apache2

Apache2 recently started crashing on reload, a full restart of apache still works.

The error log shows:
Sun Oct 31 08:14:03 2010] [notice] Graceful restart requested, doing restart
[Sun Oct 31 08:14:03 2010] [notice] seg fault or similar nasty error detected in the parent process

Because it no longer reloads, apache is down after each logrotate.

System information:

* lsb_release -a (10-31 08:36)
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 8.04.4 LTS
Release: 8.04
Codename: hardy

* dpkg -l |grep apache
ii apache2 2.2.8-1ubuntu0.18 Next generation, scalable, extendable web se
ii apache2-mpm-prefork 2.2.8-1ubuntu0.18 Traditional model for Apache HTTPD
ii apache2-utils 2.2.8-1ubuntu0.18 utility programs for webservers
ii apache2.2-common 2.2.8-1ubuntu0.18 Next generation, scalable, extendable web se
ii libapache2-mod-php5 5.2.4-2ubuntu5.12 server-side, HTML-embedded scripting languag

* dpkg -l |grep php

ii libapache2-mod-php5 5.2.4-2ubuntu5.12 server-side, HTML-embedded scripting languag
ii php-pear 5.2.4-2ubuntu5.12 PEAR - PHP Extension and Application Reposit
ii php5 5.2.4-2ubuntu5.12 server-side, HTML-embedded scripting languag
ii php5-cgi 5.2.4-2ubuntu5.12 server-side, HTML-embedded scripting languag
ii php5-cli 5.2.4-2ubuntu5.12 command-line interpreter for the php5 script
ii php5-common 5.2.4-2ubuntu5.12 Common files for packages built from the php
ii php5-curl 5.2.4-2ubuntu5.12 CURL module for php5
ii php5-gd 5.2.4-2ubuntu5.12 GD module for php5
ii php5-gmp 5.2.4-2ubuntu5.12 GMP module for php5
ii php5-ldap 5.2.4-2ubuntu5.12 LDAP module for php5
ii php5-mysql 5.2.4-2ubuntu5.12 MySQL module for php5
ii phpldapadmin 1.1.0.4-2ubuntu1 web based interface for administering LDAP s

Clint Byrum (clint-fewbar) wrote :

Hello Guy, thank you for taking the time to file this bug report and help us make Ubuntu better.

Can you enable core dumps for apache so we get a core file when it segfaults?

This *should* do the trick:

service apache2 stop
ulimit -c unlimited
service apache2 start

After it has crashed, you can backtrace it yourself:

gdb /usr/sbin/apache2 /path/to/core/file (probably /tmp/something, look in /var/log/error.log)

then run

'bt'

and paste the output here.

Marking Incomplete pending response from Guy.

Changed in apache2 (Ubuntu):
status: New → Incomplete
tags: added: hardy
Guy Van Sanden (gvs) wrote :

Hi Clint

I did exactly those steps:

root@chaos:/tmp # ulimit -c unlimited (11-05 13:25)
root@chaos:/tmp # service apache2 start (11-05 13:25)
 * Starting web server apache2
   ...done.
root@chaos:/tmp # /etc/init.d/apache2 reload (11-05 13:25)
 * Reloading web server config apache2
   ...done.

And get the same error in the log, but no core file.

Clint Byrum (clint-fewbar) wrote :

Guy, thanks for trying that out.

I just simulated it here on a maverick system and the core was dumped in /etc/apache2/core

Can you look in /etc/apache2 for the core file?

Clint Byrum (clint-fewbar) wrote :

Oh also Guy, you'll get a better backtrace output if you also install the 'libc6-dbg' package to get full debug symbols.

Guy Van Sanden (gvs) wrote :

Thanks, that worked

(gdb) bt
#0 0x00007f18631324f0 in ?? ()
#1 0x00007f186d18f055 in ERR_get_state () from /usr/lib/libcrypto.so.0.9.8
#2 0x00007f186d18fa05 in ERR_put_error () from /usr/lib/libcrypto.so.0.9.8
#3 0x00007f186d17edcc in ENGINE_add () from /usr/lib/libcrypto.so.0.9.8
#4 0x00007f186d1818c8 in ENGINE_load_dynamic () from /usr/lib/libcrypto.so.0.9.8
#5 0x00007f1867d06859 in ?? () from /usr/lib/libcurl.so.4
#6 0x00007f1867d0ed10 in curl_global_init () from /usr/lib/libcurl.so.4
#7 0x00007f1867f2fcee in zm_startup_curl () from /usr/lib/php5/20060613/curl.so
#8 0x00007f1869a28da3 in zend_startup_module_ex () from /usr/lib/apache2/modules/libphp5.so
#9 0x00007f1869a2e195 in zend_hash_apply () from /usr/lib/apache2/modules/libphp5.so
#10 0x00007f1869a275da in zend_startup_modules () from /usr/lib/apache2/modules/libphp5.so
#11 0x00007f18699e0b0b in php_module_startup () from /usr/lib/apache2/modules/libphp5.so
#12 0x00007f1869aa2e45 in ?? () from /usr/lib/apache2/modules/libphp5.so
#13 0x00007f1869aa38e3 in ?? () from /usr/lib/apache2/modules/libphp5.so
#14 0x0000000000437bcc in ap_run_post_config ()
#15 0x0000000000425a21 in main ()

Clint Byrum (clint-fewbar) wrote :

Guy, awesome, this narrows things down a bit. Can you also install these debug symbols and repeat the trace?:

php5-dbg
libssl0.9.8-dbg
libcurl3-dbg

I'm guessing that this is a bug in PHP or curl that isn't re-initializing some kind of global state for openssl.

Are you definitely using the curl module for php? If not, you can try

apt-get remove php5-curl

And see if that resolves the issue. Please report back if that does (as it will help us to isolate the issue).

Changed in apache2 (Ubuntu):
status: Incomplete → Confirmed
Clint Byrum (clint-fewbar) wrote :

Marking Confirmed as this is definitely a bug.

Guy Van Sanden (gvs) wrote :

Hi Clint

It gives me this:

(gdb) bt
#0 0x00007fb3b834a4f0 in ?? ()
#1 0x00007fb3c23a7055 in ERR_get_state () at err.c:1005
#2 0x00007fb3c23a7a05 in ERR_put_error (lib=38, func=120, reason=103, file=0x7fb3c240f0e3 "eng_list.c",
    line=116) at err.c:701
#3 0x00007fb3c2396dcc in ENGINE_add (e=0x98a6f0) at eng_list.c:115
#4 0x00007fb3c23998c8 in ENGINE_load_dynamic () at eng_dyn.c:291
#5 0x00007fb3bcf1e859 in Curl_ossl_init () at ssluse.c:560
#6 0x00007fb3bcf26d10 in curl_global_init (flags=<value optimized out>) at easy.c:226
#7 0x00007fb3bd147cee in zm_startup_curl () from /usr/lib/php5/20060613/curl.so
#8 0x00007fb3bec40da3 in zend_startup_module_ex () from /usr/lib/apache2/modules/libphp5.so
#9 0x00007fb3bec46195 in zend_hash_apply () from /usr/lib/apache2/modules/libphp5.so
#10 0x00007fb3bec3f5da in zend_startup_modules () from /usr/lib/apache2/modules/libphp5.so
#11 0x00007fb3bebf8b0b in php_module_startup () from /usr/lib/apache2/modules/libphp5.so
#12 0x00007fb3becbae45 in ?? () from /usr/lib/apache2/modules/libphp5.so
#13 0x00007fb3becbb8e3 in ?? () from /usr/lib/apache2/modules/libphp5.so
#14 0x0000000000437bcc in ap_run_post_config ()
#15 0x0000000000425a21 in main ()

I doubt it's SSL related, the server is behind a reverse proxy that does SSL termination and that part of the config didn't change. But if I am correct, it started doing this after I installed Community-ID, which is using Zend and php-curl...

Clint Byrum (clint-fewbar) wrote :

Guy, thanks. I've been digging through the code a bit, and it appears that we're probably dying because the id_callback that PHP has setup is invalid in some way. The spot where we get the "in ??" is this:

    pid=(unsigned long)CRYPTO_thread_id();

CRYPTO_thread_id() returns the result of a dynamic function pointer which is in fact set during the php curl extension's initialization step to tsrm_thread_id(), but its possible this pointer has been invalidated in some way.

I'm going to go ahead and try to install Community-ID and see if I can reproduce this on Hardy.

Guy Van Sanden (gvs) wrote :

Ok, I'm running 2.0 RC3 already.

Changed in apache2 (Ubuntu):
assignee: nobody → Clint Byrum (clint-fewbar)
tags: added: work-intensive
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers