apache2.4 mod-php5.5 random segmentation faults in zend_stack_push() and zend_hash_find()

[Leszek]

Hello,

I observe random segfaults in PHP running as Apache module. I see Segfaults in Apache error log, I have collected multiple core dumps which indicate that the crashes happen in zend_stack_push() or zend_hash_find(). It seems that the crash happens at scripts compile time since in all the cases, I see the compile_file() function at some level of the backtrace.

It is a low traffic website, 30-40 page hits /minute, VM hosted by Digital Ocean. Initially the problem was on Ubuntu 14.04 then in order to exclude hardware errors I spawned another VM that is hosted also in DO but in another region and currently I'm running it on Ubuntu 14.10, segfaults still happen:
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.10
Release: 14.10
Codename: utopic

Modules loaded by Apache:
# apachectl -M
Loaded Modules:
 core_module (static)
 so_module (static)
 watchdog_module (static)
 http_module (static)
 log_config_module (static)
 logio_module (static)
 version_module (static)
 unixd_module (static)
 access_compat_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_core_module (shared)
 authn_file_module (shared)
 authz_core_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 expires_module (shared)
 filter_module (shared)
 headers_module (shared)
 mime_module (shared)
 mpm_prefork_module (shared)
 negotiation_module (shared)
 php5_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 status_module (shared)

Modules loaded by PHP:
# ls -1 /etc/php5/apache2/conf.d/
10-pdo.ini
20-curl.ini
20-gd.ini
20-json.ini
20-mysqli.ini
20-mysql.ini
20-newrelic.ini
20-pdo_mysql.ini
20-readline.ini

Software versions:
apache2 2.4.10-1ubuntu1
apache2-bin 2.4.10-1ubuntu1
libapache2-mod-php5 5.5.12+dfsg-2ubuntu4.1
newrelic-php5 4.17.0.79
newrelic-php5-common 4.17.0.79
php5-cli 5.5.12+dfsg-2ubuntu4.1
php5-common 5.5.12+dfsg-2ubuntu4.1
php5-curl 5.5.12+dfsg-2ubuntu4.1
php5-dbg 5.5.12+dfsg-2ubuntu4.1
php5-dev 5.5.12+dfsg-2ubuntu4.1
php5-gd 5.5.12+dfsg-2ubuntu4.1
php5-json 1.3.6-1
php5-mysql 5.5.12+dfsg-2ubuntu4.1
php5-readline 5.5.12+dfsg-2ubuntu4.1

Attachments:
- php.ini
- phpinfo.html - saved phpinfo()
- index.php: the php script that is being compiled when the segfaults happen seems to be always the same and it is main wordpress index.php file
- bt-zen_stack_push.txt - backtrace of segfault from zend_stack_push()
- bt-zen_hash_find.txt - backtrace of seggault from zend_hash_find()

Let me know if I can provide any more useful info, please.

Cheers,
Leszek

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rowan Wookey (rwky)]

I've also encountered this I'm not positive that the sefaults are the same, however, I disabled APC on 12.04 and the OPCache on 14.04 and the servers have stopped segfaulting. Hopefully this helps!

[Leszek (leszek-eljasz)]

Hi Rowan,

Well I use neither OPCache nor APC, but the segfaults still happen.

Regards,
Leszek

[Gerrit Venema (gmoniker)]

We observe regular segfaults on Ubuntu 14.04 LTS with Apache and PHP in its default packages presenting these backtraces in a Coredump file:

#0 0x00007f9911e619ad in zend_stack_push (
    stack=stack@entry=0x7f9912627ca0 <compiler_globals+608>,
    element=element@entry=0x7f9912627c78 <compiler_globals+568>,
    size=size@entry=40) at /build/buildd/php5-5.5.9+dfsg/Zend/zend_stack.c:42
#1 0x00007f9911e2d34e in compile_file (
    file_handle=file_handle@entry=0x7fffe74e7e00, type=2)
    at Zend/zend_language_scanner.l:586
#2 0x00007f9911e52b2a in dtrace_compile_file (file_handle=0x7fffe74e7e00,
    type=<optimized out>)
    at /build/buildd/php5-5.5.9+dfsg/Zend/zend_dtrace.c:40
#3 0x00007f9911cdbce4 in phar_compile_file (file_handle=<optimized out>,
    type=<optimized out>) at /build/buildd/php5-5.5.9+dfsg/ext/phar/phar.c:3383
#4 0x00007f990baca1d4 in persistent_compile_file (file_handle=0x7fffe74e7e00,
    type=2) at /build/buildd/php5-5.5.9+dfsg/ext/opcache/ZendAccelerator.c:1634
#5 0x00007f990bd64f19 in ?? ()
   from /usr/lib/php5/20121212/ioncube_loader_lin_5.5.so
#6 0x00007f9911e645af in zend_execute_scripts (type=type@entry=2,
    retval=retval@entry=0x0, file_count=file_count@entry=1)
    at /build/buildd/php5-5.5.9+dfsg/Zend/zend.c:1308
#7 0x00007f9911f1452d in php_handler (r=<optimized out>)
    at /build/buildd/php5-5.5.9+dfsg/sapi/apache2handler/sapi_apache2.c:669
#8 0x00007f9918178680 in ap_run_handler (r=0x7f9912f040a0) at config.c:169
#9 0x00007f9918178bc9 in ap_invoke_handler (r=r@entry=0x7f9912f040a0)
---Type <return> to continue, or q <return> to quit---
    at config.c:439
#10 0x00007f991818e16a in ap_process_async_request (r=0x7f9912f040a0)
    at http_request.c:317
#11 0x00007f991818e444 in ap_process_request (r=r@entry=0x7f9912f040a0)
    at http_request.c:363
#12 0x00007f991818af02 in ap_process_http_sync_connection (c=0x7f991479e290)
    at http_core.c:190
#13 ap_process_http_connection (c=0x7f991479e290) at http_core.c:231
#14 0x00007f9918181cc0 in ap_run_process_connection (c=0x7f991479e290)
    at connection.c:41
#15 0x00007f99181820a8 in ap_process_connection (c=c@entry=0x7f991479e290,
    csd=<optimized out>) at connection.c:202
#16 0x00007f991333f767 in child_main (child_num_arg=child_num_arg@entry=92)
    at prefork.c:704
#17 0x00007f991333f9a6 in make_child (s=0x7f99180dfde0, slot=92)
    at prefork.c:800
#18 0x00007f991334060e in perform_idle_server_maintenance (p=<optimized out>)
    at prefork.c:902
#19 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
    s=<optimized out>) at prefork.c:1090
#20 0x00007f991815f69e in ap_run_mpm (pconf=0x7f9918115028,
    plog=0x7f99180e3028, s=0x7f99180dfde0) at mpm_common.c:96
#21 0x00007f9918158e36 in main (argc=3, argv=0x7fffe74e8508) at main.c:777

PHP is coming in to the stack push function thinking that it is already initialized (stack_max=64) while its elements pointer is null, so it segfaults when trying to store a heap segment in its stack.

This may very well be an upstream bug in the PHP SAPI module for Apache. In this case I think this bug report (https://bugs.php.net/bug.php?id=68486) on PHP is highly relevant. It is said to not be present on ...

[Gerrit Venema (gmoniker)]

The bug is instantly reproducible on a fresh Ubuntu 14.04.

Install server.
apt-get install apache2 php5 netcat

Setup a small PHP script with output in /var/www/html/
For Example as test.php
<?php
 echo date('U').PHP_EOL;
?>

shutdown Apache service:
apache2ctl stop

Disable opcache with semicolon in front in /etc/php5/apache2/conf.d/05-opcache.ini

Start Apache debug server in separate shell
apache2ctl -X

Execute this script (thanks to biggi at stefna dot is):
echo -e "GET /test.php HTTP/1.1\nHost: localhost\n\nGET /test.php HTTP/1.1\nHost: localhost\n\n"|nc localhost 80

Result: immediate segfault

If you do only one request per connection, no problem
If you activate opcache, you will only get one response, the second one appears on the stdout of the debug apache
If you fill the opcache it will segfault if it can't fit the script anymore

I confirmed the same behaviour on Centos 7 with its default packaged Apache 2.4.6, so this is an upstream bug.

[Gerrit Venema (gmoniker)]

This is hopefully the proper place for resolution of this bug.

https://bz.apache.org/bugzilla/show_bug.cgi?id=56984

[Gerrit Venema (gmoniker)]

For any parties interested in testing, there is now a patch available at https://bugs.php.net/bug.php?id=68486 which is looking to solve the segfaults that happen in processing pipelined PHP requests in Apache 2.4.

[Gerrit Venema (gmoniker)]

The root cause of the problem has been determined to lie with the apache2handler for PHP. It is not sufficiently up to date to function well with Apache 2.4 receiving pipelined requests. A patch for the most common setup has been developed, but it has not yet been integrated upstream.

I have also put in an enhancement request at https://bz.apache.org/bugzilla/show_bug.cgi?id=57756, for a configuration option that could provide a fallback for server administrator, because this may equally affect other Apache 2.4 add-on handlers that have not undergone rigorous testing for this use case.

[Gerrit Venema (gmoniker)]

Het PHP project heeft een fix uitgebracht voor dit probleem met de cleanup patch die ik heb ingediend, en deze is verwerkt in deze USN:
http://www.ubuntu.com/usn/usn-2572-1/

[Gerrit Venema (gmoniker)]

Apologies, I didn't realise I was writing in Dutch :-)

The PHP project has released a fix for this problem with my cleanup patch. It has been released with 5.6.28 and 5.5.24. It is also part of an Ubuntu security release: http://www.ubuntu.com/usn/usn-2572-1/