Pantheon site crashing every few hours

[manuj_78]

Hi,

I have installed pantheon on Ubuntu (lucid). But please note I have downgraded the php to 5.2.10 (karmic version) since the site was giving some errors with drupal 6.13.

Server softwares

Ubuntu Lucid
Apache/2.2.14
PHP 5.2.13 (cli)
Mysql 5.1.41-3ubuntu12.3
Varnish-2.1
Mercury 1.1

The site is working good.. but from time to time load increases tremendously and server just hangs. Seems like php connections are not getting closed.
This is from error.log

[Mon Jul 12 20:47:16 2010] [error] child died with signal 9
[Mon Jul 12 20:47:24 2010] [error] child died with signal 9
[Mon Jul 12 20:47:24 2010] [error] child died with signal 9
[Mon Jul 12 20:47:36 2010] [error] child died with signal 9

My server tuneables
/etc/apache2/apache2.conf

export APACHE_MAXCLIENTS="10"
/etc/apparmor.d/usr.sbin.mysqld

export APPARMOR_MYSQLD=""
/etc/default/tomcat6

export TOMCAT_MEMORY="128"
/etc/default/varnish

export VARNISH_MEMORY="64"
/etc/memcached.conf

export MEMCACHED_MEMORY="128"
/etc/mysql/my.cnf

export INNODB_BUFFER_POOL_SIZE="64"
in bytes (ie, 1Gb = 1073741824 bytes)

export INNODB_LOG_FILE_SIZE="1073741824"
export KEY_BUFFER_SIZE="8"
export MYSQL_MAX_CONNECTIONS="20"
/etc/php5/apache2/php.ini

export PHP_MEMORY="96"
/etc/php5/conf.d/apc.ini

export APC_MEMORY="128"

Could anyone tell me how i can make sure client connections are closed correctly. Due to this server load my site and even my server is going down. I think this happens after around 3-4 hours of running.

Regards

[Josh Koenig (joshkoenig)]

I don't know what we can do to support this since you are altering the configuration and there are many many places things could be going wrong (including within your Drupal application). You should also not be using Drupal 6.13.

If you can get onto a normal stack with the latest version of Pressflow, we may be able to begin isolating the problem. However, it's likely related to whatever is causing the load spike, which likely isn't the system stack. Possibly cron could be causing lockups.

[Greg Coit (gregcoit)]

This question was first posted here: http://groups.drupal.org/node/80274

[manuj_78 (manujg)]

Can you please clarify when you say we are not running normal stack..if it is because of my mention of running drupal 6.13, then apologies for the confusion..I shouldnt have mentioned Drupal..We are running Mercury 1.0 (Pressflow 6.17)

We first started with Mercury 1.1 on Lucid and as we started seeing the spikes on the server, decided to move the site to Karmic and Mercury 1

Here is the current server config

Ubuntu Karmic
Apache/2.2.12 (Ubuntu)
PHP Version 5.2.10-2ubuntu6.4
Mysql 5.0.83-0ubuntu3 (Ubuntu)
varnish-2.0.4
Mercury 1.0

But we still have the problem..
As suggested we did a Memory profile and here are the results (http://groups.drupal.org/node/80274#comment-253904)

xdebug report

Showing the 20 most costly calls sorted by 'memory-own'.

                                                                        Inclusive Own
function #calls time memory time memory
--------------------------------------------------------------------------------------------------------
MemcachePool->get 88 0.2388 18694216 0.2388 18694216
module_list 298 0.0718 9739040 0.0388 9304408
views_plugin_display->option_definition 288 0.0374 9317408 0.0321 9295864
ob_start 168 0.0040 7017848 0.0040 7017848
array_keys 2200 0.0424 5640824 0.0424 5640824
func_get_args 2284 0.0456 5168312 0.0456 5168312
str_replace 5019 0.0886 5168008 0.0886 5168008
array_merge 506 0.0130 4716592 0.0130 4716592
views_handler_field->option_definition 250 0.0197 3983032 0.0154 3904872
drupal_load 218 0.0624 4112664 0.0329 3890432
module_hook 10694 0.5985 3513888 0.4189 3513888
t 2926 0.3052 3711784 0.1357 3484616
date_part_extract 537 0.0274 3321384 0.0193 3291096
unserialize 301 0.0192 2948712 0.0192 2948712
explode 1449 0.0237 2667432 0.0237 2667432
variable_get 4294 0.0896 2350280 0.0896 2350280
preg_replace 3938 0.0864 1945864 0.0864 1945864
mysql_fetch_object 915 0.0239 1840104 0.0239 1840104
_db_query_callback 3037 0.1914 1867120 0.1152 1757128
url 651 5.0089 2557576 0.0876 1601952

Is there something else we can do...