apache running at 100% indefinitely

[Geoff Silver]

Binary package hint: apache2

Description: Ubuntu 9.10
Release: 9.10
Arch: amd64

This apache2 process has been running for days at 100% CPU. There were actually ~6 or 7 of them - I killed all but one, to get my load down from 12+ to something usable. I've seen this several times in the last 2 months since I migrated from 8.04 i386 to 9.10 amd64.

top - 09:00:31 up 20 days, 12:45, 4 users, load average: 1.49, 1.39, 3.67
Tasks: 258 total, 3 running, 254 sleeping, 1 stopped, 0 zombie
Cpu(s): 23.1%us, 0.2%sy, 0.0%ni, 74.4%id, 2.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2025104k total, 1845848k used, 179256k free, 266588k buffers
Swap: 2056248k total, 56472k used, 1999776k free, 641952k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 7966 www-data 20 0 326m 21m 1888 R 100 1.1 3144:35 apache2

root@ns:/var/log/apache2# apt-cache policy apache2
apache2:
  Installed: 2.2.12-1ubuntu2.2
  Candidate: 2.2.12-1ubuntu2.2
  Version table:
 *** 2.2.12-1ubuntu2.2 0
        500 http://us.archive.ubuntu.com karmic-updates/main Packages
        500 http://security.ubuntu.com karmic-security/main Packages
        100 /var/lib/dpkg/status
     2.2.12-1ubuntu2 0
        500 http://us.archive.ubuntu.com karmic/main Packages
root@ns:/var/log/apache2# apt-cache policy php5
php5:
  Installed: 5.2.10.dfsg.1-2ubuntu6.4
  Candidate: 5.2.10.dfsg.1-2ubuntu6.4
  Version table:
 *** 5.2.10.dfsg.1-2ubuntu6.4 0
        500 http://us.archive.ubuntu.com karmic-updates/main Packages
        500 http://security.ubuntu.com karmic-security/main Packages
        100 /var/lib/dpkg/status
     5.2.10.dfsg.1-2ubuntu6 0
        500 http://us.archive.ubuntu.com karmic/main Packages
root@ns:/var/log/apache2# apt-cache policy zend-framework
zend-framework:
  Installed: (none)
  Candidate: 1.9.4-0ubuntu2.1
  Version table:
     1.9.4-0ubuntu2.1 0
        500 http://us.archive.ubuntu.com karmic-updates/universe Packages
        500 http://security.ubuntu.com karmic-security/universe Packages
     1.9.4-0ubuntu2 0
        500 http://us.archive.ubuntu.com karmic/universe Packages
root@ns:/var/log/apache2# apt-cache policy libzend-framework-php
libzend-framework-php:
  Installed: (none)
  Candidate: 1.9.4-0ubuntu2.1
  Version table:
     1.9.4-0ubuntu2.1 0
        500 http://us.archive.ubuntu.com karmic-updates/universe Packages
        500 http://security.ubuntu.com karmic-security/universe Packages
     1.9.4-0ubuntu2 0
        500 http://us.archive.ubuntu.com karmic/universe Packages

strace and ltrace yield nothing. Attaching gdb to the running process shows:

0x00007f749e47fff1 in _zval_ptr_dtor (zval_ptr=0x7fff14c20c50)
    at /build/buildd/php5-5.2.10.dfsg.1/Zend/zend_execute_API.c:413
413 /build/buildd/php5-5.2.10.dfsg.1/Zend/zend_execute_API.c: No such file or directory.
(gdb) bt
#0 0x00007f749e47fff1 in _zval_ptr_dtor (zval_ptr=0x7fff14c20c50)
    at /build/buildd/php5-5.2.10.dfsg.1/Zend/zend_execute_API.c:413
#1 0x00007f749e4cd9bd in ZEND_ASSIGN_SPEC_CV_VAR_HANDLER (
    execute_data=0x7fff14c21dd0)
    at /build/buildd/php5-5.2.10.dfsg.1/Zend/zend_vm_execute.h:24711
#2 0x00007f749e4b075c in execute (op_array=0x7f74a4909bf0)
    at /build/buildd/php5-5.2.10.dfsg.1/Zend/zend_vm_execute.h:92
#3 0x00007f749e4b43aa in zend_do_fcall_common_helper_SPEC (
    execute_data=0x7fff14c221a0)
    at /build/buildd/php5-5.2.10.dfsg.1/Zend/zend_vm_execute.h:234
#4 0x00007f749e4b075c in execute (op_array=0x7f74a4d58d50)
    at /build/buildd/php5-5.2.10.dfsg.1/Zend/zend_vm_execute.h:92
#5 0x00007f749e48c5f6 in zend_execute_scripts (type=32767, retval=0x0,
    file_count=3) at /build/buildd/php5-5.2.10.dfsg.1/Zend/zend.c:1215
#6 0x00007f749e447673 in php_execute_script (primary_file=0x7f74a573b150)
    at /build/buildd/php5-5.2.10.dfsg.1/main/main.c:2046
#7 0x00007f749e4fe5ed in php_handler (r=0x7f749e4fe5ed)
    at /build/buildd/php5-5.2.10.dfsg.1/sapi/apache2handler/sapi_apache2.c:651
#8 0x00007f74a311f0d0 in ap_run_handler ()
   from /usr/lib/apache2/mpm-prefork/apache2
#9 0x00007f74a3122a18 in ap_invoke_handler ()
   from /usr/lib/apache2/mpm-prefork/apache2
#10 0x00007f74a3130578 in ap_process_request ()
   from /usr/lib/apache2/mpm-prefork/apache2
#11 0x00007f74a312d458 in ?? () from /usr/lib/apache2/mpm-prefork/apache2
#12 0x00007f74a3126c68 in ap_run_process_connection ()
   from /usr/lib/apache2/mpm-prefork/apache2
#13 0x00007f74a3134fcb in ?? () from /usr/lib/apache2/mpm-prefork/apache2
#14 0x00007f74a313529a in ?? () from /usr/lib/apache2/mpm-prefork/apache2
#15 0x00007f74a31355cb in ap_mpm_run ()
   from /usr/lib/apache2/mpm-prefork/apache2
#16 0x00007f74a310b350 in main () from /usr/lib/apache2/mpm-prefork/apache2

ProblemType: Bug
Architecture: amd64
Date: Tue May 4 08:55:39 2010
DistroRelease: Ubuntu 9.10
ExecutablePath: /usr/lib/apache2/mpm-prefork/apache2
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
Package: apache2.2-bin 2.2.12-1ubuntu2.2
ProcEnviron:
 PATH=(custom, no user)
 LANG=C
ProcVersionSignature: Ubuntu 2.6.31-20.58-generic
SourcePackage: apache2
Uname: Linux 2.6.31-20-generic x86_64

[Chuck Short (zulcss)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Unfortunately, we can't fix it because your description didn't include enough information. You may find it helpful to read "How to report bugs effectively" http://www.chiark.greenend.org.uk/~sgtatham/bugs.html. We'd be grateful if you would then provide a more complete description of the problem. We have instructions on debugging some types of problems at http://wiki.ubuntu.com/DebuggingProcedures.
At a minimum, we need:
1. the specific steps or actions you took that caused you to encounter the problem,
2. the behavior you expected, and
3. the behavior you actually encountered (in as much detail as possible).
Thanks!

When reporting bugs in the future please use apport, either via the appropriate application's "Help -> Report a Problem" menu or using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[Geoff Silver (geoff-rootflags)]

Is there something specific you want?

1. the specific steps or actions you took that caused you to encounter the problem,

This is a web server. It started apache on boot for me when I rebooted the host a week ago. It ran for a week or so. I logged in after being out of town for the weekend and I had 6-7 apache processes running at 100% CPU, but apparently doing nothing (netstat -anop showed no connections, strace showed nothing happening, etc). There is no specific step or action I took to cause this. Presumably some external user accessed some web page which triggered what appears to be a bug in the zend library (based on the fact that this is a webserver and the gdb backtrace of the running process I supplied)

2. the behavior you expected, and

I expect apache to not use 100% of all my CPUs!

3. the behavior you actually encountered (in as much detail as possible).

I expect apache to not run at 100% CPU. I logged in after being out of town for the weekend and I had 6-7 apache processes running at 100% CPU, but apparently doing nothing (netstat -anop showed no connections, strace showed nothing happening, etc).

[Chuck Short (zulcss)]

Can you attach the output of dpkg -l | grep apache and dpkg -l | grep php. Also if possible can you test this on lucid?

Thanks
chuck

[Geoff Silver (geoff-rootflags)]

On Wed, 05 May 2010 17:18:47 -0000, Chuck Short
<email address hidden>
wrote:
> Can you attach the output of dpkg -l | grep apache and dpkg -l | grep
> php. Also if possible can you test this on lucid?
>
> Thanks
> chuck

I only have one server, so I can't test on lucid until I can upgrade it,
but here's the dpkg info.

geoff@ns:~$ dpkg -l|egrep '(apache|php|zend)'
ii apache2 2.2.12-1ubuntu2.2
             Apache HTTP Server metapackage
ii apache2-mpm-prefork 2.2.12-1ubuntu2.2
             Apache HTTP Server - traditional non-threade
ii apache2-utils 2.2.12-1ubuntu2.2
             utility programs for webservers
ii apache2.2-bin 2.2.12-1ubuntu2.2
             Apache HTTP Server common binary files
ii apache2.2-common 2.2.12-1ubuntu2.2
             Apache HTTP Server common files
ii libapache-dbi-perl 1.07-1
             Connect apache server to database via perl's
ii libapache-session-perl 1.87-1
             Perl modules for keeping persistent user dat
ii libapache2-mod-bt 0.0.19+p4.2340-1.1ubuntu1
             BitTorrent tracker for the Apache2 web serve
ii libapache2-mod-perl2 2.0.4-5ubuntu1
             Integration of perl with the Apache2 web ser
ii libapache2-mod-php5 5.2.10.dfsg.1-2ubuntu6.4
             server-side, HTML-embedded scripting languag
ii libapache2-reload-perl 0.10-2
             Reload Perl modules when changed on disk
ii libphp-adodb 5.07-1
             The ADOdb database abstraction layer for PHP
ii libphp-phpmailer 2.1-1
             full featured email transfer class for PHP
ii libphp-snoopy 1.2.4-1
             Snoopy is a PHP class that simulates a web b
ii php-auth 1.6.1-1
             PHP PEAR modules for creating an authenticat
ii php-db 1.7.13-2
             PHP PEAR Database Abstraction Layer
ii php-file 1.3.0-1
             PHP Pear modules for common file and directo
ii php-gettext 1.0.7-8
             read gettext MO files directly, without requ
ii php-http 1.4.1-1
             PHP PEAR module for HTTP related stuff
ii php-mail 1.1.14-1+lenny1build0.9.10.1
             PHP PEAR module for sending email
ii php-mail-mime 1.5.2-0.1
             PHP PEAR module for creating MIME messages
ii php-mail-mimedecode 1.5.0-3
             PHP PEAR module to decod...

[Dave Walker (davewalker)]

@Geoff, Are you still seeing this behavior?

[Geoff Silver (geoff-rootflags)]

On Sun, 10 Oct 2010 09:55:25 -0000, Dave Walker <email address hidden>
wrote:
> @Geoff, Are you still seeing this behavior?

I have not seen this since upgrading to 10.04.

[Launchpad Janitor (janitor)]

[Expired for apache2 (Ubuntu) because there has been no activity for 60 days.]