php5 opens /proc/XXX/auxv thousands of times until Apache crashes with "too many open files"

[Jens]

Hello,

PHP5, or rather "php-cgi", seems to open /proc/$PID/auxv a lot and doesn't close all file descriptors when restarting with "apachectl graceful", until eventually, the system runs out of file descriptors.

After about four weeks of uptime my Linux box (Ubuntu 12.04.1) had over 90'000 opened files, almost all of them were the above. Then Apache refused "apachectl graceful" because Linux could not open all files any more.

After restarting Apache, all files were closed again. Now, the file descriptors are leaking again - much more slowly because my cron scripts use "apachectl restart" now instead of "apachectl graceful", but still. After two weeks, I have roughly 15'000 opened files, of which 8'000 are /proc/$PID/auxv.

I think this is quite a serious bug since it makes the system grind to a halt eventually.

Thank you for investigating!

Regards,
Jens

PS: "lsof" shows it like this:

php-cgi 32268 sdm 77r REG 0,3 0 690669699 /proc/20964/auxv
php-cgi 32268 sdm 78r REG 0,3 0 690669699 /proc/20964/auxv
php-cgi 32268 sdm 79r REG 0,3 0 690669699 /proc/20964/auxv
php-cgi 32268 sdm 80r REG 0,3 0 690669699 /proc/20964/auxv
...
php-cgi 32268 sdm 281r REG 0,3 0 690669699 /proc/20964/auxv

for each php-cgi process. 20964 is the "apache2" main process.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: php5 5.3.10-1ubuntu3.2
ProcVersionSignature: Ubuntu 3.2.0-24.37-generic 3.2.14
Uname: Linux 3.2.0-24-generic x86_64
ApportVersion: 2.0.1-0ubuntu8
Architecture: amd64
Date: Wed Jul 11 22:13:27 2012
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-color
 PATH=(custom, no user)
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
SourcePackage: php5
UpgradeStatus: Upgraded to precise on 2012-05-04 (68 days ago)

[Clint Byrum (clint-fewbar)]

Hi Jens, thanks for taking the time to file a bug report. I'm having trouble reproducing this issue. When I install apache2 and php5-cgi and copy a simple php file to /usr/lib/cgi-bin, I don't get any opening of /proc files at all. This is the list I see when cgi is forked (the parent process doesn't open any new files on requests):

[pid 30702] open("/usr/local/lib/tls/x86_64/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/local/lib/tls/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/local/lib/x86_64/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/local/lib/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/tls/x86_64/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/tls/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/x86_64/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid 30702] open("/lib/x86_64-linux-gnu/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = 3
[pid 30702] open("/usr/local/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/lib/x86_64-linux-gnu/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
[pid 30702] open("/usr/local/lib/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/lib/x86_64-linux-gnu/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
[pid 30702] open("/usr/local/lib/libdb-5.1.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/libdb-5.1.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/x86_64-linux-gnu/libdb-5.1.so", O_RDONLY|O_CLOEXEC) = 3
[pid 30702] open("/usr/local/lib/libbz2.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/libbz2.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/lib/x86_64-linux-gnu/libbz2.so.1.0", O_RDONLY|O_CLOEXEC) = 3
[pid 30702] open("/usr/local/lib/libpcre.so.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/libpcre.so.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
[pid 30702] open("/usr/local/lib/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/usr/lib/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 30702] open("/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
[pid 30702] open("/usr/local/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directo...

[Jens (jens-launchpad-net)]

I'm running PHP as a FCGI (FastCGI) module to isolate my user environments from each other properly.

No specific script causes this behaviour, I have multiple "handwritten" PHP scripts as well as Typo3 and Mambo / Joomla / whatever processes running and they all collect open file descriptors roughly at the same speed.

Here is an example virtualhost configuration:

<VirtualHost xx.xxx.xxx.xx:80>
  ServerName foobar.name
  ServerAlias *.foobar.name
  ServerAdmin <email address hidden>
  DocumentRoot "/xxxxx/webs/jens/name.foobar/"
  FcgidIdleTimeout 30
  SuexecUserGroup "jens" "jens"
  <Directory "/xxxxx/webs/jens/name.foobar/">
    <FilesMatch "\.(php)$">
      SetHandler fcgid-script
      FCGIWrapper /xxxxx/jens/foobar.name/php-fcgi-starter .php
      Options +ExecCGI
    </FilesMatch>
    Order allow,deny
    allow from all
  </Directory>
  Alias /awstats "/xxxxx/webs/jens/awstats/foobar.name"
  Alias /awstats-icon "/usr/share/awstats/icon/"
  ErrorLog "/xxxxx/logs/jens-foobar.name-error.log"
  CustomLog "/xxxxx/logs/jens-foobar.name-access.log" combined
</VirtualHost>

Here is the referenced (autogenerated) "php-fcgi-starter" script:

umask 022
PHPRC='/xxxxx/jens/foobar.name/'
export PHPRC
PHP_FCGI_CHILDREN=0
export PHP_FCGI_CHILDREN
PHP_FCGI_MAX_REQUESTS=250
export PHP_FCGI_MAX_REQUESTS
exec /usr/bin/php-cgi -c '/xxxxx/jens/foobar.name/'

I will read up on FPM, thank you for the info. But I still regard this as a bug ;)

Cheers
Jens

[Clint Byrum (clint-fewbar)]

Thanks Jens, I'll set the status back to New and hopefully take some time next week to try and reproduce the problem with your configs.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[alder (alexmi)]

I reckon https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1177684 is related. If so, Clint cannot reproduce it because he isn't using the gd module.

Cheers,
Alessandro