php 5.2.6 'random' segfault

[Pieter Steyn]

Binary package hint: php5-cli

I'm getting random php segfaults in my /var/log/messages file:

ug 28 09:28:02 www kernel: [122051.459511] php[15082]: segfault at 7f2459eecf3b ip 00007f2459c8a4ef sp 00007f2457ff3c60 error 4 in libgcc_s.so.1[7f2459c7a000+16000]
Aug 28 09:47:01 www kernel: [123191.280575] php[21125]: segfault at 7fd16f10ef30 ip 00007fd16f10ef30 sp 00007fd16d2160f8 error 14 in libuuid.so.1.2[7fd16fb36000+3000]
Aug 28 10:14:07 www -- MARK --
Aug 28 10:16:01 www kernel: [124931.139171] php[29406]: segfault at 7f16486c4f30 ip 00007f16486c4f30 sp 00007f16467cc0f8 error 14 in librt-2.9.so[7f164da68000+7000]

These are most likely triggered by some of my cron jobs which are php scripts, but it seems completely random. Sometime the script will complete without segfaulting, other times it will fault.

I've never had these segfaults on the Centos 5.3 machine I migrated from.

Here is an example of how to reproduce:

<?php

$a = 2;
$b = 1;

$c = $a ^ $b;

var_dump($c);
?>

Run this script on the command line, and once out of about 30 times it will segfault.

Here is a link to someone else who had the same problem a few months ago but found no solution:

http://osdir.com/ml/ubuntu-users/2009-06/msg02580.html

Some details:
root@www:~# lsb_release -rd
Description: Ubuntu 9.04
Release: 9.04

root@www:~# uname -a
Linux www.servername.com 2.6.28-15-server #49-Ubuntu SMP Tue Aug 18 20:09:37 UTC 2009 x86_64 GNU/Linux

root@www:~# apt-cache policy php5-cli
php5-cli:
  Installed: 5.2.6.dfsg.1-3ubuntu4.2
  Candidate: 5.2.6.dfsg.1-3ubuntu4.2
  Version table:
 *** 5.2.6.dfsg.1-3ubuntu4.2 0
        500 http://za.archive.ubuntu.com jaunty-updates/main Packages
        500 http://security.ubuntu.com jaunty-security/main Packages
        100 /var/lib/dpkg/status
     5.2.6.dfsg.1-3ubuntu4 0
        500 http://za.archive.ubuntu.com jaunty/main Packages

root@www:~# php -m:

[PHP Modules]
bcmath
bz2
calendar
ctype
curl
date
dba
dom
exif
filter
ftp
gd
gettext
gmp
hash
iconv
imagick
json
libxml
mbstring
mime_magic
mssql
mysql
mysqli
ncurses
openssl
pcntl
pcre
PDO
pdo_dblib
pdo_mysql
pdo_pgsql
pgsql
posix
pspell
readline
Reflection
session
shmop
SimpleXML
soap
sockets
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xml
xmlreader
xmlrpc
xmlwriter
zip
zlib

[Zend Modules]

Hope this is enough information,

Thanks,
Pieter

[Niels Peen (niels-peen)]

Adding a 'me too'. These are PHP cronjobs that run every few minutes. Most of the times nothing is wrong, occasionally one will segfault. Machine load is low.

2.6.28-14-server #47-Ubuntu SMP Sat Jul 25 01:18:34 UTC 2009 i686 GNU/Linux

[1667242.984012] feeder.php[22564]: segfault at b703c760 ip b703c760 sp b6d7039c error 14 in libpam.so.0.81.12[b7271000+a000]
[1671023.312542] runDB[24828]: segfault at b7010760 ip b7010760 sp b6d4439c error 14 in librt-2.9.so[b77e4000+7000]
[1672043.050297] runDB[25416]: segfault at b6f4e760 ip b6f4e760 sp b6c8239c error 14 in librt-2.9.so[b7722000+7000]
[1672703.042364] feeder.php[25821]: segfault at b6fb4760 ip b6fb4760 sp b6ce839c error 14 in librt-2.9.so[b7788000+7000]
[1675703.352735] uploader.php[27630]: segfault at b6f28760 ip b6f28760 sp b6c5c39c error 14 in librt-2.9.so[b76fc000+7000]
[1677502.632613] runDB[28702]: segfault at b6fdd760 ip b6fdd760 sp b6d1139c error 14 in librt-2.9.so[b77b1000+7000]
[1683442.852024] feeder.php[32316]: segfault at b6ea4760 ip b6ea4760 sp b6bd839c error 14 in librt-2.9.so[b7678000+7000]
[1684702.464171] runDB[603]: segfault at b6f5877e ip b7828816 sp b6c8c088 error 4 in libgcc_s.so.1[b7821000+d000]
[1689022.976655] feeder.php[3551]: segfault at b6ef1760 ip b6ef1760 sp b6c2539c error 14 in librt-2.9.so[b76c5000+7000]
[1699163.412581] feeder.php[9710]: segfault at b6fc6760 ip b6fc6760 sp b6cfa39c error 14 in librt-2.9.so[b779a000+7000]
[1699462.460828] feeder.php[9918]: segfault at b6ef2760 ip b6ef2760 sp b6c2639c error 14 in librt-2.9.so[b76c6000+7000]
[1699942.970096] feeder.php[10201]: segfault at b6e9c760 ip b6e9c760 sp b6bd039c error 14 in librt-2.9.so[b7670000+7000]
[1700123.420937] runDB[10290]: segfault at b6f61760 ip b6f61760 sp b6c9539c error 14 in librt-2.9.so[b7735000+7000]
[1706783.311784] uploader.php[14445]: segfault at b6ed2760 ip b6ed2760 sp b6c0639c error 14 in librt-2.9.so[b76a6000+7000]
[1707263.200251] runDB[14720]: segfault at b6f35760 ip b6f35760 sp b6c6939c error 14 in librt-2.9.so[b7709000+7000]
[1708343.275339] feeder.php[15354]: segfault at b6efa760 ip b6efa760 sp b6c2e39c error 14 in librt-2.9.so[b76ce000+7000]
[1709962.970937] feeder.php[16463]: segfault at b6e58760 ip b6e58760 sp b6b8c39c error 14 in libgpg-error.so.0.3.0[b74ea000+3000]
[1710682.748884] runDB[16882]: segfault at b6f9f760 ip b6f9f760 sp b6cd339c error 14 in librt-2.9.so[b7773000+7000]
[1712003.024355] runDB[17719]: segfault at b6ecd760 ip b6ecd760 sp b6c0139c error 14 in librt-2.9.so[b76a1000+7000]
[1714282.577907] feeder.php[19153]: segfault at b6ef3760 ip b6ef3760 sp b6c2739c error 14 in librt-2.9.so[b76c7000+7000]
[1715243.422550] runDB[19761]: segfault at b6f4d760 ip b6f4d760 sp b6c8139c error 14 in librt-2.9.so[b7721000+7000]
[1717822.629017] runDB[21484]: segfault at b6f7c760 ip b6f7c760 sp b6cb039c error 14 in librt-2.9.so[b7750000+7000]
[1718542.894834] runDB[21957]: segfault at b6f29760 ip b6f29760 sp b6c5d39c error 14 in librt-2.9.so[b76fd000+7000]
[1718662.454723] feeder.php[22031]: segfault at b6f4c760 ip b6f4c760 sp b6c8039c error 14 in librt-2.9.so[b7720000+7000]
[1719802.590656] feeder.php[22767]: segfault at b6e5a77e ip b7732816 sp b6b8e088 error 4 in libgcc_s....

[Chuck Short (zulcss)]

Anything in your apache log files?

Regards
chuck

[Pieter Steyn (pieterste)]

No, nothing related - but I still think it's caused by php-cli from the cron jobs...

[Pieter Steyn (pieterste)]

Sorry, also by changing the status to incomplete what do you need from me? I stated in the initial report that it's caused by php-cli, so me not providing apache logs is irrelevant imo.

[Chuck Short (zulcss)]

If possible can you try the newer version I uploaded earlier this week?

Thanks
chuck

[Pieter Steyn (pieterste)]

Sure, I would like to try. How would I though?

I've enabled the backports repositoriry and did an apt-get update but still don't see any PHP upgrades available for my system:

Ubuntu Server 64bit
Release: 9.04

[Carl Friis-Hansen (carl-friis-hansen)]

I noticed the same problem on 9.04 32 bit headless server version on 64 bit dual core machine:
///////////////////////////////////////////////////////////////////
Oct 1 22:35:35 fiskeryd kernel: [117903.467192] php-cgi[1167]: segfault at b6dc4760 ip b6dc4760 sp b6d3639c error 14 in librt-2.9.so[b70dd000+7000]
Oct 1 22:35:35 fiskeryd kernel: [117903.543668] php-cgi[1174]: segfault at b6d3c760 ip b6d3c760 sp b6cae39c error 14 in libpam.so.0.81.12[b6f1d000+a000]
///////////////////////////////////////////////////////////////////

The log for suphp shows for this unique time:
///////////////////////////////////////////////////////////////////
[Thu Oct 01 22:35:35 2009] [info] Executing "/var/www/clients/client1/web8/web/webcam/showimg.php" as UID 5009, GID 5003
[Thu Oct 01 22:35:35 2009] [info] Executing "/var/www/clients/client1/web8/web/webcam/showimg.php" as UID 5009, GID 5003
///////////////////////////////////////////////////////////////////

There is nothing interesting in Apache2 logs.

However, the showimg.php shows images from from one of many webcams and looks like this:
///////////////////////////////////////////////////////////////////
  if( $_REQUEST["img_nocam"] ) {
    if( file_exists( $_REQUEST["ftp_dir"] . "/nocam.jpg" ) ) {
      $nocam = $_REQUEST["ftp_dir"] . "/nocam.jpg";
    } else {
      $nocam = "nocam.png";
    }
  } else {
    $nocam = "nocam.png";
  }

  if( $_REQUEST["img_type"] != "nohead" ) {
    header("Content-type: image/" . $_REQUEST["img_type"] );
    header("Cache-control: no-cache");
    header("Pragma: no-cache");
  }
  if( $_REQUEST["img_file"] && !$_REQUEST["domain"] && !$_REQUEST["port"] ) {
    //
    // Only a URL is supplied.
    // So, let's just show this to the user.
    //
    if( $_REQUEST["img_file"] == monitor1 ) {
      $_REQUEST["img_file"] = "http://192.168.0.151/cgi-bin/nph-zms?mode=single&monitor=1&scale=100";
    } else if( $_REQUEST["img_file"] == monitor2 ) {
      $_REQUEST["img_file"] = "http://192.168.0.151/cgi-bin/nph-zms?mode=single&monitor=2&scale=100";
    }
    if( $f = fopen( $_REQUEST["img_file"], "r" ) ) {
      fpassthru( $f );
      fclose( $f );
    } else {
      //
      // The actual URL didn't work.
      // So, let's show the user this.
      //
      @readfile( $nocam );
    }
  } else if( $_REQUEST["img_file"] && $_REQUEST["domain"] && $_REQUEST["port"] &&
              $fp = @fsockopen( $_REQUEST["domain"], $_REQUEST["port"], $errnum, $errstr, $_REQUEST["socktime"] ) ) {
    fclose($fp);
    //
    // The above test proves that a connection to the above server is possible within the time frame "socktime".
    // So, let's try with the actual URL.
    //
    if( $_REQUEST["img_type"] != "nohead" ) {
      if( ! @readfile( $_REQUEST["img_file"] ) ) {
        //
        // The actual URL didn't work.
        // So, let's show the user this.
        //
        @readfile( $nocam );
      }
    } else {
      if( $f = fopen( $_REQUEST["img_file"], "r" ) ) {
        fpassthru( $f );
        fclose( $f );
      } else {
        //
        // The actual URL didn't work.
        // So, let's show the user this.
        //
        @readfile( $nocam );
      }
   ...

[Chuck Short (zulcss)]

If possible can you add a backtrace?

https://wiki.ubuntu.com/DebuggingProgramCrash

Thanks
chuck

[Chuck Short (zulcss)]

We'd like to figure out what's causing this bug for you, but we haven't heard back from you in a while. Could you please provide the requested information? Thanks!

[Pieter Steyn (pieterste)]

Hi,

Unfortunately I cannot, as the bug seems to be random. I can't reproduce at will.

I have upgraded the server to 9.10, and unfortunately these segfaults are still happening:

Nov 15 22:03:01 www kernel: [290867.628175] php[9268]: segfault at 7f054294d140 ip 00007f054294d140 sp 00007f0541a570e8 error 14 in librt-2.10.1.so[7f05457b8000+7000]
Nov 15 22:36:01 www kernel: [292847.213232] php[15503]: segfault at 7f3007d48140 ip 00007f3007d48140 sp 00007f3006e520e8 error 14 in librt-2.10.1.so[7f300abb3000+7000]
Nov 15 22:51:01 www kernel: [293747.192936] php[18251]: segfault at 7ff53d442140 ip 00007ff53d442140 sp 00007ff53c54c0e8 error 14 in librt-2.10.1.so[7ff5402ad000+7000]
Nov 15 22:54:01 www kernel: [293927.350036] php[18711]: segfault at 7fbf9f6f0140 ip 00007fbf9f6f0140 sp 00007fbf9e7fa0e8 error 14 in libgpg-error.so.0.4.0[7fbfa1a12000+3000]
Nov 16 01:35:01 www kernel: [303587.746765] php[13568]: segfault at 7f9ffe710140 ip 00007f9ffe710140 sp 00007f9ffd81a0e8 error 14 in librt-2.10.1.so[7fa00157b000+7000]
Nov 16 04:15:02 www kernel: [313188.102502] php[32091]: segfault at 7fd1c6d86140 ip 00007fd1c6d86140 sp 00007fd1c5e900e8 error 14 in librt-2.10.1.so[7fd1c9bf1000+7000]
Nov 16 09:20:01 www kernel: [331487.740127] php[31943]: segfault at 7fea8b92a140 ip 00007fea8b92a140 sp 00007fea8aa340e8 error 14 in librt-2.10.1.so[7fea8e795000+7000]
Nov 16 09:50:01 www kernel: [333288.023970] php[26424]: segfault at 7fed516e3140 ip 00007fed516e3140 sp 00007fed507ed0e8 error 14 in libXdmcp.so.6.0.0[7fed51caa000+5000]
Nov 16 11:50:02 www kernel: [340488.086574] php[25747]: segfault at 7f8e8b0f9140 ip 00007f8e8b0f9140 sp 00007f8e8a2030e8 error 14 in librt-2.10.1.so[7f8e8df64000+7000]
Nov 16 12:40:01 www kernel: [343487.155872] php[6534]: segfault at 7f5fd7ad9140 ip 00007f5fd7ad9140 sp 00007f5fd6be30e8 error 14 in librt-2.10.1.so[7f5fda944000+7000]

I understand it's not much to work with though. :/

Regards,
Pieter

[sl45sms (sl45sms)]

Also have the same random segfault's

Nov 29 10:32:31 www kernel: [1001023.990633] php-cgi[8018]: segfault at 7f5d059f5140 ip 00007f5d059f5140 sp 00007f5d0514b0e8 error 14 in librt-2.10.1.so[7f5d07173000+7000]
Nov 29 18:53:47 www kernel: [1031099.580485] php-cgi[21374]: segfault at 7fcf033cb140 ip 00007fcf033cb140 sp 00007fcf02b210e8 error 14 in librt-2.10.1.so[7fcf04b49000+7000]

[kallisti5 (kallisti5)]

Throwing a me too in there for Jaunty.

messages.0:Jan 11 10:18:01 homestar kernel: [7500063.231430] php[22259]: segfault at b7165760 ip b7165760 sp b704a39c error 14 in librt-2.9.so[b7795000+7000]
messages.0:Jan 11 18:42:01 homestar kernel: [7530303.011300] php[32480]: segfault at b6fe2760 ip b6fe2760 sp b6ec739c error 14 in librt-2.9.so[b7612000+7000]
messages.0:Jan 12 19:18:01 homestar kernel: [7618863.250701] php[10402]: segfault at b718f760 ip b718f760 sp b707439c error 14 in librt-2.9.so[b77bf000+7000]
messages.0:Jan 14 12:54:02 homestar kernel: [7768623.870568] php[23386]: segfault at b7029760 ip b7029760 sp b6f0e39c error 14 in librt-2.9.so[b7659000+7000]
messages.0:Jan 15 19:48:02 homestar kernel: [7879863.811327] php[15973]: segfault at b7142760 ip b7142760 sp b702739c error 14 in librt-2.9.so[b7785000+7000]

root@homestar:/var/log# php -v
PHP 5.2.6-3ubuntu4.5 with Suhosin-Patch 0.9.6.2 (cli) (built: Jan 6 2010 22:25:33)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend Technologies

I've been trying to get the basic script provided to segfault but after 10,000 runs it doesn't. It seems the errors only occur when php is run through cron.

[karpa (k-valhalla-age)]

simple code triggers sigserv:
while true; do php -r ""; done
) wait some time until you get Segmentation fault (core dumped)
gdb says:
Core was generated by `php -r '.
Program terminated with signal 11, Segmentation fault.
#0 0xb67481a0 in ?? ()
(gdb) bt
#0 0xb67481a0 in ?? ()
#1 0xb71f98de in clone () from /lib/tls/i686/cmov/libc.so.6

ubuntu 9.10 up-to-date

uname -a
Linux megahost 2.6.31-19-generic-pae #56-Ubuntu SMP Thu Jan 28 02:29:51 UTC 2010 i686 GNU/Linux

php -m
[PHP Modules]
bcmath
bz2
calendar
ctype
curl
date
dba
dom
exif
filter
ftp
gd
gettext
hash
iconv
json
libxml
mbstring
mcrypt
mime_magic
mssql
mysql
mysqli
ncurses
openssl
pcntl
pcre
PDO
pdo_dblib
pdo_mysql
posix
readline
Reflection
session
shmop
SimpleXML
soap
sockets
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xml
xmlreader
xmlwriter
zip
zlib

[Zend Modules]

php -v
PHP 5.2.10-2ubuntu6.4 with Suhosin-Patch 0.9.7 (cli) (built: Jan 6 2010 22:41:56)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2009 Zend Technologies