php5-fpm segfaults with error 4 in libc-2.15.so

[Alexander Pyhalov]

[Impact]

[Fix]

[Test Case]

[Regression Potential]

[Original Report]

On Ubuntu Server 12.04 on amd64 (Intel Xeon E5620 CPU) php5-fpm randomly crashes with the following message:

May 29 06:38:25 srv2 kernel: [315922.148835] php5-fpm[20902]: segfault at 0 ip 00007fecd9032558 sp 00007fff18136898 error 4 in libc-2.15.so[7fecd8f00000+1b3000]
May 29 06:38:25 srv2 kernel: [315922.148849] php5-fpm/20902: potentially unexpected fatal signal 11.
May 29 06:38:25 srv2 kernel: [315922.148851]
May 29 06:38:25 srv2 kernel: [315922.148853] CPU 7
May 29 06:38:25 srv2 kernel: [315922.148854] Modules linked in: vesafb psmouse i7core_edac edac_core ioatdma dca serio_raw joydev mac_hid lp parport usbhid hid e1000e megaraid_sas
May 29 06:38:25 srv2 kernel: [315922.148870]
May 29 06:38:25 srv2 kernel: [315922.148873] Pid: 20902, comm: php5-fpm Not tainted 3.2.0-24-generic #39-Ubuntu Supermicro X8DTT-H/X8DTT-H
May 29 06:38:25 srv2 kernel: [315922.148878] RIP: 0033:[<00007fecd9032558>] [<00007fecd9032558>] 0x7fecd9032557
May 29 06:38:25 srv2 kernel: [315922.148884] RSP: 002b:00007fff18136898 EFLAGS: 00010206
May 29 06:38:25 srv2 kernel: [315922.148887] RAX: 0000000000000000 RBX: 00007fecdb38b000 RCX: 0000000000000011
May 29 06:38:25 srv2 kernel: [315922.148890] RDX: 0000000000000066 RSI: 0000000000af8af5 RDI: 0000000000000000
May 29 06:38:25 srv2 kernel: [315922.148892] RBP: 00000000023cc228 R08: 0000000000000011 R09: 0000000000000000
May 29 06:38:25 srv2 kernel: [315922.148895] R10: 00007fecd9035750 R11: 00007fecd9081710 R12: 0000000000000001
May 29 06:38:25 srv2 kernel: [315922.148897] R13: 0000000000000000 R14: 0000000000000000 R15: 00000000023cc2f0
May 29 06:38:25 srv2 kernel: [315922.148901] FS: 00007fecdb382700(0000) GS:ffff8803332e0000(0000) knlGS:0000000000000000
May 29 06:38:25 srv2 kernel: [315922.148904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 29 06:38:25 srv2 kernel: [315922.148906] CR2: 0000000000000000 CR3: 0000000327434000 CR4: 00000000000006e0
May 29 06:38:25 srv2 kernel: [315922.148909] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 29 06:38:25 srv2 kernel: [315922.148912] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 29 06:38:25 srv2 kernel: [315922.148915] Process php5-fpm (pid: 20902, threadinfo ffff88032aeec000, task ffff88032aa55bc0)
May 29 06:38:25 srv2 kernel: [315922.148917]
May 29 06:38:25 srv2 kernel: [315922.148919] Call Trace:

GDB information:

Core was generated by `php-fpm: pool mypool '.
Program terminated with signal 11, Segmentation fault.
#0 __strstr_sse42 (s1=0x0, s2=<optimized out>) at ../sysdeps/x86_64/multiarch/strstr.c:175
175 ../sysdeps/x86_64/multiarch/strstr.c: No such file or directory.
(gdb) bt
#0 __strstr_sse42 (s1=0x0, s2=<optimized out>) at ../sysdeps/x86_64/multiarch/strstr.c:175
#1 0x0000000000736d13 in fpm_status_handle_request () at /build/buildd/php5-5.3.10/sapi/fpm/fpm/fpm_status.c:128
#2 0x000000000042b4ab in main (argc=11237155, argv=0x0) at /build/buildd/php5-5.3.10/sapi/fpm/fpm/fpm_main.c:1809

# uname -a
Linux srv2 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
# dpkg -l |grep php
ii php-apc 3.1.7-1 APC (Alternative PHP Cache) module for PHP 5
ii php5-cli 5.3.10-1ubuntu3.1 command-line interpreter for the php5 scripting language
ii php5-common 5.3.10-1ubuntu3.1 Common files for packages built from the php5 source
ii php5-curl 5.3.10-1ubuntu3.1 CURL module for php5
ii php5-dbg 5.3.10-1ubuntu3.1 Debug symbols for PHP5
ii php5-fpm 5.3.10-1ubuntu3.1 server-side, HTML-embedded scripting language (FPM-CGI binary)
ii php5-gd 5.3.10-1ubuntu3.1 GD module for php5
ii php5-mcrypt 5.3.5-0ubuntu1 MCrypt module for php5
ii php5-memcache 3.0.6-1 memcache extension module for PHP5
ii php5-mysql 5.3.10-1ubuntu3.1 MySQL module for php5

The crash occurs several times per hour and is repeatable on two different servers with the same software versions.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1006738/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Scott Moser (smoser)]

Hi,
  Are you able to reproduce this on other hardware?
  from your stack trace, it seems you're segfaulting in strstr of libc. Which would just be a very unlikely path to segfault in.
  You may want to run memcheck or some cpu burn in cycle.

[Alexander Pyhalov (alp-rsu)]

Hello.
The issue is reproducible on two different servers (however, hardware
configuration is the same).

On 05/31/2012 17:50, Scott Moser wrote:
> Hi,
> Are you able to reproduce this on other hardware?
> from your stack trace, it seems you're segfaulting in strstr of libc. Which would just be a very unlikely path to segfault in.
> You may want to run memcheck or some cpu burn in cycle.

--
Best regards,
Alexander Pyhalov,
system administrator of Computer Center of Southern Federal University

[Alexander Pyhalov (alp-rsu)]

It seems that strstr gets NULL as its first argument
(SG(request_info).query_string)
It seems that the following is incorrect in sapi/fpm/fpm/fpm_status.c:

  /* full status ? */
full = SG(request_info).request_uri &&
strstr(SG(request_info).query_string, "full");
short_syntax = short_post = NULL;
full_separator = full_pre = full_syntax = full_post = NULL;
encode = 0;

it should be
  /* full status ? */
full = SG(request_info).query_string &&
strstr(SG(request_info).query_string, "full");
short_syntax = short_post = NULL;
full_separator = full_pre = full_syntax = full_post = NULL;
encode = 0;

The bug is present also in upstream php git. Attached patch should
solve the problem.

On 05/31/2012 20:10, Scott Moser wrote:
> ** Changed in: php5 (Ubuntu)
> Status: Incomplete => New
>

--
Best regards,
Alexander Pyhalov,
system administrator of Computer Center of Southern Federal University

[Alexander Pyhalov (alp-rsu)]

Upstream bug report: https://bugs.php.net/bug.php?id=62205

--
Best regards,
Alexander Pyhalov,
system administrator of Computer Center of Southern Federal University

[Alexander Pyhalov (alp-rsu)]

The issue was fixed upstream. Attached patch (from upstream fix -
http://git.php.net/?p=php-src.git;a=patch;h=4fc989fbbd0405d200872219b409f685a495f3aa;hp=487e2fc0d50aca979864b59ff01450cf5e381874)
applies clearly to current Ubuntu php version and fixes the issue.

--
Best regards,
Alexander Pyhalov,
system administrator of Computer Center of Southern Federal University

[Alexander Pyhalov (alp-rsu)]

The issue was fixed upstream month ago. Will be upstream fix applied it in Ubuntu?

[Clint Byrum (clint-fewbar)]

It was only released a few days ago. The patch looks simple enough, so we'll try and incorporate it in the next upload to quantal. I think this one is probably worth fixing in precise as well.

[Thomas Ward (teward)]

(1) Matching importance level to the Quantal bug. (Precise: Undecided -> High)

(2) Changing to "In progress", and assigning bug to myself for debdiff generation, and SRU-ing

[Thomas Ward (teward)]

The debdiff for both this bug (LP Bug 1006738), and LP Bug 1014044 for Precise is attached to LP Bug 1014044.

This was generated based off of the package in precise-security and precise-updates, php5 5.3.10-1ubuntu3.2.

This debdiff includes patches based upstream for both bugs, and includes a new changelog entry stating the patches were applied. It does not give extremely detailed information about every change done by those patches.

*** The patch used for this bug originated upstream. This patch was already applied upstream, and will be included in Quantal after the version of php5 shows up in Debian (php 5.4.x chain). ***

[Thomas Ward (teward)]

Attached is a Quantal debdiff, based off of Quantal package php5_5.4.4-3ubuntu1.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Quantal debdiff with Upstream Patch for LP #1006738" of this bug report has been identified as being a patch in the form of a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Ondřej Surý (ondrej)]

Better would be to pull 5.4.4-4 which already includes more complete fix from upstream git.

[Launchpad Janitor (janitor)]

This bug was fixed in the package php5 - 5.4.6-1ubuntu1

---------------
php5 (5.4.6-1ubuntu1) quantal; urgency=low

  * Merge from Debian experimental (LP: #1006738 , LP: #1040212)
    Remaining changes:
    - d/rules: Simplify apache config settings since we never build
      interbase or firebird.
    - debian/rules: export DEB_HOST_MULTIARCH properly.
    - Add build-dependency on lemon, which we now need.
    - Dropped firebird2.1-dev, libc-client-dev, libmcrypt-dev as it is
      in universe.
    - Dropped libcurl-dev not in the archive.
    - debian/control: replace build-depends on mysql-server with
      mysql-server-core-5.5 and mysql-client-5.5 to avoid upstart and
      mysql-server-5.5 postinst confusion with starting up multiple
      mysqlds listening on the same port.
    - Dropped php5-imap, php5-interbase, php5-mcrypt since we have
      versions already in universe.
    - Dropped libonig-dev and libqgdbm since its in universe. (libonig
      MIR has been declined due to an inactive upstream. So this is
      probably a permanent change).
    - modulelist: Drop imap, interbase, sybase, and mcrypt.
    - debian/rules:
      - Dropped building of mcrypt, imap, and interbase.
      - Install apport hook for php5.
      - stop mysql instance on clean just in case we failed in tests
    - debian/control, debian/rules: Re-enable libedit-dev.
  * Dropped Changes:
    - debian/rules: change memory limits on example .ini files.

php5 (5.4.6-1) experimental; urgency=low

  * Imported Upstream version 5.4.6
  * Apply another fix to compile --without-system-tzdata
    (Courtesy of Michael Heimpold)
  * Get rid of empty examples directory (Closes: #684108), but
    keep parent directory to store test-results.txt among others
  * Provide sensible default configuration for PHP-CGI files
    (Closes: #685340)
  * Add NEWS text about default extension configuration
  * Update NEWS and README.Debian based on debian-l10n-english review
    (Courtesy of Justing B Rye)

php5 (5.4.5-1) experimental; urgency=low

  * Imported Upstream version 5.4.5
  * Update patches for PHP 5.4.5 release
  * Compile with system libzip (upstream has added support for that)

php5 (5.4.4-4) unstable; urgency=low

  * Fix php5-fpm segfault (PHP#62205)
  * CVE-2012-2688: potential overflow in _php_stream_scandir
    (Closes: #683274)
  * Improve security in CGI section in README.Debian (Closes: #674205)
 -- Clint Byrum <email address hidden> Wed, 22 Aug 2012 13:40:18 -0700

[Adam Conrad (adconrad)]

Hello Alexander, or anyone else affected,

Accepted php5 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/php5/5.3.10-1ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Alexander Pyhalov (alp-rsu)]

The suggested package from proposed repository fixes the problem for me.
Thank you for work.

[Adam Conrad (adconrad)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package php5 - 5.3.10-1ubuntu3.3

---------------
php5 (5.3.10-1ubuntu3.3) precise-proposed; urgency=low

  * Applies upstream bug fixes for several issues and bugs:
    * php5-fpm segfaults with error 4 in libc-2.15.so
        (LP: #1006738. Bug Priority: High)
    * PHP5-FPM not reporting errors to web server (nginx)
        (LP: #1014044. Bug Priority: Medium)
 -- Thomas Ward <email address hidden> Tue, 31 Jul 2012 21:15:08 -0400