apache2 crashed with SIGSEGV in pdo_parse_params()

[imm6]

Binary package hint: apache2

This crash occurred in a PHP application that uses the Zend framework. I can always make it crash but it will not crash consistently for the same SQL call. Sometimes the idential call works fine and sometimes it causes a crash. The application is importing data so it is issuing many SQL statements rapidly.

ProblemType: Crash
Architecture: amd64
CrashCounter: 1
DistroRelease: Ubuntu 9.04
ExecutablePath: /usr/sbin/apache2
NonfreeKernelModules: nvidia
Package: apache2-mpm-prefork 2.2.11-2ubuntu2.3
ProcCmdline: /usr/sbin/apache2 -k start
ProcEnviron:
 PATH=(custom, no user)
 LANG=C
Signal: 11
SourcePackage: apache2
StacktraceTop:
 pdo_parse_params (stmt=0x7fb524c8abf8,
 zim_PDOStatement_execute (ht=720694112,
 xdebug_execute_internal (
 zend_do_fcall_common_helper_SPEC (
 execute (op_array=0x7fb5247b5448)
Title: apache2 crashed with SIGSEGV in pdo_parse_params()
Uname: Linux 2.6.28-15-generic x86_64
UserGroups:

[Ondřej Surý (ondrej)]

I guess this could be same bug as in http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=402181

The solution is also same, ask your vendor to fix binary compatibility or don't use packaged php5.

[Chuck Short (zulcss)]

Thanks for the bug report but what program is causing this crash?

Thanks
chuck

[imm6 (imm6)]

I've been investigating this and am not sure this is the same a Debian bug 402181 as referenced above. That bug references Zend Optimizer, which I am not using. My understanding is that the Zend Optimizer is a binary program which is causing a conflict in that bug. The application I am using is Zend Framework, which is simply a collection of PHP files, and does not have any compiled component, as far as I'm aware at least.

The program that is actually using the Zend Framework is Magento, which is a PHP e-commerce package. It also does not have any compiled components but is just PHP.

I am definitely willing to try different things to debug this if someone can point me in the proper direction. Thanks.

[Ondřej Surý (ondrej)]

Hi,

where does this:

#2 0x00007fb51a1354d2 in xdebug_execute_internal (
    current_execute_data=0x7fff2af4ef70, return_value_used=1)
    at /build/buildd/xdebug-2.0.3/build-php5/xdebug.c:1605
 edata = (zend_execute_data *) 0x7fff2af4ef70
 fse = (function_stack_entry *) 0x7fb5246ab4c0
 cur_opcode = (zend_op *) 0x1
 do_return = 0
 function_nr = 266972

come from? (Is that packaged php5-xdebug? What version?)

Is the bug reproducible?

Could you try disabling xdebug, and trying to reproduce the problem?

Or could you try setting up karmic virtual machine and trying to reproduce it there?

Ondrej

[imm6 (imm6)]

Hello Ondrej,

Thanks for the response. Yes, that line was from package php5-xdebug. I had tried using that package to help track down the bug. I uninstalled it and the crash still occurs in the same way, only the stacktrace now doesn't include the xdebug part of course.

The bug is reproducible in that I created a simple test case which causes the crash about 90% of the time. I'm not sure why the other 10% of the time the crash does not occur.

Yesterday I created a Jaunty Server virtual machine and tried my test and the crash did not occur under the virtual machine. I am currently trying to figure out what, if any, differences there are between my physical computer and the virtual machine, other than my physical machine is running Jaunty (plain) and the other is Jaunty Server. All the PHP and Apache packages are the same version. Thanks.

[Ondřej Surý (ondrej)]

Hmm, could it be related to https://bugs.launchpad.net/ubuntu/+source/php5/+bug/343870 ? Do you use mysql?

[imm6 (imm6)]

First of all, thank you so much for your help Ondrej! At this point I think it is probably this bug. I tried running the test cases described in the bug ( http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=524366 ) from the CLI and indeed got a segmentation fault in the one other people were getting a seg fault in and the one that was supposed to complete fine did indeed complete fine. I then tried upgrading the libmysqlclient15off library to the upstream debian version that was supposed to fix the bug by grabbing it from:

http://ftp.fr.debian.org/debian/pool/main/m/mysql-dfsg-5.0/libmysqlclient15off_5.0.51a-24+lenny2+spu1_amd64.deb

I then reran the test case the bug tracker provided and it did not crash any more. BUT before I could test my apache PHP test case, Ubuntu's package manager upgraded some packages and subsequently went on to delete mysql, apache, and a bunch of packages that depend on mysql from my computer! Apparently since the version number for the newly install libmysqlclient15off was 5.0.51a-24+lenny2+spu1 and the "current" Ubuntu version is 5.1.30really5.0.75-0ubuntu10.2 it thought it needed to remove the one I installed and upgrade it. I'm not sure why it also deleted all these other packages without telling me first. So now I think I have all the proper packages reinstalled correctly, but I don't want that to happen again. Before I can test apache I need to figure out the proper way of installing this package so that Ubuntu doesn't freak out.

[imm6 (imm6)]

I was able to get the test libmysqlclient15off package properly installed and I tested with it and confirmed that that did fix the case reported in bug 524366, BUT unfortunately it did not fix my bug as I've reported here. It happens in the exact same way. So if it is actually caused by the same bug as https://bugs.launchpad.net/ubuntu/+source/php5/+bug/343870 then that fix doesn't work on my bug, otherwise it is a different bug. Since that bug was related to a race condition that affected multiple cores, I disabled all but one core using the "maxcpus" kernel boot option and the bug still occurred.

[imm6 (imm6)]

I have been analyzing the differences between my Jaunty server which has the segmentation fault problem, and a Jaunty virtual machine which works fine. I've determined the key difference:

The problem machine has package php5-mysql (version 5.2.6.dfsg.1-3ubuntu4.2) installed. To eliminate the segmentation fault, I installed a different (apparently older) version of pdo and pdo_mysql by doing "sudo pecl install pdo_mysql" which overwrote Ubuntu's installed pdo and pdo_mysql libraries.

These modified libraries are found at /usr/lib/php5/20060613/pdo.so and /usr/lib/php5/20060613/pdo_mysql.so.

Key file version differences:

pdo.c: BAD: 1.57.2.17.2.10.........GOOD: 1.57.2.17
pdo_dbh.c: BAD: 1.82.2.31.2.22.........GOOD: 1.82.2.30
pdo_sql_parser.c: BAD: 1.35.2.6.2.15 ...........GOOD: 1.35.2.6
pdo_sql_parser.re: BAD: 1.28.2.4.2.12...........GOOD: 1.28.2.4
pdo_sqlstate.c: BAD: 1.7.2.1.2.2..............GOOD: 1.7.2.1
pdo_stmt.c: BAD: 1.118.2.38.2.34............GOOD: 1.118.2.38
php_pdo_driver.h: BAD: 1.66.2.11.2.8...........GOOD: 1.66.2.11
php_pdo.h: BAD: 1.7.2.5.2.4...............GOOD: 1.7.2.5

mysql_driver.c: BAD: 1.59.2.13.2.6...........GOOD: 1.59.2.13
mysql_statement.c: BAD: 1.48.2.14.2.7...........GOOD: 1.48.2.14

So the file versions are very close so there wasn't much changed between them. It looks like the newer version either had a bug introduced that causes this segmentation fault, or it is also possible that the way the Ubuntu package was compiled is causing a problem I suppose? I don't know much about the internals of this library so is there someone who is familiar with the internals of PDO comment? Thanks.

Nick

[imm6 (imm6)]

I just built the pdo.so and pdo_mysql.so library files from the Ubuntu source package by doing "apt-get -b source php5-mysql" and then copying those 2 .so files to /usr/lib/php5/20060613 and restarted Apache. The segmentation fault happened in the same way. This seems to indicate that there was a bug introduced with this newer version and the problem probably wasn't with the way the package was compiled.

[Ondřej Surý (ondrej)]

Could you perhaps isolate this to source code diff?

O.

[imm6 (imm6)]

I traced the problem to a bug in ext/pdo/pdo_sql_parser.c that must have been introduced somewhere around 5.2.6. I started working on a fix but then thought to replace that file with one from a newer version of PHP. I tried replacing with the one from 5.2.10.dfsg.1-2ubuntu4 and the bug had already been fixed in that version. It looks like a fix was submitted almost a year ago in bug: http://bugs.php.net/bug.php?id=44251

I've attached a diff of the changes between 5.2.6.dfsg.1-3ubuntu4.2 and 5.2.10.dfsg.1-2ubuntu4 in case you want them, but you may consider this already fixed since it is at least fixed in 5.2.10 which will come out in Karmic. Using the Karmic package works fine for me, but it would probably be nice if it was backported to Jaunty for other people. Thanks for your help,

Nick

[Ondřej Surý (ondrej)]

It's not related to the Debian bug mentioned. See the reporter's comment below.