hoary: intermittent SIGSEGV from php pages after upgrade to 4.3.10-10ubuntu4.5

[James Manning]

Binary package hint: libapache2-mod-php4

Early this morning (2006-07-20), I got the security update for php4 (4.3.10-10ubuntu4.5) including libapache2-mod-php4. Since then, a lot of my gallery2 (http://gallery2.sf.net/) pages SIGSEGV.

Since this is a security update that's caused the SIGSEGV (admittedly, this is based on timing, and the actual cause of the SIGSEGV's may not be the security update), I'm checking the "security vulnerability" flag (it's at least arguably a DoS, as my gallery is now only intermittently available).

On a related note, it appears that security.ubuntu.com doesn't keep around older packages, as I don't see a route that lets me downgrade to php4=4:4.3.10-10ubuntu4.4 so if someone could point out where I could get those older bits, that'd be great as both a workaround and confirmation that it's the fix in the *4.5 bits that's causing this on my system.

Here's an strace from one death:

...
fstat64(17, {st_mode=S_IFREG|0644, st_size=6147, ...}) = 0
mmap2(NULL, 6147, PROT_READ, MAP_SHARED, 17, 0) = 0xb75db000
munmap(0xb75db000, 6147) = 0
close(17) = 0
access("/var/www/gallery2/g2data/cache/entity/2/3", F_OK) = 0
stat64("/var/www/gallery2/g2data/cache/entity/2/3", {st_mode=S_IFDIR|0755, st_size=20480, ...}) = 0
gettimeofday({1153449599, 634894}, NULL) = 0
getpid() = 27932
open("/var/www/gallery2/g2data/cache/entity/2/3/23610.incLIybig", O_RDWR|O_CREAT|O_EXCL, 0600) = 17
fcntl64(17, F_GETFL) = 0x2 (flags O_RDWR)
fstat64(17, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb75dc000
_llseek(17, 0, [0], SEEK_CUR) = 0
close(17) = 0
munmap(0xb75dc000, 4096) = 0
getcwd("/var/www/gallery2", 4096) = 18
lstat64("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/var/www", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
lstat64("/var/www/gallery2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/var/www/gallery2/g2data", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/var/www/gallery2/g2data/cache", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/var/www/gallery2/g2data/cache/entity", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/var/www/gallery2/g2data/cache/entity/2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/var/www/gallery2/g2data/cache/entity/2/3", {st_mode=S_IFDIR|0755, st_size=20480, ...}) = 0
lstat64("/var/www/gallery2/g2data/cache/entity/2/3/23610.incLIybig", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
open("/var/www/gallery2/g2data/cache/entity/2/3/23610.incLIybig", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 17
fstat64(17, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
lseek(17, 0, SEEK_CUR) = 0
chmod("/var/www/gallery2/g2data/cache/entity/2/3/23610.incLIybig", 0644) = 0
write(17, "modules/core/classes/GalleryDeri"..., 1256) = 1256
close(17) = 0
rename("/var/www/gallery2/g2data/cache/entity/2/3/23610.incLIybig", "/var/www/gallery2/g2data/cache/entity/2/3/23610.inc") = 0
fcntl64(12, F_SETFL, O_RDWR|O_NONBLOCK) = 0
read(12, 0x89ce3e0, 8192) = -1 EAGAIN (Resource temporarily unavailable)
fcntl64(12, F_SETFL, O_RDWR) = 0
write(12, "p\0\0\0\3\n\tSELECT\n\t g2_Derivative.g"..., 116) = 116
read(12, "\1\0\0\1", 4) = 4
read(12, "\1", 1) = 1
read(12, "\35\0\0\2", 4) = 4
read(12, "\rg2_Derivative\4g_id\3\v\0\0\1\3\3\3@\0", 29) = 29
read(12, "\1\0\0\3", 4) = 4
read(12, "\376", 1) = 1
read(12, "\1\0\0\4", 4) = 4
read(12, "\376", 1) = 1
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
chdir("/etc/apache2") = 0
rt_sigaction(SIGSEGV, {SIG_DFL}, {SIG_DFL}, 8) = 0
getpid() = 27932
getpid() = 27932
kill(27932, SIGSEGV) = 0
sigreturn() = ? (mask now [])
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 27932 detached

[James Manning (jmm)]

backtrace from a sigsegv'ing apache2 instance (no symbols installed currently)

(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1213597280 (LWP 29347)]
0xb78d0c85 in shutdown_memory_manager ()
   from /usr/lib/apache2/modules/libphp4.so
(gdb) bt
#0 0xb78d0c85 in shutdown_memory_manager ()
   from /usr/lib/apache2/modules/libphp4.so
#1 0xb78b18db in php_request_shutdown ()
   from /usr/lib/apache2/modules/libphp4.so
#2 0x00000000 in ?? ()

[Martin Pitt (pitti)]

Hi James,

James Manning [2006-07-21 2:53 -0000]:
> Early this morning (2006-07-20), I got the security update for php4
> (4.3.10-10ubuntu4.5) including libapache2-mod-php4. Since then, a lot
> of my gallery2 (http://gallery2.sf.net/) pages SIGSEGV.

I'll try to reproduce this.

> Since this is a security update that's caused the SIGSEGV (admittedly,
> this is based on timing, and the actual cause of the SIGSEGV's may not
> be the security update), I'm checking the "security vulnerability" flag
> (it's at least arguably a DoS, as my gallery is now only intermittently
> available).

Agreed.

> On a related note, it appears that security.ubuntu.com doesn't keep
> around older packages, as I don't see a route that lets me downgrade to
> php4=4:4.3.10-10ubuntu4.4 so if someone could point out where I could
> get those older bits, that'd be great as both a workaround and
> confirmation that it's the fix in the *4.5 bits that's causing this on
> my system.

They should still be in your apt cache, unless you clean it regularly.
Try

  sudo dpkg -i /var/cache/apt/archives/libapache2-mod-php4_4.3.10-10ubuntu4.4_*.deb

Martin

[Martin Pitt (pitti)]

Do you see any PHP errors when it's crashing? anything in /var/log/apache2/error.log?

Can you please try to set 'log_errors = On' in /etc/php4/apache2/php.ini, restart apache, and reproduce the bug?

[Martin Pitt (pitti)]

Oh, and "display_startup_errors = On" could help as well.

[Martin Pitt (pitti)]

I installed gallery2, created a gallery and put some photos into it, and generally played around with it for quite some time without any crash. However, I am on amd64. Which platform are you on?

[James Manning (jmm)]

- downgrade worked great. had to downgrade php4-common as well, and there's an epoch (or whatever dpkg calls it) of 4:

sudo dpkg -i /var/cache/apt/archives/libapache2-mod-php4_4%3a4.3.10-10ubuntu4.4_i386.deb /var/cache/apt/archives/php4-common_4%3a4.3.10-10ubuntu4.4_i386.deb

The great news (in terms of a cause/effect) is that the sigsegv's went away when I downgraded (and then ran ab to stress gallery) and they came back when I re-upgraded to *4.5

All the error.log entries are the same (even when I modified the php.ini settings and restarted.

[Fri Jul 21 17:44:07 2006] [notice] child pid 25320 exit signal Segmentation fault (11)

Here are the errors-related entries in php.ini:

jmm@sublogic:/% grep _errors /etc/php4/apache2/php.ini|grep -v ^\;
display_errors = On
display_startup_errors = On
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
track_errors = Off

Here's a later sigsegv that may be useful, although it is a potentially corrupt stack.

(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0xb78d0699 in _efree () from /usr/lib/apache2/modules/libphp4.so
(gdb) bt
#0 0xb78d0699 in _efree () from /usr/lib/apache2/modules/libphp4.so
#1 0xb78ddd71 in _zval_dtor () from /usr/lib/apache2/modules/libphp4.so
#2 0xb78d963a in destroy_op_array () from /usr/lib/apache2/modules/libphp4.so
#3 0xb78d94c2 in destroy_zend_function () from /usr/lib/apache2/modules/libphp4.so
#4 0xb78e3d8c in zend_hash_destroy () from /usr/lib/apache2/modules/libphp4.so
#5 0xb78d9524 in destroy_zend_class () from /usr/lib/apache2/modules/libphp4.so
#6 0xb78e3c04 in zend_hash_del_key_or_index () from /usr/lib/apache2/modules/libphp4.so
#7 0xb78e425a in zend_hash_reverse_apply () from /usr/lib/apache2/modules/libphp4.so
#8 0xb78d751f in shutdown_executor () from /usr/lib/apache2/modules/libphp4.so
#9 0xb78d7022 in execute_internal () from /usr/lib/apache2/modules/libphp4.so
Previous frame inner to this frame (corrupt stack?)

How can install the symbols for this? Or does it need to be a debug version of the package (that hopefully someone else could build for me?)

[James Manning (jmm)]

Another (different) stack (still potentially corrupt, although at least they're starting with execute_internal)

(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1213597280 (LWP 31500)]
0xb78d061f in _efree () from /usr/lib/apache2/modules/libphp4.so
(gdb) bt
#0 0xb78d061f in _efree () from /usr/lib/apache2/modules/libphp4.so
#1 0xb78e3d6e in zend_hash_destroy () from /usr/lib/apache2/modules/libphp4.so
#2 0xb78dddc6 in _zval_dtor () from /usr/lib/apache2/modules/libphp4.so
#3 0xb78d764b in _zval_ptr_dtor () from /usr/lib/apache2/modules/libphp4.so
#4 0xb78e3d8c in zend_hash_destroy () from /usr/lib/apache2/modules/libphp4.so
#5 0xb78dddc6 in _zval_dtor () from /usr/lib/apache2/modules/libphp4.so
#6 0xb78d764b in _zval_ptr_dtor () from /usr/lib/apache2/modules/libphp4.so
#7 0xb78e3d8c in zend_hash_destroy () from /usr/lib/apache2/modules/libphp4.so
#8 0xb78dddc6 in _zval_dtor () from /usr/lib/apache2/modules/libphp4.so
#9 0xb78d764b in _zval_ptr_dtor () from /usr/lib/apache2/modules/libphp4.so
#10 0xb78e3d8c in zend_hash_destroy () from /usr/lib/apache2/modules/libphp4.so
#11 0xb78d9647 in destroy_op_array () from /usr/lib/apache2/modules/libphp4.so
#12 0xb78d94c2 in destroy_zend_function () from /usr/lib/apache2/modules/libphp4.so
#13 0xb78e3d8c in zend_hash_destroy () from /usr/lib/apache2/modules/libphp4.so
#14 0xb78d9524 in destroy_zend_class () from /usr/lib/apache2/modules/libphp4.so
#15 0xb78e3c04 in zend_hash_del_key_or_index () from /usr/lib/apache2/modules/libphp4.so
#16 0xb78e425a in zend_hash_reverse_apply () from /usr/lib/apache2/modules/libphp4.so
#17 0xb78d751f in shutdown_executor () from /usr/lib/apache2/modules/libphp4.so
#18 0xb78d7022 in execute_internal () from /usr/lib/apache2/modules/libphp4.so
Previous frame inner to this frame (corrupt stack?)

[James Manning (jmm)]

The 10-deep version of the stack seems pretty consistent at this point.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1213597280 (LWP 8992)]
0xb78d0699 in _efree () from /usr/lib/apache2/modules/libphp4.so
(gdb) bt
#0 0xb78d0699 in _efree () from /usr/lib/apache2/modules/libphp4.so
#1 0xb78ddd71 in _zval_dtor () from /usr/lib/apache2/modules/libphp4.so
#2 0xb78d963a in destroy_op_array () from /usr/lib/apache2/modules/libphp4.so
#3 0xb78d94c2 in destroy_zend_function () from /usr/lib/apache2/modules/libphp4.so
#4 0xb78e3d8c in zend_hash_destroy () from /usr/lib/apache2/modules/libphp4.so
#5 0xb78d9524 in destroy_zend_class () from /usr/lib/apache2/modules/libphp4.so
#6 0xb78e3c04 in zend_hash_del_key_or_index () from /usr/lib/apache2/modules/libphp4.so
#7 0xb78e425a in zend_hash_reverse_apply () from /usr/lib/apache2/modules/libphp4.so
#8 0xb78d751f in shutdown_executor () from /usr/lib/apache2/modules/libphp4.so
#9 0xb78d7022 in execute_internal () from /usr/lib/apache2/modules/libphp4.so
Previous frame inner to this frame (corrupt stack?)

[James Manning (jmm)]

I know the comments are getting long at this point, but just to confirm this is specific to the *4.5 bits, I hammered on the *4.4 bits for a few hours (and I'm going to leave them on for the next few days, at least until hearing what other steps I can do to help debug this). It usually takes around 10-15 seconds to start getting the sigsegv's from the *4.5 bits, so at this point I'm extremely confident that this is a regression introduced in the 4.5 bits resulting from advisory USN-320-1. Unfortunately, there was a large amount of changes in the 4.5 bits which makes it more difficult to track down which change is causing this (and incidentally is a good reason for security updates to not bundle so many changes :)

[James Manning (jmm)]

oh, and arch is i386, but I'm guessing you already noticed that given the dpkg line I included for downgrading to the 4.4 bits :)

Linux sublogic.com 2.6.10-5-686-smp #1 SMP Mon Oct 10 11:40:56 UTC 2005 i686 GNU/Linux

[James Manning (jmm)]

I linked this bug to CVE-2006-3018, although I only really know that it's tied to USN-320-1 which is actually tied to 11 (!) CVE's:

CVE-2006-0996, CVE-2006-1490, CVE-2006-1494, CVE-2006-1608, CVE-2006-1990, CVE-2006-1991, CVE-2006-2563, CVE-2006-2660, CVE-2006-3011, CVE-2006-3017, CVE-2006-3018

[James Manning (jmm)]

BTW, I've included all the info requested, but I don't see how to change the status away from Need Info, so I'm assuming I don't have perms to do so.

[James Manning (jmm)]

On another related note, since I don't see a -dbg package already available, I'd recommend (please!) having it. Some notes are getting them built are at http://ze-dinosaur.livejournal.com/7036.html

It'd be much nicer if I could just apt-get install the -dbg package then re-start and get useful/rich/full stacks/vars/etc from this :)

[James Manning (jmm)]

Since there's no -dbg available, I've been trying to apt-build build-source php4 but that fails with
E: Unable to find a source package for php4
even though I have the relevant deb-src in sources.list

% grep ^deb-src /etc/apt/sources.list
deb-src http://archive.ubuntu.com/ubuntu/ hoary main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu/ hoary-security main restricted

I've tried on to get an answer on #ubuntu with no response (and no pitti logged in currently :) so any hints as to how I can get this package rebuilt would be appreciated. Note that I've already said to pass -g for gcc flags while installing apt-build and its deps.

[Martin Pitt (pitti)]

Thanks for your great investigations. Candidates for the offending patches are debian/patches/CVE-2006-1990.patch and debian/patches/CVE-2006-3017.patch. Building a debugging version of php4, and removing either of those patches (and see which is the responsible one) would be great!

You can build a debugging version like this:

  sudo apt-get build-dep php4
  sudo apt-get install devscripts fakeroot
  apt-get source php4
  cd php4-*
  DEB_BUILD_OPTIONS=nostrip,noopt debuild -us -uc -b
  cd ..
  sudo dpkg -i php4-common*.deb php4-cgi*.deb php4-cli*.deb libapache2-mod-php4*.deb

In the meantime I will scrutinize the patches again and try to write a reproducer based on the stack traces above. Thank you!

[Martin Pitt (pitti)]

BTW, we are currently developing the infrastructure to automatically get debugging symbols for all packages, and automatic crash reporting in Edgy.

https://wiki.ubuntu.com/AptElfDebugSymbols
https://wiki.ubuntu.com/AutomatedProblemReports

[Martin Pitt (pitti)]

Oh, BTW, I'm 95% sure it's the patch for CVE-2006-3017. I will do some investigations now.

[Martin Pitt (pitti)]

zend_hash_del_key_or_index() is used in three different occasions:

 * unset()
 * array_pop()
 * in the cleanup of shutdown_executor()

For the first two functions I wrote a little test program that works perfectly well for me with the latest Hoary version. Can you please test this both on the command line (package php4-cli, do 'php test.php' and over apache (~/public_html/test.php, http://localhost/~yourlogin/test.php)? Does it crash for you?

[Martin Pitt (pitti)]

For the record: upstream, SuSe, RedHat, and Mandrake all use the very same patch.

Two more questions:

 * To confirm that this patch is really the culprit, can you please do 'rm debian/patches/CVE-2006-3017.patch' before building the package (debuild -us -uc -b)? Then it should work again.

 * Do you have the chance of trying this with php5 on breezy or dapper? (Hoary will only be supported for another 2 months anyway) If php5 is affected as well, I should start notifying other vendors and upstream about this regression.

Thank you!

[James Manning (jmm)]

The test works both from the command-line and apache with the same output - so 2 of the 3 consumers are fine, and shutdown_executor is the problem child here.

Array
(
    [banana] => 1
    [apple] => 2
    [kiwi] => 3
)
Array
(
    [0] => 2
    [1] => 3
    [2] => 5
    [3] => 7
    [4] => 11
    [5] => 13
    [6] => 17
)
Array
(
    [banana] => 1
    [kiwi] => 3
)
Array
(
    [0] => 2
    [1] => 3
    [2] => 5
    [3] => 7
    [4] => 11
)

Also, removing CVE-2006-3017.patch did NOT fix the problem - rebuilding after removing that patch still gives the problem.

With a debug build (thanks for instructions!) I'm getting a much more detailed stacks. The first 2 stacks below (second being "full") are before removing CVE-2006-3017.patch, the third is after (but the same

I'll attach 3 stack traces:

stack-debug-1.txt is the full 4.5 bits (including 3017) built debug, normal backtrace
stack-debug-full-2.txt is the same full 4.5 bits (including 3017) built debug, full backtrace (different stack from the first for the outermost, but the important parts are the same)
stack-with-CVE-2006-3017-removed.txt is the 4.5 bits with 3017 removed before building (wiped and re-fetched source, removed patch, built)

[James Manning (jmm)]

Removing 1990 also didn't fix it.

[James Manning (jmm)]

Just for more sanity checking, I re-downgraded to the 4.4 bits. With the 4.5 bits (all flavors tried so far), I hit sigsegv's on about 50%-70% of attempted operations, but with the 4.4 bits I hammer away on all kinds of operations and never see a problem. It's definitely a bug new to the 4.5 bits (just reaffirming this)

[Martin Pitt (pitti)]

Thank you, you are awesome!

Then the only other potential culprit is debian/patches/CVE-2006-1990.patch (in particular, the second part that patches Zend/zend_alloc.c). Can you please remove this patch and check whether it works afterwards?

[Martin Pitt (pitti)]

real_size = 8 looks fine, but cache_index = 10663344 is totally out of range (it's supposed to be real_size >> 3, i. e. 1).

I think I have an idea about the reason. If you feel totally adventurous, you can try to leave debian/patches/CVE-2006-1990.patch as-is, and change line 48 in Zend/zend_alloc.h from

  unsigned int size:31;

to

  size_t size:63;

However, this requires some more discussion with upstream, thus for now I will just revert the offending part of debian/patches/CVE-2006-1990.patch. I would just be happy to get a confirmation that it really works for you again after removing that bit.

Thank you!

[Martin Pitt (pitti)]

> Removing 1990 also didn't fix it.

Argh, I overlooked this comment. Now, *that* gives me something to think about...

CVE-2006-0996.patch: only affects phpinfo()
CVE-2006-1490.patch: only affects php_unescape_html_entities()
CVE-2006-1494.patch: only affects tempnam()
CVE-2006-1608.patch: only affects copy()
CVE-2006-2563.patch: only affects the curl module
CVE-2006-3011.patch: only affects error_log()
CVE-2006-3018.patch: only affects session module, straightfoward patch

According to the backtraces, none of above seem to be an issue here.

The remaining two patches do seem to be related to your crash:

CVE-2006-1990.patch: affects low-level memory handling in Zend (_emalloc() and _efree())
CVE-2006-3017.patch: affects zend_hash_del_key_or_index()

However, if removing both of them still lead to the crash, then I'm seriously confused. Then the only remaining possibility is that some of the 'innocent' patches created a side effect which causes a crash later.

I'm still unable to reproduce the crash, so I'm afraid I have to ask you to continue to debug this. With bisectioning (i. e. halving the set of patches), you should be able to pinpoint the offending patch in three steps. Thank you!

[James Manning (jmm)]

Off-topic, but while bisectioning, I noticed that the changelog doesn't mention a fix for CVE-2006-1991 - I'm guessing this is because it was a php5-only bug, so there was no php4 fix needed for it? Just wanted to confirm.

[James Manning (jmm)]

data points (still in progress as I write this)

previous data points
- remove 3017 == still breaks
- remove 1990 == still breaks
- remove 1990 + 3017 == still breaks

bisectioning
- remove 1990 + 3017 + 2563 + 3011 + 3018 == still breaks
  - note that this is equal to applying 0996 + 1490 + 1494 + 1608
- remove all 9 (sanity check) == works great
- apply 0996 + 1490 == works great
- apply 0996 + 1490 + 1494 == breaks

So, it appears (for now) that 1494 is the culprit (will be trying 1494 on its own as well). What's interesting is that in that scenario, the stack was different than the others (attaching), so there may indeed be some interaction among the patches.

[Martin Pitt (pitti)]

Hi James,

James Manning [2006-07-26 0:57 -0000]:
> Off-topic, but while bisectioning, I noticed that the changelog doesn't
> mention a fix for CVE-2006-1991 - I'm guessing this is because it was a
> php5-only bug, so there was no php4 fix needed for it? Just wanted to
> confirm.

Right, php4 does not yet provide substr_compare().

[James Manning (jmm)]

1494 by itself breaks.

[Martin Pitt (pitti)]

Aaah, got it. Sorry, my bad, I made a silly error when backporting the CVE-2006-1494 patch. I attach the fixed one. I'm pretty sure that it'll work now, can you please confirm?

I still wonder why it didn't crash for me, tempnam() works fine on my hoary here. Must be a very strange coincidence, or the fact that gallery2 just uses a lot more memory than my small test scripts.

Thank you a lot for tracking this down!

[Martin Pitt (pitti)]

Currently preparing an updated package, awaiting confirmation that everything works now.

[Martin Pitt (pitti)]

For the record, I confirmed that this does not affect php5, i. e. it only affects Hoary.

[James Manning (jmm)]

Cool - I'm doing a clean build with the fixed patch to confirm that everything's fine with this fix in place. old celeron 366's take awhile to rebuild php4 :)

[James Manning (jmm)]

The updated patch worked great - no more sigsegv's

Thanks, Martin!

[Martin Pitt (pitti)]

http://www.ubuntu.com/usn/usn-320-2

The fixed packages will be on the mirrors in about an hour, then the email USN announcement will go out as well.

[James Manning (jmm)]

Excellent! Thanks for the fast turnaround, Martin!