canary mismatch on efree()

Bug #503396 reported by Hal
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
php5 (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

Binary package hint: php5

After spending some time researching this, I realize the root cause may not be in php itself (but might), but not knowing the root cause, I am reporting it here.

Environment: Ubuntu 8.04, PHP 5.2.4-2ubuntu5.9 with Suhosin-Patch 0.9.6.2 , suhosin, xcache, xdebug, mysql, gd, curl, ffmpeg, cli. The server runs several vhosted sites. The problem occurs consistently on one line of one site only. The site in question runs Drupal, and the error is triggered by the Drupal webforms module (at the same line every time) upon a form submission.

Symptoms: After several days (3 to 14 days), the following error is reported:

Jan 4 22:07:14 Garth suhosin[25113]: ALERT - canary mismatch on efree() - heap overflow detected (attacker '67.141.28.120', file '/raid/clients/midway.edu/htdocs/modules/webform/webform.module', line 2201)
Jan 4 22:07:15 Garth suhosin[25116]: ALERT - canary mismatch on efree() - heap overflow detected (attacker '67.141.28.120', file '/raid/clients/midway.edu/htdocs/modules/webform/webform.module', line 2201)
Jan 4 22:11:47 Garth suhosin[25119]: ALERT - canary mismatch on efree() - heap overflow detected (attacker '68.42.206.239', file '/raid/clients/midway.edu/htdocs/modules/webform/webform.module', line 2201)
Jan 4 22:11:47 Garth suhosin[25141]: ALERT - canary mismatch on efree() - heap overflow detected (attacker '68.42.206.239', file '/raid/clients/midway.edu/htdocs/modules/webform/webform.module', line 2201)
Jan 4 22:21:57 Garth suhosin[25154]: ALERT - canary mismatch on efree() - heap overflow detected (attacker '67.141.28.49', file '/raid/clients/midway.edu/htdocs/modules/webform/webform.module', line 2201)
Jan 4 22:21:58 Garth suhosin[25139]: ALERT - canary mismatch on efree() - heap overflow detected (attacker '67.141.28.49', file '/raid/clients/midway.edu/htdocs/modules/webform/webform.module', line 2201)

etc, etc.

At always the exact same line number. At this point, anybody submitting any form on the site in question will trigger the error. Forms are an important aspect of the site, and this is breaking that functionality as none of the forms work as expected. Restarting Apache temporarily solves/works around the problem.

Line 2201, that triggers the error: return $strict ? filter_xss($string) : $string;

The filter_xss() Drupal function that is referenced:

function filter_xss($string, $allowed_tags = array('a', 'em', 'strong', 'cite', 'code', 'ul', 'ol', 'li', 'dl', 'dt', 'dd')) {
  // Only operate on valid UTF-8 strings. This is necessary to prevent cross
  // site scripting issues on Internet Explorer 6.
  if (!drupal_validate_utf8($string)) {
    return '';
  }
  // Store the input format
  _filter_xss_split($allowed_tags, TRUE);
  // Remove NUL characters (ignored by some browsers)
  $string = str_replace(chr(0), '', $string);
  // Remove Netscape 4 JS entities
  $string = preg_replace('%&\s*\{[^}]*(\}\s*;?|$)%', '', $string);

  // Defuse all HTML entities
  $string = str_replace('&', '&', $string);
  // Change back only well-formed entities in our whitelist
  // Named entities
  $string = preg_replace('/&([A-Za-z][A-Za-z0-9]*;)/', '&\1', $string);
  // Decimal numeric entities
  $string = preg_replace('/&#([0-9]+;)/', '&#\1', $string);
  // Hexadecimal numeric entities
  $string = preg_replace('/&#[Xx]0*((?:[0-9A-Fa-f]{2})+;)/', '&#x\1', $string);

  return preg_replace_callback('%
    (
    <(?=[^a-zA-Z!/]) # a lone <
    | # or
    <[^>]*(>|$) # a string that starts with a <, up until the > or the end of the string
    | # or
    > # just a >
    )%x', '_filter_xss_split', $string);
}

This same site was moved from another Ubuntu 8.04 server with a very similar environment, and in almost 1 year, this error never occurred there.

# apt-cache policy php5
php5:
  Installed: 5.2.4-2ubuntu5.9
  Candidate: 5.2.4-2ubuntu5.9
  Version table:
 *** 5.2.4-2ubuntu5.9 0
        500 http://us.archive.ubuntu.com hardy-updates/main Packages
        500 http://security.ubuntu.com hardy-security/main Packages
        100 /var/lib/dpkg/status
     5.2.4-2ubuntu5 0
        500 http://us.archive.ubuntu.com hardy/main Packages

# php -v
PHP 5.2.4-2ubuntu5.9 with Suhosin-Patch 0.9.6.2 (cli) (built: Nov 26 2009 14:00:44)
Copyright (c) 1997-2007 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies
    with Xdebug v2.0.2, Copyright (c) 2002-2007, by Derick Rethans
    with Suhosin v0.9.22, Copyright (c) 2007, by SektionEins GmbH

# lsb_release -rd
Description: Ubuntu 8.04.3 LTS
Release: 8.04

Revision history for this message
Ondřej Surý (ondrej) wrote :

Is there a difference in the architecture? There are some bugs in PHP triggered by 64-bit (amd64) arch.

Revision history for this message
Hal (hal-foobox) wrote :

Thanks. Not 64bit. Here are the kernels:

Problematic system:
# uname -a
Linux Garth 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 UTC 2008 i686 GNU/Linux

Previous system:
# uname -a
Linux ratt 2.6.24-24-server #1 SMP Tue Aug 18 17:46:20 UTC 2009 i686 GNU/Linux

Revision history for this message
Thierry Carrez (ttx) wrote :
Changed in php5 (Ubuntu):
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Hal (hal-foobox) wrote :

Thierry, yes, thanks, I had actually come across both of these.

A couple of points: On the first link ... this same site ran fine on another 8.04 system with a very similar configuration for 8-9 months without this error. Secondly, if I have hit a php bug that is corrupting memory (ie suhosin is just the messenger of bad news), why does it take days worth of submitting these forms, to trigger the bug? (My wild, wild theory is that it is xcache related, but who knows ... BTW the problem system is 3G RAM vs the non-problem former system 1G).

On the second link, it looks like all these are using mysqli extension, or mssql. There is no shortage of similar problems with mssql. The site in question is using plain mysql (not mysqli). Not sure this is necessarily significant or not.

Lastly, I have probably 4 or 5 systems using 8.04 with suhosin, and this is the only time I ever have seen this error.

Thanks.

Revision history for this message
Ondřej Surý (ondrej) wrote :

Hal, is your system fully upgraded?

Where does the difference between kernel versions come from?

I remember there was some security upgrade in libpcre last year...

Ondrej

Revision history for this message
Hal (hal-foobox) wrote :

The server is updated, except for some packages "held back", which includes a newer kernel. libpcre is up to date. I haven't done the kernel upgrade (yet), because of the reboot, and the server is remote from where I am (and /me worry over such things :). I will do that soon, just in case it is relevant.

I also yesterday am trying 2 suhosin ini setttings, one references apc and the other related to encrypted sessions (based on googling). Shots in the dark though.

Revision history for this message
Ondřej Surý (ondrej) wrote :

As far as I remember, the session encryption bug was related to new version of php (in karmic?).

Another shot in the dark - is there different setting in MaxRequestsPerChild setting in apache2? Could you set it to some arbitrary number, so apache2 child gets reloaded once in a while? (I know it's not a solution, but it could help...)

Revision history for this message
Hal (hal-foobox) wrote :

Ondřej, both systems were already set to 1000 MaxRequestsPerChild.

Revision history for this message
Ondřej Surý (ondrej) wrote :

Are you able to reproduce this bug under controlled environment? On testing machine, inside KVM, etc.?

I know that I may be asking too much, but could you try running PHP with suhosin under valgrind?

Revision history for this message
Hal (hal-foobox) wrote :

I have not been able to reproduce this problem outside of the current production environment. And I can't realistically try to run php under valgrind since these are paying clients, and I can't risk their sites stability, etc.

The other problem with reproducing this is that it takes some time to manifest. Its related to the form submissions on the site. On any given day there probably are 25-50 form submissions (some are very large multi-page forms too), and it takes maybe a week or so of this, for the error to start showing up. I don't really know a way to duplicate all that in a controlled environment. My guess is that if I restart apache every day, I'd never see the error at all.

I do have a copy of the site in a staging environment. I have never seen the error there, but there is really no traffic to speak of.

Revision history for this message
Hal (hal-foobox) wrote :

We had another episode of this today. The previous configuration changes didn't help, obviously. The client mildly freaked. I am removing suhosin for the time being. I hate to do it ... but ...

If I have time, I will try the updated kernel today.

Revision history for this message
Hal (hal-foobox) wrote :

I've installed the latest kernel. Then had a change of heart, and put sushosin back. Maybe the reboot brings good ju-ju.

Revision history for this message
Hal (hal-foobox) wrote :

Another episode of this happened last night, about 4.5 hours after rebooting to install the new kernel. This is a much shorter timeframe between episodes. I don't know if this is a fluke or something else.

Same pattern though: the exact same line in the exact same file on the same site. Again, I've removed sushosin. Hopefully this helps.

Revision history for this message
Ondřej Surý (ondrej) wrote :

Hal,

are we speaking about php5-suhosin package or just plain libapache2-mod-php5 and your sentence "I've removed suhosin" means that you have recompiled php5 from source and quilt-deleted suhosin patch from series? Those are two different things.

Ondrej

Revision history for this message
Hal (hal-foobox) wrote :

Sorry, I removed the php5-suhosin package for now. I'd really like to get it back, but I need to have things stabilized for a while for the client's sake.

Revision history for this message
Ondřej Surý (ondrej) wrote :

It's fine with me :), and I understand your concern...I've worked for webhosting company some time ago.

I was just asking if you have just removed php5-suhosin (extension) or if you recompiled php5 to disable suhosin patch.

http://www.hardened-php.net/suhosin/download.html

If you have just removed php5-suhosin package - then you are still protected by suhosin patch (and it is suhosin patch detecting canaries). So either you will be hit by same issue again or there is some bug in current suhosin extension which triggers those canaries.

Revision history for this message
Hal (hal-foobox) wrote :

Yes, I thought I had seen that in my googling. And I really hope, this helps isolate the problem (for everybody's sake).

If there are other ideas on things to try, I am open to those. Just keeping in mind this is in a production environment so I have some limitations.

Thanks.

Revision history for this message
Hal (hal-foobox) wrote :

Didn't help. The error is back again today (without extension).

Revision history for this message
Hal (hal-foobox) wrote :

Hello. I am back. We have moved all the vhosted clients from this system to another Ubuntu 8.04 and this problem has not resurfaced since then. These systems are very, very similar in their configurations. Both are primarily web servers. Its been now 28 days, and that is probably a good sign that the problem was left behind. Until now, it had been persistent, anywhere from 1 day to maybe 2.5 weeks. Always same error, at the same line, in the same file. I am glad to be rid of it, but don't have a good explanation to explain it.

Thanks for the help.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.