[jaunty] php5-odbc module broken

[John Wards]

Binary package hint: php5

I am unable to remove the Suhosin patch from PHP without manual intervention as documented on this blog:
http://ambitonline.com/nextrelease/archives/113-How-to-Ubuntu-PHP-Remove-Suhosin.html

I can get my head around why it is installed by default, but not the reasons why it doesn't use the provided package to do it so people can remove it if they need to.

This is a serious flaw in the package management.

[Thijs Kinkhorst (kink)]

I do not understand what you mean. What part of the process doesn't work for you, and what concrete change do you suggest to the packaging?

Why is it not enough to use Suhosin's configuration variables to disable certain or alll functionality? See
http://www.hardened-php.net/suhosin/configuration.html

[John Wards (johnwards)]

In the package management you have a package called php5-suhosin, it makes sense to use that rather than compiled in patches. That way I can remove it.

What is the reasoning behind not using the package?

Also if the "official" way of disabling Suhosin is to use conf files should a /etc/php5/conf.d/suhosin.ini file not be provided with the package with examples.

I'm a touch surprised this is the first time that using conf to disable this feature has been mentioned, as I've been hunting around for this on the forums etc for ages.

[cubical10 (cubical10)]

I 100% agree with the opinion of John Wards.
There has to be an easy and effective way to remove Suhosin from PHP with having to recompile.
There should be two methods available:
1. I should be able to remove the php5-suhosin package.
2. I should be able to comment out the second line (;extension=suhosin.so) in /etc/php5/apache2/conf.d/suhosin.ini.

Thank you;

[Jan Wagner (waja)]

Speaking as Debian Maintainer of the source package php-suhosin, I think you didn't understand, what the package "php5-suhosin" stands for.
If you did have a look into the Upstream homepage[1], you can read the following in the beginning of the page:

"Suhosin comes in two independent parts, that can be used separately or in combination. The first part is a small patch against the PHP core, that implements a few low-level protections against bufferoverflows or format string vulnerabilities and the second part is a powerful PHP extension that implements all the other protections."

So we are talking about 2 different things .... php5-suhosin isn't the equvalent to php5 with the suhosin patch, it is the package which ships the suhosin (modul-) extension for PHP.
php5 is default patched with the suhosin patch by the Debian PHP Maintainers, but this shouldn't harm you, cause it just provides logging functions, see [2].

If you what to get rid of the suhosin stuff you have serveral options. Removing php5-suhosin is the most radical option. But you can also force suhosin into simulation mode[3], which can be set global in PHP or local (for example in vhost).

Thanks for your attention, Jan.

[1] http://www.hardened-php.net/suhosin/
[2] http://www.hardened-php.net/suhosin/configuration.html
[3] http://www.hardened-php.net/suhosin/faq.html#will_my_application_break_because_suhosin_is_too_restrictive

[StephenA (steve-atty)]

The problem is that adding anything suhosin related to the php.ini file does not seem to work.

Joomla and WPMU and other PHP applications regularly seem to cause FATAL errors in the php version with Suhosin. For example:

[Thu Jul 09 12:13:23 2009] [error] [client 192.168.0.55] ALERT - canary mismatch on efree() - heap overflow detected (attacker '192.168.0.55', file '/webstuff/canalblogs/wp-admin/index-extra.php'), referer: http://canalplan.blogdns.com/wp-admin/

These errors happen at random times and on random files so its suggesting its not just down to bad coding by the developers.

Once this has happened then Apache stops serving php files and just offers them for download.

So the statement that all the suhosin patch for php5 is doing is providing "logging functions" doesn't seem to tie in with what people are seeing.

I'm now faced with having to download new sources for php5 and recompile without the suhosin patch each time you release a new version. Which seems rather counter productive.

[Jan Wagner (waja)]

> The problem is that adding anything suhosin related to the php.ini file does not seem to work.

Which php.ini file do you use and do you use libapache2-mod-php5 or cgi?
What about the following:

# echo "suhosin.simulation = on" >> /etc/php5/conf.d/suhosin.ini

Restarting the webserver and you will be fine.

With kind regards, Jan.

[Diego Malatesta (diego-malatesta)]

I have the same problem as the other guys.

And doing:

> # echo "suhosin.simulation = on" >> /etc/php5/conf.d/suhosin.ini
>
> Restarting the webserver and you will be fine.

does not solve the problem. phpinfo() shows the flag as on, but the php scripts which cause the error still stop being executed and are offered for download.

*In my opinion* you shouldn't provide a package with a patch that is known to break code (even if intentionally) without providing an alternative one, say something like php5-no-suhosin, or a way to disable the patch without recompiling the whole package.

[Jan Wagner (waja)]

Hi,

sorry ... from my side (Debian Maintainer), I cant reproduce the problem. You neither provided logs (suhosin logs to syslog) nor any example php scripts to verify your problem.

Until I don't have any reproducible facts, I can't anything for you. Anyways .. maybe the guys from Ubuntu can fix a bug which is unreproducible.

With kind regards, Jan.

[Diego Malatesta (diego-malatesta)]

The error I get is the same StephenA reported:

"ALERT - canary mismatch on efree() - heap overflow detected (attacker '<ip here>', file '<file here>')"

and I get it by calling odbc_execute() or odbc_exec() with any query. The script I used to reproduce the problem is a simple test script with just the db connection and the query.

I don't have the logs at hand right now, I'll post them tomorrow.

[Jonathan Marsden (jmarsden)]

To those who are experiencing this issue, and would like it fixed:

PLEASE provide more specific detail on exactly how to reproduce this issue.

So far, we do not even seem to have information on which release of Ubuntu is involved, much less which versions of apache2 and php5 and Joomla. Clear and informative bug reports are essential. Without a clear set of steps show in detail how to reproduce this issue, it is highly unlikely any further progress can be made. A complete bug report should include:

 * The specific version of Ubuntu that the reporter is running (example: Ubuntu Server 9.04 Jaunty on amd64)
 * The specific version of the package(s) the reporter is using (use dpkg-query -W PACKAGENAME for this)
 * The actions taken to produce the problem (including any relevant changes to configuration files, full details of any software installed "by hand" or from non-Ubuntu package repositories) and what the web browser user does to trigger the bug, if we are dealing with a web application)
 * Whether or not it is possible for the reporter to reproduce the bug (by following these actions)
 * The expected result of these actions
 * The actual result of these actions (including all relevant log file entries)

If you are experiencing this reported issue, please provide as many of the above items of information as you possibly can.

Thanks!

[Jonathan Marsden (jmarsden)]

An attempted set of steps to reproduce this issue follows. I failed to reproduce it!

Those who can reproduce it, please document, in a way similar to this,
exactly how you (and so others!) can also reproduce this issue.

Just in case the web display on LP messes up my PHP script, I am attaching the
odbctest.php test script I used, too.

Steps to Try to Reproduce LaunchPad Bug #315507

1) Create fresh virtual machine, install Ubuntu Server 9.04 Jaunty.
   Choose all the defaults, set your own time zone and your own user
   name and password. Pick no tasks, do a base system install only.

   Install was done from from ISO image ubuntu-9.04-server-i386.iso
   (md5sum is 20480057590ff8b80ad9094f40698030 and the ISO was
   downloaded from
   http://releases.ubuntu.com/jaunty/ubuntu-9.04-server-i386.iso ).

   Note: virtualbox-ose 2.1.4-dfsg-1ubuntu3 was used for this VM, but
   any other virtual machine setup (KVM, vmware server, etc) should
   also work fine, as would installing to a spare physical machine.

2) Update System and reboot

   sudo apt-get update && sudo apt-get dist-upgrade -y ## Update system
   sudo shutdown -r now ## Reboot system to pick up new kernel etc.

3) Install LAMP Server packages

   sudo tasksel install lamp-server ## Install LAMP server

   Note: Provide a password for MySQL server when the installer
   requests one. Remeber this password (I used "secret").

4) We need ODBC to reproduce issue, so set up for ODBC to MySQL.

   sudo apt-get install php5-odbc libmyodbc unixodbc -y
   sudo cp -p /usr/share/libmyodbc/odbcinst.ini /etc/
   sudo cp -p /usr/share/doc/libmyodbc/examples/odbc.ini /etc/
   sudo service apache2 restart

   Note: The only two config files changed from their defaults are
   /etc/odbc.ini and /etc/odbcinst.ini which are zero length by default.
   The cp commands above copy the supplied example files, no
   changes to these examples are needed for this test setup.

5) Create a PHP test web page under /var/www/ and verify it runs

   echo -e "<?php\nphpinfo();\n?>\n" |sudo tee /var/www/phpinfo.php
   wget -O info.html http://localhost/phpinfo.php
   w3m info.html ## Examine carefully, esp. Suhosin info

   Note: info.html should show the full phpinfo output, and it should
   include the information that "This server is protected with the
   Suhosin patch". Keep the info.html file in case it is needed later
   on during testing.

6) Create a test database and a testdb table in it, and 2 records

   PW=secret ## Use the password you set for mysql root earlier
   echo "create database test;" |mysql -uroot -p"$PW"
   echo "create table testdb ( id int );" |mysql -uroot -p"$PW" test
   echo "insert into testdb values (42);" |mysql -uroot -p"$PW" test
   echo "insert into testdb values (2001);" |mysql -uroot -p"$PW" test

7) Create PHP page that uses odbc_connect() and odbc_exec()
   cat >odbctest.php <<"EOF"
<?php
$pw="secret";
$connection = odbc_connect("myodbc", "root", $pw);
$sql = 'select id from testdb';
$result = odbc_exec($connection, $sql);

while (odbc_fetch_row($result)) {
  $id = odbc_result($result, 'id');
  echo "$id<br>\n";
}

odbc_free_result($result);
odbc_close($connection);
?>
EOF
...

[Jonathan Marsden (jmarsden)]

As a further test, I have also installed php5-suhosin,
rebooted the virtual machine, and then retested it
with 100,000 repetitions using ab.

It all still works fine. /var/log/apache2/error.log contains
no errors relating to "canary mismatch", and even doing

  sudo grep -ri "canary mismatch" /var/log/

shows no output.

Jonathan

[Diego Malatesta (diego-malatesta)]

Jonathan, thanks for taking the time to post an exhaustive reply.

I'm creating a new VM right now to do a complete test as you suggested, but as that's not going to reproduce our real world situation, I'm going to post the details of the actual machine where the thing is happening.

Later on I'll post the results from the complete test on the new vm.

The server is a vmware esxi 4 VM (like the new vm I'm creating).

Ubuntu release:
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 8.04.2
Release: 8.04
Codename: hardy

Packages version (note: I don't have php5-cli installed):
# dpkg-query -W apache2 libapache2-mod-php5 libmyodbc php5-common php5-odbc
apache2 2.2.8-1ubuntu0.10
libapache2-mod-php5 5.2.4-2ubuntu5.6
libmyodbc 3.51.15r409-2
php5-common 5.2.4-2ubuntu5.6
php5-odbc 5.2.4-2ubuntu5.6

This is the last request from apache2 log (/var/log/apache2/error.log) and syslog:
[Mon Jul 20 08:42:55 2009] [error] [client ip here] ALERT-SIMULATION - canary mismatch on efree() - heap overflow detected (attacker 'ip here', file '/var/www/services/reports/odbc.php')

Note that even if it shows "ALERT-SIMULATION" I still get the php script offered for download. Of course the same thing applies without simulation mode on (except it shows "ALERT" without the "-SIMULATION").
Note also that I've tried to run the script both without and with the suhosin extension (php5-suhosin).

The test script is basically:

$connection = odbc_connect($dsn, $user, $pass);
$result = odbc_exec("select * from table");

Then there is the while to loop on the resultset, but the script hangs on the odbc_exec line (tested by deleting one line at the time until I got no error).
The mysql server is on another (phisical) machine. I've tested the connection and the same query with isql and everything works fine.

Oh and everything is on https (I can test with http if needed).

If I missed something or you need more info, just ask.
Thanks.

[Diego Malatesta (diego-malatesta)]

erhm, of course in my test script in the odbc_exec function I've specified the connection parameter, I just missed it here in the comment.

[Diego Malatesta (diego-malatesta)]

Darn there should be an edit function for comments here...
I forgot to mention an essential thing, the ubuntu release is the AMD64 one.

[Diego Malatesta (diego-malatesta)]

Ok I was able to reproduce the problem on a new VM

Steps:

1) Create fresh vm: done, installed Ubuntu 8.04.2 amd64 as denoted by

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 8.04.3 LTS (it shows .3 because I issued the command after the update I think)
Release: 8.04
Codename: hardy

2) Update system and reboot: done

3) Install LAMP Server packages: done. I didn't install mysql, only apache2 and php5 (I have the db on another machine)

4) We need ODBC: done. Installed php5-odbc libmyodbc unixodbc, copied the sample configurations and adapted odbc.ini to connect to my db server. Plus I tested the connection with isql and worked.

5) Create a PHP test page: done. I've attached the info.html file (with ip and domain hidden for privacy reasons)

6) I already have a database ready (MySQL 5.0.24)

7) Create PHP page to test odbc: done. It's the exact copy of your example script, with the connection data and the table changed of course

8) Try the script.. and here the browser serves me the file as a download. In /var/log/apache2/error.log there is the canary error. Here's the complete log:

[Mon Jul 20 11:39:37 2009] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.6 with Suhosin-Patch configured -- resuming normal operations
[Mon Jul 20 11:47:47 2009] [error] [client <client ip>] ALERT - canary mismatch on efree() - heap overflow detected (attacker '<client ip>', file '/var/www/odbctest.php', line 11), referer: http://<server ip>/
[Mon Jul 20 11:47:47 2009] [error] [client <client ip>] ALERT - canary mismatch on efree() - heap overflow detected (attacker '<client ip>', file '/var/www/odbctest.php', line 11), referer: http://<server ip>/

9) and 10) no sense doing these. The issue in not intermittent, it happens every time.

11) Document exact versions of packages:

# dpkg-query -W apache2 libapache2-mod-php5 libmyodbc php5-common php5-odbc
apache2 2.2.8-1ubuntu0.10
libapache2-mod-php5 5.2.4-2ubuntu5.6
libmyodbc 3.51.15r409-2
php5-common 5.2.4-2ubuntu5.6
php5-odbc 5.2.4-2ubuntu5.6

[Ondřej Surý (ondrej)]

Just for the record: It's not suhosin patch which needs to be removed, it's the php or php extension which needs fixing, since corrupted canary means that there is stack/buffer overflow somewhere. See: http://en.wikipedia.org/wiki/Stackguard#Canaries

[Diego Malatesta (diego-malatesta)]

To be honest it wouldn't be a problem for me if the simulation mode actually worked. I don't really mind if the odbc functions are badly coded (or whatever). What I do mind is suhosin breaking my scripts without a way to prevent it which doesn't include recompiling php without the patch.

But of course if I can help in debugging the real problem and fix the root of the problem (which of course would be better), hey I'm here :)

[Diego Malatesta (diego-malatesta)]

I've reproduced the problem on a 9.04 amd64 fully updated, same configuration as above.

[Ondřej Surý (ondrej)]

Diego, could you try it to reproduce under i386 in vm?

Maybe it has something to do with alignment on 64-bit.

Ondrej

P.S.: You should care if odbc (or php) is badly coded, because buffer overflows can trigger remote attacks on your server leading do DoS or even intrusion into your system.

[Diego Malatesta (diego-malatesta)]

Ondrej, sure as soon as I'm done with this vm I'm building I'll try with i386.

It's not that I don't care, but as we're migrating tons of stuff around and this migration has to be finished soon, I really need the odbc thing working asap, so that's why I don't mind the buffer overflow for now, if I can get the thing to work with a workaround.

But as I said I'l gladly try my best to solve the root problem because I understand that's not something to be underestimated.

Thanks for your help.

[Diego Malatesta (diego-malatesta)]

Ok, I've tried to reproduce the problem on Ubuntu 8.04.3 i386 and it does NOT show up, so it seems to be related to the amd64 architecture as Ondrej suggested.

let me know if I have to do some more tests.

[Jan Wagner (waja)]

Ondřej, to clearly pointing out ... you are talking about the php5 package and/or any php5 extention, which causes the Canaries. Suhosin makes the problem just visible. Please correct me, if I'm wrong.

With kind regards, Jan.

[Ondřej Surý (ondrej)]

Jan, you're absolutely right. Right now we know only about php5-odbc extension (and it can even be buried somewhere in odbc libraries), but there seems to be more (according to blogpost in first report there is something which is triggered by Joomla).

Ondrej

[Diego Malatesta (diego-malatesta)]

http://chrisblunt.com/blog/2009/05/01/php-fixing-mismatched-canaries-how-to-remove-suhosin-from-debianubuntu-packages/

here the guy talks about mssql_query causing the canary mismatch.

[shaberer (shaberer)]

Hi,
I am suffering the same odbc issue as mentioned in #16 (also on amd64), but can live with it, as this is my private machine.

My real problem (as mentioned in #18) is that while I have set "suhosin.simulation = on", suhosin not only logs the issue but also seems to abort it, so the browser offers an empty file for download.

Is the simulation-mode function broken or am I doing something wrong?
ThanX

[Ondřej Surý (ondrej)]

Hi,

just a note. New suhosin patch for 5.3 will be more customizable:

The following environment variables are supported by now:

SUHOSIN_MM_USE_CANARY_PROTECTION
default: 1
Set to 0 to disable canary protection. A copy of the MM will be used that does not have canaries. This is nearly the same as the MM of vanilla PHP.

SUHOSIN_MM_DESTROY_FREE_MEMORY
default: 0
Set to 1 to enable free memory destruction. Every piece of free memory will be overwritten. This allows debugging e.g. use after free memory corruption bugs easier without using a debug PHP.

SUHOSIN_MM_IGNORE_CANARY_VIOLATION
default: 0
Set to 1 stops Suhosin from aborting the process when it detects canary violations. The violations will be logged and the canary restored. It is strongly recommended to NOT use this feature. But it is more secure to use this feature instead of disabling Suhosin completely which happend in the past when people saw canary violation error messages

SUHOSIN_HT_IGNORE_INVALID_DESTRUCTOR
default: 0
Set to 1 stops Suhosin from aborting the process when it detects an invalid Hashtable destructor. It is strongly recommended to NOT use this feature.

SUHOSIN_LL_IGNORE_INVALID_DESTRUCTOR
default: 0
Set to 1 stops Suhosin from aborting the process when it detects an invalid LinkedList destructor. It is strongly recommended to NOT use this feature.

See http://www.suspekt.org/2009/08/13/suhosin-patch-098-for-php-530-beta-please-test/ for more information.

[Ondřej Surý (ondrej)]

Jonathan,

I am able to reproduce the bug.

odbc extension is obviously broken as hell :(

If you do only odbc_connect in the script it freezes. I'll look into possibility of backporting odbc/pdo_odbc from 5.2.10 upstream.

Ondrej

[Ondřej Surý (ondrej)]

Hi,

finally it turned out, that suhosin is not at fault here. But the odbc module is broken (almost beyond repair).

I have attached patch to apply in php5 source which fixes the canary mismatch, but odbc module doesn't play well with mysql and mysqli modules, so you'll have to disable them to avoid locks in php destructors (I guess that this is because libmyodbc).

I could try to hunt it down, but this is more a candidate for upstream bug than me hunting bad php programming.

Ondrej.

[Chuck Short (zulcss)]

Thanks for the bug report. I was wondering if you could try out the karmic version.

Thanks
chuck

[Diego Malatesta (diego-malatesta)]

Hi Chuck,

I've set up a karmic alpha 5 amd64 test server on virtualbox.

root@karmic:/var/www# uname -a
Linux karmic 2.6.31-9-server #29-Ubuntu SMP Sun Aug 30 18:37:42 UTC 2009 x86_64 GNU/Linux

root@karmic:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu karmic (development branch)
Release: 9.10
Codename: karmic

everything updated to the latest version available:

root@karmic:/var/www# dpkg-query -W apache2 libapache2-mod-php5 libmyodbc php5-common php5-odbc
apache2 2.2.12-1ubuntu2
libapache2-mod-php5 5.2.10.dfsg.1-2ubuntu2
libmyodbc 3.51.19r646-1
php5-common 5.2.10.dfsg.1-2ubuntu2
php5-odbc 5.2.10.dfsg.1-2ubuntu2

created a test page to connect to a mysql server I have on another VM and...

everything works just fine! NO canary mismatches, yay! :)

I'll be able to do some more tests when I'll be at work next week. Let me know If you need something else.

[Chuck Short (zulcss)]

Good its nice to know that this bug has been resolved for jaunty.

Thanks
chuck

[Chuck Short (zulcss)]

Hi,

Im closing this bug because it has been fixed in the development release for Ubuntu. This bug will be kind of hard to track down to fix in Jaunty.

Regards
chuck

[Diego Malatesta (diego-malatesta)]

uhm, yeah.. and what about 8.04 LTS? Will the fix be ported from karmic?

[StephenA (steve-atty)]

I'd also like to know if its going to be back ported to 8.04 LTS, or are you going to force us to move off LTS onto the short term support 9 series?

[Jan Wagner (waja)]

bug in php5

[rawdmon (raw-dmon)]

I can't quite explain how incredibly annoying this suhosin patch is. Even if I use the config option which supposedly disables it, it continues to mess with code responsible for saving password crypts to a database (it's forcing it to use sha-512 instead of salted md5, incredibly annoying and a huge pain to try to fix. It would really be nice to have the option of installing a package that doesn't have this patchset if I choose to do so, so that the code will work as it did in the past.