PHP5 under Apache2 on 64 bit system is not completely 64 bit

Bug #564920 reported by R. Jones
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
php5 (Ubuntu)
Low
Thierry Carrez
Lucid
Medium
Unassigned

Bug Description

Binary package hint: php5

Something in php5 running under apache2 under Ubuntu 10.04 beta 2 (with updates) is not fully 64 bit aware on a 64 bit system. PHP scripts run under Apache correctly under Ubuntu 8.04 LTS (with latest updates). Under Ubuntu 10.04 beta 2, some place a value is being treated as 32 bits on a 64 bit system and is wrapping instead of using full 64 bits. Related to sending large data (> 4G) in response to an HTTP post request.

What is expected: Full data (>4G) downloaded via dynamic PHP based URL.

What happens: Data aborts based on wrap. Calculated number of bytes to be served are correct (>4G) as listed in the HTTP length header. Actual bytes served are (size) && 0xFFFFFFFF instead of (size) && 0xFFFFFFFFFFFFFFFF on 64 bit system.

Most of data is being transfered via fpassthru($openFileHandle) call, providing >4G bytes of data.

This works under Ubuntu 8.04 LTS on 64 bit system.

root@opusdev2:~# lsb_release -rd
Description: Ubuntu lucid (development branch)
Release: 10.04
root@opusdev2:~# apt-cache policy php5
php5:
  Installed: 5.3.2-1ubuntu4
  Candidate: 5.3.2-1ubuntu4
  Version table:
 *** 5.3.2-1ubuntu4 0
        500 http://us.archive.ubuntu.com/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status
root@opusdev2:~# apt-cache policy php5
php5:
  Installed: 5.3.2-1ubuntu4
  Candidate: 5.3.2-1ubuntu4
  Version table:
 *** 5.3.2-1ubuntu4 0
        500 http://us.archive.ubuntu.com/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status

Related branches

Revision history for this message
R. Jones (robejo05) wrote :
Revision history for this message
R. Jones (robejo05) wrote :

Was php5 compiled with -D_LARGEFILE_SOURCE and -D_FILE_OFFSET_BITS=64 on 64 bit systems?

Revision history for this message
Chuck Short (zulcss) wrote :

Please provide a sample php script so I can try to replicate this.

Regards
chuck

Changed in php5 (Ubuntu):
importance: Undecided → Low
status: New → Incomplete
Revision history for this message
R. Jones (robejo05) wrote :

Test file opened by the attached script, test.php, was 4,700,110,848 bytes in length. Under Ubuntu 10.04 beta 2, download socket was closed after transfering 405,143,552 bytes. (a 0xFFFFFFFF mask of the full file size).

Test needs to run on 64 bit system in order to properly work on files larger than 0xFFFFFFFF bytes in size.

Script works when run under Ubuntu 8.04 LTS (with updates) and fails when run under Ubuntu 10.04 beta 2 (with updates), both [amd64] bit Ubuntu based systems (running Intel CPUs).

An example (public) large file is: http://cdimage.debian.org/debian-cd/5.0.4/amd64/iso-dvd/debian-504-amd64-DVD-1.iso .

Revision history for this message
R. Jones (robejo05) wrote :

Bug also exists in current Ubuntu 9.10 release.

Tested using the test.php script previously attached, and using the suggested debian dvd .iso image as the input file (listed as test.txt in test.php script). .iso image is 4,697,126,912 bytes. Download socket is closed after 402,159,616 bytes, which again is the file size & 0xFFFFFFFF.

Attached is the apport report for php under Ubuntu 9.10, currrent release, latest updates.

Revision history for this message
R. Jones (robejo05) wrote :

Attached requested script to reproduce bug.

Changed in php5 (Ubuntu):
status: Incomplete → New
Revision history for this message
R. Jones (robejo05) wrote :

Bug does not exist in Debian 5.04 release (with updates).

Revision history for this message
Peter Matulis (petermatulis) wrote :

I have confirmed this.

A 64-bit Lucid Apache server hosting a file of size

4697126912 bytes

and a client (happens to be Lucid also) downloading the file via the above test.php script only gets

402159616 bytes

Changed in php5 (Ubuntu):
status: New → Confirmed
Revision history for this message
Peter Matulis (petermatulis) wrote :

I sent this bug upstream.

http://bugs.php.net/bug.php?id=52102

Revision history for this message
Peter Matulis (petermatulis) wrote :

This turned out to be a duplicate of

http://bugs.php.net/bug.php?id=44522

Submitted: 2008-03-24 18:21 UTC

Revision history for this message
R. Jones (robejo05) wrote :

Actually, http://bugs.php.net/bug.php?id=44522 is concerned with uploading data to the server, which according to 44522 breaks at the 2G byte mark (a signed 32 bit int issue). This bug (564920, php # 52102) is an issue with downloading data from the server, which breaks at the 4G byte mark. Both are 32 bit vs 64 bit integer issues. So, this is not exactly a duplicate bug.

Peter:

Can you look at the patches to http://bugs.php.net/bug.php?id=44522 and see if we can apply them and test the results to see if that fixes the download issue?

Revision history for this message
Peter Matulis (petermatulis) wrote :

Yes, we are doing just that.

Revision history for this message
Chuck Short (zulcss) wrote :

Can you try the version in my ppa when its built?

thanks
chuck

Revision history for this message
R. Jones (robejo05) wrote :

Ok. I'm back to trying to test this fix. I think I have lucid-proposed added to my test configuration. I'm not seeing the build. A pointer to a web page documenting what I need to do next to access your "ppa" area and use it would be useful.

Thanks.

Revision history for this message
Peter Matulis (petermatulis) wrote :

@R. Jones

Go to

https://launchpad.net/~zulcss/+archive/ppa

and follow instructions under "Adding this PPA to your system".

Install

php5 5.3.2-1ubuntu4.3

I tested this myself and the results were unsatisfactory (Apache error after about 10% download of the test ISO file). Can you confirm the same?

Thierry Carrez (ttx)
Changed in php5 (Ubuntu):
assignee: nobody → Chuck Short (zulcss)
Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

FYI, this is probably what caused the regression, but reverting it wouldn't be a proper fix:

http://svn.php.net/viewvc?view=revision&revision=280678

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Here a patch that should fix this. It will stop php from mmaping large files. It may not be the approach upstream would like.

tags: added: patch
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Marc, 4*1024*1024 (4MB) seems rather arbitrary. An ini setting would be a natural choice there. Upstream may or may not like it, but I think limiting mmap to a certain size actually makes sense.

I haven't tried the patch yet, does it solve the issue?

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Well, it's not completely arbitrary. It's the size that was used previous to php 5.2.10. See _php_stream_mmap_range() in main/streams/mmap.c.

In my limited testing, it does solve the issue. Unfortunately, it also has the side effect of re-introducing php bug #48309 for files over 4MB.

Maybe making it size_t - 1 would be better?

Revision history for this message
Chuck Short (zulcss) wrote :

So this is caused by a regression by the fix for php bug #48309. Marc patch does work but the side effect of the fix is that it re-introduces the same bug, so we need to find a better fix for bug #48309.

The plan going forward is the following:

- Make a PHP version with the patch available in a PPA.
- Ask for testers
- Make sure there isnt any more regressions
- Get it fixed in maverick
- Get it fixed for lucid through an SRU.
- Let upstream know about the regression and submit the patch upstream.

If you have any questions please let me know.

chuck

Revision history for this message
Chuck Short (zulcss) wrote :

Can you try the version in ppa:zulcss/server-lucid-bug-fixes ?

thanks
chuck

Revision history for this message
R. Jones (robejo05) wrote :

Is ppa:zulcss/server-lucid-bug-fixes different from what is pointed at by https://launchpad.net/~zulcss/+archive/ppa ? The later is still showing a 2010/06/28 build date which is broken for trying to run PHP scripts under apache2. It is missing "libapache2-mod-php5{a} and php5-intl{a} from the build. aptitude then removes the existing libapache2-mod-php5 module and leaves the system broken for running PHP scripts under apache.

I'm not familiar enough with the ppa configuration to see a later php5 update.

And the documented script for adding ppa's to Ubuntu 10.04 (add-apt-repository ) is broken for access through firewalls.

Revision history for this message
Chuck Short (zulcss) wrote :

Hi,

You can install the PPA manually as documented at https://help.launchpad.net/Packaging/PPA/InstallingSoftware, see the section "On older (pre 9.10) Ubuntu systems". After following the steps you should be able to install it without a problem.

chuck

Revision history for this message
Peter Matulis (petermatulis) wrote :

I tested this and confirmed that the entire test ISO was properly downloaded.

Revision history for this message
R. Jones (robejo05) wrote :

I've now had a chance to install the PHP5 build from ppa:zulcss/server-lucid-bug-fixes. With that build, I was able to download a file > 2^32 bytes (over the 4.2Gb boundary) successfully. I still need to test the scripts I have running under Apache2 / PHP5 / Mysql 5.0 from Ubuntu 8.04 LTS to see if they continue to run under a Ubuntu 10.04 LTS configuration.

Installing the version of PHP5 from ppa:zulcss/server-lucid-bug-fixes using aptitude resulted in uninstalling a number of items (such as autoconf, automake, autotools-dev, libssl-dev, libtools, m4, shtool, zlib_g-dev), so it looks like there are some additional regression builds that will be needed to make the package system happy. Prior to the PHP5 install, the system was running Ubuntu desktop 10.04 LTS with the latest public standard production updates. The system had the previous PHP5 build installed and then backed out to the standard configuration prior to this update.

Changed in php5 (Ubuntu):
status: Confirmed → In Progress
assignee: Chuck Short (zulcss) → Clint Byrum (clint-fewbar)
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

attaching debdiff of same thing as linked merge proposal

Changed in php5 (Ubuntu):
assignee: Clint Byrum (clint-fewbar) → Chuck Short (zulcss)
Thierry Carrez (ttx)
Changed in php5 (Ubuntu):
assignee: Chuck Short (zulcss) → Thierry Carrez (ttx)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package php5 - 5.3.3-1ubuntu3

---------------
php5 (5.3.3-1ubuntu3) maverick; urgency=low

  * debian/patches/lp564920-fix-big-files.patch: Fix downloading of large
    files (LP: #564920)
 -- Clint Byrum <email address hidden> Fri, 06 Aug 2010 13:10:17 -0700

Changed in php5 (Ubuntu):
status: In Progress → Fix Released
Changed in php5 (Ubuntu Lucid):
status: New → Confirmed
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Regarding the SRU to lucid:
https://code.launchpad.net/~clint-fewbar/ubuntu/lucid/php5/lucid-sru-lp564920/+merge/32803

IMPACT:

This bug fix is intended to make it possible to access/download large files. Users who attempt it now are given a file of the wrong size (silently) as the pointer appears to wrap.

DEV RELEASE STATUS(maverick):

This has been fixed in version 5.3.3-1ubuntu3 in Maverick

TEST CASE:

Please see https://bugs.launchpad.net/ubuntu/+source/php5/+bug/564920/comments/4

REGRESSION:

This fix is known to regress the upstream bug fix to php bug #48309

http://bugs.php.net/bug.php?id=48309

However, that bug is far less critical than this one, as it can be worked around in php code, while this cannot.

Thierry Carrez (ttx)
Changed in php5 (Ubuntu Lucid):
assignee: nobody → Thierry Carrez (ttx)
importance: Undecided → Medium
status: Confirmed → In Progress
Revision history for this message
Thierry Carrez (ttx) wrote :

Uploaded to lucid-proposed

Changed in php5 (Ubuntu Lucid):
status: In Progress → Confirmed
assignee: Thierry Carrez (ttx) → nobody
Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Accepted php5 into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in php5 (Ubuntu Lucid):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

FYI, on monday we will be releasing security updates for php that will have versioning newer than the package currently in -proposed. I will upload an updated package to -proposed once the security updates come out.

Revision history for this message
R. Jones (robejo05) wrote :

Marc:

Will the next version have the proper support for handling large files (> 2^32 bytes) in it?

Test case for this is listed above.

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

The security update coming out monday will not. Monday, a new package that does contain the fix will be uploaded to -proposed for the required testing waiting period.

Revision history for this message
R. Jones (robejo05) wrote :

The proposed fix (in lucid-proposed as 5.3.2-1ubuntu4.3.) corrects this bug. I have now tested version 5.3.2-1ubuntu4.3 and successfully downloaded a large file (> 2^32 bytes) via a dynamic PHP script on a 64 bit Ubuntu 10.04.1 based system.

Revision history for this message
R. Jones (robejo05) wrote :

The proposed fix (in lucid-proposed as 5.3.2-1ubuntu4.3.) corrects this bug. I have now tested version 5.3.2-1ubuntu4.3 and successfully downloaded a large file (> 2^32 bytes) via a dynamic PHP script on a 64 bit Ubuntu 10.04.1 based system.

Just to add details:

Test was with the PHP script ( https://bugs.edge.launchpad.net/ubuntu/+source/php5/+bug/564920/+attachment/1338515/+files/test.php ) listed in comment 3 of the bug report ( https://bugs.edge.launchpad.net/ubuntu/+source/php5/+bug/564920/comments/4 ). My large test file on one test system is 4697126912 bytes (4.4G) in size.

Do you need any further input from me at this time? Once the next proposed update comes out, do you need anything from me other than to re-run the test at that time and provide similar comments on the test results?

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Thanks for testing it!

The packages that will get pushed to -proposed on monday will contain the same bug fix, on top of the security updates. Once the one week waiting period is up, they will get pushed to -updates. Since it's the same fix you've already tested, I don't think the SRU team will require more testing. Of course, feel free to test them and comment here if you'd like.

Revision history for this message
Martin Pitt (pitti) wrote :

Marc,

Now that this has been verified, would it be possible to fold this fix into the security update, to avoid yet another upload?

tags: added: verification-done
removed: verification-needed
Revision history for this message
Martin Pitt (pitti) wrote :

I can officially move this version to -updates on Monday morning or Sunday if that would help.

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Martin,

No problem, I will incorporate the fix into the security updates. No need to move the version currently in -proposed.

Thanks!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package php5 - 5.3.2-1ubuntu4.5

---------------
php5 (5.3.2-1ubuntu4.5) lucid-security; urgency=low

  * SECURITY UPDATE: denial of service and possible memory corruption via
    negative size in HTTP chunked encoding stream
    - debian/patches/CVE-2010-1866.patch: prevent chunk_size from
      overflowing in ext/standard/filters.c.
    - CVE-2010-1866
  * SECURITY UPDATE: arbitrary code execution via empty SQL query
    - debian/patches/CVE-2010-1868.patch: use ecalloc instead of emalloc in
      ext/sqlite/sqlite.c.
    - CVE-2010-1868
  * SECURITY UPDATE: denial of service via fnmatch stack consumption
    - debian/patches/CVE-2010-1917.patch: limit size of pattern in
      ext/standard/file.c.
    - CVE-2010-1917
  * SECURITY UPDATE: arbitrary memory disclosure and possible code
    execution via phar extension
    - debian/patches/CVE-2010-2094.patch: use correct format string in
      ext/phar/dirstream.c, ext/phar/stream.c.
    - CVE-2010-2094
    - CVE-2010-2950
  * SECURITY UPDATE: sensitive information disclosure or arbitrary code
    execution via use-after-free in SplObjectStorage unserializer
    - debian/patches/CVE-2010-2225.patch: fix logic in
      ext/spl/spl_observer.c, ext/standard/{php_var.h,var_unserializer.*},
      add tests to ext/spl/tests.
    - CVE-2010-2225
  * SECURITY UPDATE: sensitive information disclosure via error messages
    - debian/patches/CVE-2010-2531.patch: don't display data when flushing
      output buffer in ext/standard/{var.c,php_var.h}, fix tests in
      ext/standard/tests/general_functions.
    - CVE-2010-2531
  * SECURITY UPDATE: arbitrary session variable modification via crafted
    session variable name
    - debian/patches/CVE-2010-3065.patch: handle PS_UNDEF_MARKER marker in
      ext/session/session.c.
    - CVE-2010-3065
  * debian/patches/lp564920-fix-big-files.patch: Fix downloading of large
    files (LP: #564920)
 -- Marc Deslauriers <email address hidden> Fri, 17 Sep 2010 08:14:26 -0400

Changed in php5 (Ubuntu Lucid):
status: Fix Committed → Fix Released
Revision history for this message
R. Jones (robejo05) wrote :

Now that a fix has been released for Ubuntu 10.04, what needs to happen to get this fixed in the PHP 5 code base? PHP bug http://bugs.php.net/bug.php?id=52102 was closed out as a duplicate of http://bugs.php.net/bug.php?id=44522 , which it isn't an exact duplicate? How do we get the PHP5 code base updated with an acceptable fix?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.