zend_map_ptr opcache-less php8.1-fpm memory leak

[Patrick Othmer]

[ Impact ]

When opcache is off, php-fpm stores references to interned strings for class names in a map for every request. During the request clean-up phase, these strings and their references are cleaned up, but the map pointer is not reset. The map keeps getting larger as more class names are interned, resulting in high memory consumption (leakage) when too many classes are declared, given the php-fpm service runs for long periods (which should be expected).

[ Test plan ]

Set up php-fpm with opcache turned off:

# apt update && apt install -y php-fpm php apache2 php-ds
# a2enmod proxy_fcgi proxy

set /etc/apache2/sites-enabled/000-default.conf with the following:
<VirtualHost *:80>
 ServerAdmin webmaster@localhost
 DocumentRoot /var/www/html
<Directory /var/www/html>
        Options -Indexes +FollowSymLinks +MultiViews
        AllowOverride All
        Require all granted
    </Directory>
    <FilesMatch \.php$>
        SetHandler "proxy:unix:/run/php/php8.1-fpm.sock|fcgi://localhost/"
    </FilesMatch>
 ErrorLog ${APACHE_LOG_DIR}/error.log
 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Edit /etc/php/8.1/fpm/php.ini to uncomment the line with opcache.enable and set it to 0:
opcache.enable=0

Restart the services

# systemctl restart apache2
# systemctl restart php8.1-fpm

Then, set a test page with

# echo '<?php phpinfo(); ?>' > /var/www/html/test.php

and browse to IP_ADDR/test.php to ensure we are running FPM and opcache is off (just search for the info in this page).

Now, create the following test script:

create /var/www/html/classes.php with the following contents:
<?php

$set = new \Ds\Set();
for($i=0;$i<100000;$i++) {
 $random = substr(md5(mt_rand()), 0, 7);
 $class_name = "C" . $random;
 if($set->contains($class_name)) {
  continue;
 }
 $set->add($class_name);
 var_dump($class_name);
 eval("class $class_name {};");
}

and start performing requests for that script (e.g.,

# while true; do curl -sS http://10.195.56.78/classes.php > /dev/null; done

)

In affected systems, you should see the memory allocated to the fpm process slowly increasing indefinitely. A simple way to verify memory consumption would be through htop (e.g., "htop -p PID" do track php-fpm)

[ Where problems could occur ]

The proposed fix calls "zend_map_ptr_reset();" when class name strings are interned, to ensure the map ptr is reset for the next request. A potential issue would most likely manifest by making requests slower, in case any references in the map ptr should indeed be stored (and re-used) for the next request. However, I would not expect such potential delays to be easily perceived.

Moreover, a larger change is being applied to the test suite to account for the changes in the map ptr so the fix can be tested. If any of the patches in our current delta (or from debian) indirectly changes the map ptr during the test run, we may experience test flakiness, although this is unlikely.

Finally, see the "Other info" section bellow.

[ Other info ]

The upstream bug which resulted in the proposed fix (https://github.com/php/php-src/commit/ff62d117a35509699f8bac8b0750a956914da1b7) was re-opened because different users were potentially dealing with different leakage issues.

This fix may be incomplete and we may need a follow-up leakage fix in the near future (users may continue to experience memory leakage).

[ Original message ]

The PHP-Team fixed an memory leak issue in the newest version: https://github.com/php/php-src/issues/8646

Changelog: https://www.php.net/ChangeLog-8.php#8.1.18

Is it possible to get the fix also in the current supported php-fpm versions?

[Athos Ribeiro (athos-ribeiro)]

Hi Patrick,

thanks for reporting this one.

Would you also be able to provide a reproducer? It would be nice to understand how exactly we are affected since Ubuntu's PHP does not use the regular PHP GC (it uses a cron job. LP: #1772915 has more context on it).

Upstream bug: https://github.com/php/php-src/issues/8646
Upstream fix PR: https://github.com/php/php-src/pull/10783
Actual fix: https://github.com/php/php-src/commit/ff62d117a35509699f8bac8b0750a956914da1b7

[Athos Ribeiro (athos-ribeiro)]

The upstream bug related to this fix at https://github.com/php/php-src/issues/8646 was re-opened. It seems different reporters are dealing with different memory leak issues in PHP 8.1.

The fix provided in 8.1.18 is related to a interned strings clean up after a request (they were not being properly cleaned is specific situations).

The next step here will be to reproduce the issue with our current package, provide a fix and file a second bug to keep tracking the other memory leak issues in that upstream bug.

[Steve Langasek (vorlon)]

Hello Patrick, or anyone else affected,

Accepted php8.1 into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php8.1/8.1.12-1ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Patrick, or anyone else affected,

Accepted php8.1 into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php8.1/8.1.7-1ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Patrick, or anyone else affected,

Accepted php8.1 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php8.1/8.1.2-1ubuntu2.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package php8.1 - 8.1.7-1ubuntu3.5

---------------
php8.1 (8.1.7-1ubuntu3.5) kinetic-security; urgency=medium

  * SECURITY UPDATE: Missing error check and insufficient random
    bytes
    - debian/patches/CVE-2023-3247-1.patch: fixes missing randomness
      check and insufficient random byes for SOAP HTTP digest
      in ext/soap/php_http.c.
    - debian/patches/CVE-2023-3247-2.patch: fix wrong backporting of previous
      soap patch.
    - CVE-2023-3247

 -- Leonidas Da Silva Barbosa <email address hidden> Wed, 28 Jun 2023 11:05:45 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package php8.1 - 8.1.2-1ubuntu2.13

---------------
php8.1 (8.1.2-1ubuntu2.13) jammy-security; urgency=medium

  * SECURITY UPDATE: Missing error check and insufficient random
    bytes
    - debian/patches/CVE-2023-3247-1.patch: fixes missing randomness
      check and insufficient random byes for SOAP HTTP digest
      in ext/soap/php_http.c.
    - debian/patches/CVE-2023-3247-2.patch: fix wrong backporting of previous
      soap patch.
    - CVE-2023-3247

 -- Leonidas Da Silva Barbosa <email address hidden> Wed, 28 Jun 2023 11:01:49 -0300

[Athos Ribeiro (athos-ribeiro)]

These were all released by the security team. I am still performing the verification for this bug in the updates/security pocket to ensure we introduced no regressions here. I will report back as soon as I get my test results.

[Athos Ribeiro (athos-ribeiro)]

Following the test plan for the packages in the security/release pocket, I found out that the memory consumption stabilizes after a few minutes performing the proposed requests.

I am now marking this as verification-done.