OPcache PHP autoloader crashes on PHP8 < 8.1.6

[Kraut.Hosting]

[Impact]

This is a known OPcache regression in 8.1 versions below 8.1.6 upstream release which has been fixed upstream. The bug manifests itself through PHP autoloader crashes.
Known affected use cases include Drupal, which returns 500 errores on such PHP crashes. New Drupal versions warn it does not supporting PHP < 8.1.6 due to this issue, which has been reported and fixed in https://github.com/php/php-src/issues/8164.

[Test Plan]

While there is no minimal known test case to fully reproduce the issue and its side-effects, the upstream patch proposed includes unit-tests for the proposed fix. These tests are aimed at a different bug fixed through the same patch, and cannot confirm this fix, but should confirm no regressions are introduced. Finally, we should rely on affected users to test the patched package and verify there are no manifestations of the bug in a few days interval.

[Where problems could occur]

 * Regression are potentially isolated to the OPcache area of PHP;
 * Such changes usually tend to affect FPM instead of the CLI but this depends on user configuration;
 * An SRU would cause FPM to restart (if in use) and hence wipe previous OPcache data (this should be expected by the user for every PHP SRU);
 * The patch in question was introduced in php 8.1.6, therefore, the backport could trigger unexpected codepaths not predicted during our verification steps. If we get to this point, we will need a new regression bug report and SRU.

[Other Info]

 * The Drupal community is starting to deny-list Ubuntu's PHP due to this issue:
   https://www.drupal.org/project/drupal/issues/3258987
 * SRU with the backport is a minimum to actually fix the breakage
 * PHP developers argue against SRU backports and demand 8.1.6

[Kraut.Hosting (kraut.hosting)]

@Marc @Athos Hey there, seems you got some work on 8.1.5 floating in ubuntu/devel :D
Would you mind giving it a push to 8.1.6 and push to ubuntu/jammy-devel for testing?

[Athos Ribeiro (athos-ribeiro)]

Hi Kraut,

Thanks for taking the time to report this bug.

After reading
https://github.com/php/php-src/issues/8164,
it seems that we do not currently have a straightforward reproducer for the bug. Would you be able to provide one? This would be quite helpful here when it comes to uploading a fix for the issue (keep reading below).

In Ubuntu stable releases, we usually apply patches following the SRU process
(see https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template)
instead of updating software to newer versions. There are exceptions to this process, which are documented in https://wiki.ubuntu.com/StableReleaseUpdates#Documentation_for_Special_Cases, but PHP is currently not part of those exceptions.

Circling back to the linked github issue, it seems that
https://github.com/php/php-src/pull/8297
was the fix that closed that issue. Therefore, applying that patch should hopefully fix the issue you have been experiencing.

For kinetic (currently development release) I'd rather merge Debian's PHP 8.1.7, which should contain the fix. I will start working on this merge soon.

Now for jammy, while the upstream bug and the drupal bug could not provide a reliable, minimal reproducer [1], it would be nice if we could find one so we are able to confirm that that one specific patch is indeed the fix for this issue. Once again, would you be able to provide one while I work on the fix for kinetic? If not, I will get to that after I am done working on the kinetic merge.

[1]:
"""
Steps to reproduce

There are no obvious steps to reproduce, you need a module installed which triggers a PHP deprecation notice, possibly php-fpm, and it may take several days for this error to appear.
"""

[Kraut.Hosting (kraut.hosting)]

Hey Athos, thanks for your full reply :) Got your point about SRU for jammy.

Wanted to build and test merging current sid and slurp Debian's latest work.
But "git ubuntu merge start ubuntu/devel" got changelog values not agreeing.
Desired to stayed in the git workflow but sure can also just apply a patch.

Will try to build and distribute packages to PHP devs for test & reproducing.
Know bunch of folks on Drupal bug report in person and trust their feedback.
Hope someone got time/patience for a reproducer to our cherry picked fix :)

Overall without SRU exception i see more downstream work for point releases.
Keeping point release for PHP8.1 in sync with Debian could ease maintenance.
Meaning jammy and bookworm will share quite some time in their life-cycles.

But sure off topic so let's first get this bug squashed without SRU. Thanks :)

[Kraut.Hosting (kraut.hosting)]

@Athos Pushed now a backport only build for easy testing:
https://launchpad.net/~kraut.hosting/+archive/ubuntu/php
A version 8.1.2-1ubuntu3 would supersede this for jammy.
Can you debdiff if the bug fix changes are as intended?
Will reach out into Drupal community after your confirm :)

[Athos Ribeiro (athos-ribeiro)]

Hi Kraut,

Thanks for driving that one.

I just uploaded PHP 8.1.7 to kinetic [1], which should fix the bug in the development release and unblock us to possibly start the SRU process to get this fixed in jammy.

Your PPA looks good. Would you like to provide a debdiff in this bug so we could use it in the SRU process?

Here is a few improvements it would be nice to apply to the package in your PPA before submitting the debdiff here:

- Could you change the version to "8.1.2-1ubuntu2.3"? I am driving another, unrelated, php SRU and I will need to change the version there myself (I will use "2.4" or merge my changes with yours before the upload if we manage to finish our works in a similar time frame).

- Could you extract your patch file directly from https://github.com/php/php-src/commit/f20e11cbe12057bddade6ff9a2ff9ea1bb6084c9 with `git format-patch`? Then you can keep all the headers in the patch file, as git format-patch prepares it. It contains useful information and will also help in the next point below.

- Could you fix the DEP3 headers? You can add them in the end of the file. The final patch would look similar to [3].

- Could you also reference this bug both in the changelog and in the DEP3 header? In this case, the changelog entry would look like "d/p/0046-Fix-Opcache-breaks-autoloading.patch: backport OPcache fix from 8.1.6 (LP: #1983205)".

- Finally, if you want to push the SRU forward, you would need to file the SRU template in this bug. you can do this by editing this bug description by filling the entries in [4]. You can also add a new section in the end such as "[Original message]" To keep the old description of this bug under it. In [5] you will find an example of this template being filed. Do note that the "[Test Plan]" section contains a complete guide on how someone could verify that the change fixes the bug. In our case, if finding such test/reproducer is not feasible, as we discussed before in this bug, you could say that one should run php with some setup you know will trigger the bug for N hours or days and verify that the error do not occur.

Once 8.1.7 migrates to kinetic, this SRU template is filled, and we have a debdiff attached to this bug, I will upload your changes on your behalf (i.e., sponsor your changes).

Do note that we are currently in a freeze for the 22.04.1. So we may need to wait until 22.04.1 is out so we can land your changes.

[1] https://launchpad.net/ubuntu/+source/php8.1/8.1.7-1ubuntu1
[2] https://dep-team.pages.debian.net/deps/dep3/
[3] https://git.launchpad.net/ubuntu/+source/php8.1/tree/debian/patches/0046-Update-gcc-func-attr-macro.patch?h=ubuntu/kinetic-devel
[4] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template
[5] https://bugs.launchpad.net/ubuntu/+source/php8.1/+bug/1975626

[Launchpad Janitor (janitor)]

This bug was fixed in the package php8.1 - 8.1.7-1ubuntu1

---------------
php8.1 (8.1.7-1ubuntu1) kinetic; urgency=medium

  * Merge with Debian unstable (LP: #1983285, #1983205). Remaining changes:
    - Force upgrade from earlier mod-php's to version 8.1 (LP #1890263):
      + d/control: add transitional packages and Breaks/Replaces.
      + d/rules: exclude transitional packages in dh_install.
    - d/rules: Don't fill up build log with pedantic warnings.
    - d/rules: document garbage collection in ini files. (LP #1772915)
    - SECURITY UPDATE: Memory corruption in libmagic
      + debian/patches/CVE-2022-31627.patch: use the same memory allocator in
        ext/fileinfo/libmagic.patch, ext/fileinfo/libmagic/softmagic.c,
        ext/fileinfo/tests/bug81723.phpt.
      + CVE-2022-31627
  * Dropped changes:
    - d/p/0046-Update-gcc-func-attr-macro.patch: fix detection of unknown gcc
      function attributes. (LP #1882279)
      [ Fixed in 8.1.7-1 ]
    - d/p/0047-Fix-ssl3-unexpected-eof.patch: fix OpenSSL3 related
      unexpected EOF failure. (LP #1975626)
      [ Fixed in 8.1.7-1 ]
    - SECURITY UPDATE: RCE via Uninitialized array in pg_query_params()
      + debian/patches/CVE-2022-31625.patch: don't free parameters which
        haven't initialized yet in ext/pgsql/pgsql.c,
        ext/pgsql/tests/bug81720.phpt.
      + CVE-2022-31625
      [ Fixed in 8.1.7-1 ]
    - SECURITY UPDATE: RCE via mysqlnd/pdo password buffer overflow
      + debian/patches/CVE-20022-31626.patch: properly calculate size in
        ext/mysqlnd/mysqlnd_wireprotocol.c.
      + CVE-2022-31626
      [ Fixed in 8.1.7-1 ]

 -- Athos Ribeiro <email address hidden> Mon, 01 Aug 2022 17:04:27 -0300

[Kraut.Hosting (kraut.hosting)]

@Athos Thanks for your detailed feedback and suggestion :)
Version lowered, patch reformated and bug mentioned now:
https://launchpad.net/~kraut.hosting/+archive/ubuntu/php
For the SRU template i like to gather community feedback:
https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template
But the debdiff you find already attached for your review.

[Athos Ribeiro (athos-ribeiro)]

Hi Kraut,

Thanks for the debdiff, it looks much better now. I have only one minor nitpick regarding the DEP3 headers in the patch: the "Forwarded" field is not needed since we already have the Origin field in there. No need to update the debdiff though, we can do this after we gather the community feedback and have the SRU template ready, when I upload it (I can remove the field then, if you agree).

[Kraut.Hosting (kraut.hosting)]

@Athos Agree on DEP-3 minor and changed. Also add SRU template.
Community feedback bit low. Refers to upstream's test suite.
Given you landed something new in proposed can you take over?

[Stefan Mueller (mullzk)]

Hi Kraut
> Community feedback bit low. Refers to upstream's test suite.

So that you get at least one feedback, although not the most competent one: Thank you, your fix works for us.
Environment: Ubuntu 22.04, PHP 8.1.2, Drupal 9.4.5

[Impact]
 * In two of our Drupal Installations, every time Drupal wants to show an Error (e.g. HTTP Error 403
on unauthorized access), PHP crashes on its autoloader and Drupal produces a Server-Error 500.
We also had the situation were a PHP deprecated warning became an Error 500, but I am not able to reproduce this situation as the deprecated code was removed.
The issue was introduced when we upgraded from 18.04 to 22.04 and from PHP 8.0 to 8.1.2.

[Test Plan]
 * We can reproduce the bug in our Drupal Installation every time, by accessing the (login-protected) URL /admin. Unfortunately 1, a complete Drupal Installation is not a good and definitly not a minimal test. Unfortunately 2, we did not experience the bug in a fresh Drupal installation yet - so even a dockerized drupal would probably not be a reproducing test.
 * When we install your PPA, the bug vanishes, the errors are presented as expected.

[Other Info]
 * The commands in the PPA for adding the PPA to the system did not work out with us, sudo apt update did not install it. In the end, we had to re-install all php:
```
sudo add-apt-repository ppa:kraut.hosting/php
sudo apt remove php8.1-*
sudo apt install php8.1-*=8.1.2-1ubuntu2.3~kraut.hosting1~jammy1
```
 * Our sysadmin would like to stay on the official release. It would be great if the SRU could backport the fix to 8.1.2 in a timely manner and we are not forced to update to 8.1.7

[Athos Ribeiro (athos-ribeiro)]

Thanks, Stefan!

Hi Kraut,

I updated the SRU template expanding some of your topics (and using Stefan's suggestions as well).

I will now file a MP to get an extra pair of eyes in these changes and proceed with sponsoring the upload.

Kraus, Stefan, would you be able to verify these changes if/once they are accepted by the SRU team in the -proposed pocket?

[Athos Ribeiro (athos-ribeiro)]

Hello Kraut,

I filed a MP with your changes in https://code.launchpad.net/~athos-ribeiro/ubuntu/+source/php8.1/+git/php8.1/+merge/429179. I rebased your changes on top of the most recent jammy SRU and performed minor cosmetic changes. Please, confirm this is OK before we upload the package.

I pushed the proposed changes to the following PPA: https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp1983205-opcache-php

[Kraut.Hosting (kraut.hosting)]

@Athos Approved in MR/MP as all good and was about to rebase as well :)
Updated our PPA to supersede for folks till SRU lands in jammy-updates.
BTW nice work you did with the GCC optimizations via 8.1.2-1ubuntu2.4 :D

Can you recommend how to check from PHP context it's a fixed version?
https://www.drupal.org/project/drupal/issues/3258987#comment-14673470
I.e. Drupal core will still warn despite fix as version below 8.1.6.

[Athos Ribeiro (athos-ribeiro)]

Hi Kraut,

To answer your question, I extracted the test cases introduced with the upstream patch to provide a short reproducer for the bug. It seems it is not enough to provide information on the crash reported in this bug. Meaning we will need to rely on your verification once/if this SRU is accepted.

For the Drupal case, they could suggest that users on Ubuntu Jammy verify their PHP version and if that is >= 8.1.2-1ubuntu2.5, then they could ignore the warnings. They could even perform some checks to see if the users are running this patched version (by checking /etc/os-realease and/or verifying the version with dpkg), but then this could become tricky for users running multiple PHP installations.

[mark burdett (mfb)]

Is there any reason that Ubuntu no longer updates the PHP_EXTRA_VERSION part of PHP_VERSION - see https://www.php.net/manual/en/reserved.constants.php#constant.php-extra-version

I found some code in the debian package build script that attempts to update EXTRA_VERSION, but it seems to no longer be working since PHP 7.4 - the Ubuntu Bionic PHP_VERSION was "7.2.24-0ubuntu0.18.04.13" but Focal is "7.4.3" and Jammy is "8.1.2"

[mark burdett (mfb)]

See also this upstream report re: PHP 7.4 needing some extra finessing to set EXTRA_VERSION: https://bugs.php.net/bug.php?id=79026

[Kraut.Hosting (kraut.hosting)]

@Athos Regarding verification guess we can advise people to enable jammy-proposed?
We will test an SRU but would love to get more Drupal devs like mfb to test as well :)

Version compares via /etc and /usr aren't doable because of open_basedir security:
https://www.php.net/manual/en/ini.core.php#ini.open-basedir
Wouldn't expect professional PHP hostings to not have that enforced in production.
Know from Typo3 nasty checks to test "/usr/bin/gm" instead just clean the PHP libs.

Hence the desire via this Drupal bug to use version reference inside the PHP runtime:
https://www.drupal.org/project/drupal/issues/3307248

[Timo Aaltonen (tjaalton)]

Hello Kraut.Hosting, or anyone else affected,

Accepted php8.1 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php8.1/8.1.2-1ubuntu2.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Athos Ribeiro (athos-ribeiro)]

Hi Mark,

Thanks for the hint on PHP_EXTRA_VERSION no longer working. I filed LP: #1989196 and will start working on it. If we are able to backport that to jammy soon, this should provide a simple path to patch the Drupal warning upstream to allow specific (patched) Ubuntu releases, if they are willing to hadcode that info in their codebase.

Kraut,

I believe I have enough to continue providing means to get this checked, as commented above, thanks!
Regarding verification, I wouldn't advise enabling proposed for all packages in a production environment. If you have testing/staging environments to test, you could enable proposed solely to upgrade php (or just pick the php packages from proposed) and then disable it.

Note that this is now in proposed, as reported by Timo. Let us know when you are able to complete this verification steps and are satisfied with the patch! I will keep following up with LP: #1989196

[Stefan Mueller (mullzk)]

Hi Timo and everyone

I just tested jammy-proposed on our Drupal-Installation. Result: It fixes our issue with the bug.

On php8.1=8.1.2-1ubuntu2.4/jammy-updates, when accessing a Drupal-Site, every PHP-Warning or -Notice escalates to a PHP Fatal Error, resulting in a Error500-Page displayed to the User.
On php8.1=8.1.2-1ubuntu2.5/jammy-proposed, when accessing a Drupal-Site, the Warnings and Notices are logged, but they do not escalate and the User sees the requested page.

This can be 100% reproduced in some, but not in all our Drupal-Environments. Unfortunately, we cannot produce a good and minimal test.

Thanks everyone who has part in this bugfix.

[Athos Ribeiro (athos-ribeiro)]

Thanks, Stefan!

[Launchpad Janitor (janitor)]

This bug was fixed in the package php8.1 - 8.1.2-1ubuntu2.5

---------------
php8.1 (8.1.2-1ubuntu2.5) jammy; urgency=medium

  * d/p/0048-Clear-recorded-errors-before-executing-shutdown-func.patch:
    backport OPcache autoloading fix from 8.1.6. (LP: #1983205)

 -- <email address hidden> (Kraut.Hosting) Mon, 08 Aug 2022 09:28:23 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for php8.1 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.