PHP built from source performs much better than the Ubuntu packaged version

[Dustin Falgout]

[Impact]

PHP has a compilation option to use x86_64 v2 and v3 capabilities for some of its operations, when these capabilities are available during runtime. This compilation option is enabled by default, but due to a bug in an old version of a GCC macro embedded in the PHP code, this option is disabled when "-Wall" is present in CFLAGS.

Since PHP is compiled with "-Wall" in Ubuntu, but the upstream shipped binaries (including the ones in docker images) are not, users of modern x86_64 machines (with v2 or v3 capabilities) will perceive huge performance differences in some operations (such as base64 encoding/decoding) when running Ubuntu's PHP vs PHP distributions not compiled with "-Wall" (e.g., upstream docker images).

https://bugs.launchpad.net/ubuntu/+source/php8.1/+bug/1882279/comments/9 has a detailed assessment of the issue.

[Test Plan]

1) Verify the fix:

On a x86_64 machine with v2/v3 capabilities, run the following script with a version of PHP without the fix (jammy or focal)

# BEGIN
<?php

$time_start = microtime(true);
$test_string = 'Amet in elit incididunt qui qui sint consectetur do eiusmod eiusmod voluptate voluptate adipisicing aute. Mollit consectetur adipisicing ad nostrud duis voluptate in non. Culpa reprehenderit et laboris deserunt aliqua ut. Adipisicing ad aute ullamco reprehenderit ex amet occaecat eiusmod sit sint eiusmod ad aute eiusmod. Commodo dolore exercitation culpa do ad consectetur est adipisicing aliquip. In fugiat adipisicing culpa elit ad culpa fugiat.';
$about_one_sec = 200000;
$about_5_sec = $about_one_sec * 5;
for ($i=0; $i < $about_5_sec; $i++) {
    $_ = base64_encode($test_string);
    $_ = base64_decode($_);
}
$time_end = microtime(true);
$time = $time_end - $time_start;
echo $time;
die();
# END

Then, apply the fix and re-run the script.
The time it takes to run should have been greatly reduced (up to 10x from the perceived experimentation discussed in this bug).

2) Ensure no regressions are introduced in x86_64 v1 machines:

Run the same test above (from step 1) on a x86_64 v1 machine using an affected package, i.e., no fix applied (you can use a vm, e.g., virt-manager provides a menu with several CPUs to pick from, or actual v1 hardware). Make sure to register the time in the script output.

Apply the fix and re-run the test. Verify that it does not trigger an error (successful output) and that the script output (time) is in the same order of magnitude of the previous run (without the fix).

You can also run perf to verify that the test is calling php_base64_decode_ex and not php_base64_decode_ex_avx2 nor php_base64_decode_ex_sse3.

[Where problems could occur]

This patch will have PHP use codepaths that have been available in PHP for years, but were not used in known distributions (we verified, Ubuntu, Debian, Fedora, and CentOS). This could have 2 different side effects:

1) bugs in those codepaths, which may not have been exhaustively tested in production environments before, may be discovered. In these cases, we will need to work with upstream to get them fixed.

2) while there are, as far as we could examine and test, fallbacks in place for whenever the x86_64 v2 or v3 capabilities are not available for a certain operation (i.e., user is running on a x86_64 v1 machine), if there are any codepaths where a fallback is not triggered or is not available, PHP will crash and we will need to either work with upstream to fix that specific codepath, or revert this patch until we find a solution, which may be non-trivial.

[Other Info]

The fix is already available in kinetic. It was forwarded to Debian and accepted, and it was also accepted upstream, but was later reverted because it introduced issues in the MacOS builds due to linux specific codepaths being triggered due to the GCC macro fix.

[ Original message ]

When you compare the performance of PHP installed using this package with PHP built from source (as it is in the official PHP Docker image) there is a pretty substantial performance difference. It is not specific to Docker but using Docker is the easiest way to test it.

When the attached script is served using php-fpm built from source it performs 10 times better than using php-fpm installed via this package.

[Paride Legovini (paride)]

Hi Dustin and thanks for this bug report. Benchmarking is difficult and repeating one single operation several time is often not a good performance indicator, however I agree we shouldn't see such a huge difference. I can reproduce it too as follows:

1. Launch a Focal LXD container
2. apt install php
3. Run `php test.php` (your test script, unmodified). On my machine this it runs in about 1.5s.

Now in docker:

4. mkdir php-perf-test, copy test.php in this directory and create a Dockerfile containing the following:

FROM php:7.4-cli
COPY . /tmp/php
WORKDIR /tmp/php
CMD [ "php", "./test.php" ]

5. Run: `docker build -t my-php-app .`
6. Run: `docker run -it --rm --name my-running-app my-php-app`. The test script runs in about 0.25 in this case, on the same machine.

Version info for the Ubuntu packaged PHP:

$ apt policy php7.4
php7.4:
  Installed: 7.4.3-4ubuntu2.2

$ php --version
PHP 7.4.3 (cli) (built: May 26 2020 12:24:22) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.3, Copyright (c), by Zend Technologies

while the Docker version is:

$ docker run -it --rm --name my-running-app my-php-app php --version
PHP 7.4.6 (cli) (built: May 15 2020 12:47:30) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies

This is worth some more investigation.

[Paride Legovini (paride)]

I retitled the bug as even the version of PHP from the Docker image is pulled in binary form. It is not compiled from source on the user's machine.

[Dustin Falgout (lots0logs)]

Hi,

The docker image is built from source. Maybe you checked the wrong one? Here is the Dockerfile[1]

[1] https://github.com/docker-library/php/blob/master/7.4/buster/fpm/Dockerfile

[Paride Legovini (paride)]

@Dustin,

I used a Docker image that comes with php precompiled (php:7.4-cli, see my Dockerfile above) but still managed to reproduce the performance difference, so I don't think it's building from source on the same machine that will run the php perf test that makes the difference. The difference likely comes from a compile-time flag or optimization. I'm convinced that compiling php from the Ubuntu source package would produce the same "slow" php.

[Dustin Falgout (lots0logs)]

That is true but I am guessing that would be because of one of the patches in the Ubuntu source package. There are A LOT of patches. Most appear to be security related but I haven't had time to look past the filenames of the patches so take that with a grain of salt.

[Rafael David Tinoco (rafaeldtinoco)]

2 seconds in my old AMD (4.4 GHz) CPUs.. before and after local compilation, with perf record getting samples at the same frequency.

(c)rafaeldtinoco@groovy:~$ sudo perf report --stdio

# Overhead Command Shared Object Symbol
# ........ ....... ................. .................................................
#
    73.54% php php7.4 [.] php_base64_decode_ex
    21.96% php php7.4 [.] php_base64_encode
     1.07% php php7.4 [.] execute_ex

and if you annotate it:

https://paste.ubuntu.com/p/Qb9rvT84MX/

you're basically expending almost 75% of the time in decoding base64:

https://pastebin.ubuntu.com/p/YFcqrqfZBm/

but there isn't a single bottleneck, suggesting overall optimizations at compilation time, SPECIALLY if you judge the source code and generated instructions (by disassembling php_base64_decode_ex).

A bit amount of time (from these 75%) is in between the branches from line 274 <-> 264.. suggesting that lots of unknown chars are hitting continue; (~10% of 75%). Another considerable source of time is the bit shifts (~8% of the 75%).

next step here is *likely* doing a "perf diff" among the 2 different reports and it will show the main deviations from the 2 executions.. then you will have to identify:

- is the compiler putting a diff subset of instructions (like SIMD for vectors, usually using vectorization HW extensions/instructions - sse for x64 for example)
- is the compiler just doing better loop generation for the decoding ? (usually related to better intermediate code generation because of a specific flag)

etc...

this is the TODO here.

[Athos Ribeiro (athos-ribeiro)]

This is still valid for php 8.1;

Rafael's next steps seem to still be the path forward here.

[Athos Ribeiro (athos-ribeiro)]

While proceeding to the next steps on Rafael's analysis, I realized that the Ubuntu package and the self compiled version of PHP were calling different functions to process the same input file (the test file uploaded by the bug reporter).

The Ubuntu PHP package calls the function reported above (php_base64_decode_ex), while the local compiled package calls php_base64_decode_ex_avx2.

The locally compiled binary does process the test script 10x faster than the packaged one. This is also true for the Debian and the Fedora packages (they are slower) but not to the upstream PHP Docker image shipped in dockerhub under docker.io/php.

After some investigation, I realized that PHP will use different x86_64 instructions for some tasks when such instructions are available. That is either done when the binaries are compiled to target specific x86_64 micro-architectures or through the "target" function attribute provided by gcc (see https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html)

When specific (newer) x86_64 instructions are available, PHP will use them to speed up some specific tasks (such as base64 encoding/decoding). If they are not available, it falls back to other (older) available instructions.

For instance, in the base64 decoding case, it will use avx2, introduced in x86_64-v3 when available; it then falls back to sse3, introduced in x86_64-v2, and finally uses the common v1 instructions when neither is available.

The option to compile such additional functions is set during configuration time, through the "ax_cv_have_func_attribute_target" configuration. This configuration option is set by a modified version of an embedded gcc macro shipped in $PHP_SRC_ROOT/build/ax_gcc_func_attribute.m4.

The original macro is described at https://www.gnu.org/software/autoconf-archive/ax_gcc_func_attribute.html.

This macro relies on warnings being thrown to decide if the system supports a specific function attribute. The (old) version being shipped by PHP considers any warnings as a negative (i.e., the system does not support the function attribute in question), always generating false negatives when "-Wall" is enabled, since it relies on a test that declares and does not define a fuction, resulting in:

"warning: ‘bar’ declared ‘static’ but never defined [-Wunused-function]",

which leads the configuration step to always define

"ax_cv_have_func_attribute_target=no"

When "-Wall" is present in the CFLAGS (which is true for Ubuntu, Debian, Fedora, etc).

The issue with the macro have been fixed in autoconf-archive upstream at

http://git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=commitdiff;h=df0894ad1a8195df67a52108b931e07d708cec9a

but this has not been backported into the php embedded macro yet.

Updating the macro and rebuilding the package generates PHP binaries with the improved performance perceived by the bug reporter.

A PPA with the proposed change is available at
https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp1882279-php-perf/+packages.

I proposed updating the macro upstream at https://github.com/php/php-src/pull/8483.

I tested the patched package emulating different x86_64 CPUs, and it will either use the avx2,...

[Athos Ribeiro (athos-ribeiro)]

Debian salsa MR: https://salsa.debian.org/php-team/php/-/merge_requests/12

[Athos Ribeiro (athos-ribeiro)]

Since testing the proposed changes for i386 may be trickier for kinetic, I tested it in the following fashion:

- Deployed a Debian stable i386 VM and upgraded it to Debian testing (which contains php 8.1);
- Ensured the VM CPU supported the sse2 instruction (which is checked in the configuration test to enable the performance boost related to this bug;
- Built and installed Debian's PHP i386 package with the related patch (see https://salsa.debian.org/php-team/php/-/merge_requests/12);
- Ran the test script provided by this bug reporter to look for regressions (no regressions found, no performance boost observed);
- Changed the VM CPU to support avx2/sse3 instructions and re-ran the test. Now, the performance boost was observed;
- Finally, Changed the VM CPU for a pre-pentium 4 micro-architecture, where sse2 is not available (this is the most important i386 test here).
- Re-ran the test script. No performance boost observed. No regressions observed.

This provides enough evidence that no regressions should be introduced if the patch is introduced in stable releases for i386 users with machines that do not support avx2/sse3/sse2 instructions. It is important to note that this feature will be enabled in the i386 builds since LP builders do support sse2.

The next steps here are to introduce this change as a delta into kinetic and SRU it all the way back to focal.

[Launchpad Janitor (janitor)]

This bug was fixed in the package php8.1 - 8.1.2-1ubuntu4

---------------
php8.1 (8.1.2-1ubuntu4) kinetic; urgency=medium

  * d/p/0046-Update-gcc-func-attr-macro.patch: fix detection of unknown gcc
    function attributes. (LP: #1882279)
  * d/rules: document garbage collection in ini files. (LP: #1772915)

 -- Athos Ribeiro <email address hidden> Mon, 02 May 2022 19:54:49 -0300

[Athos Ribeiro (athos-ribeiro)]

I pushed patched packages for jammy and focal in the following PPA:
https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp1882279-php-performance/+packages

I will proceed to test those changes by running the upstream benchmark scripts on them and follow-up with MPs and fill an SRU template.

[Athos Ribeiro (athos-ribeiro)]

Here are the test results for running the test script attached to this bug report, i.e.,

# BEGIN
<?php

$time_start = microtime(true);
$test_string = 'Amet in elit incididunt qui qui sint consectetur do eiusmod eiusmod voluptate voluptate adipisicing aute. Mollit consectetur adipisicing ad nostrud duis voluptate in non. Culpa reprehenderit et laboris deserunt aliqua ut. Adipisicing ad aute ullamco reprehenderit ex amet occaecat eiusmod sit sint eiusmod ad aute eiusmod. Commodo dolore exercitation culpa do ad consectetur est adipisicing aliquip. In fugiat adipisicing culpa elit ad culpa fugiat.';
$about_one_sec = 200000;
$about_5_sec = $about_one_sec * 5;
for ($i=0; $i < $about_5_sec; $i++) {
    $_ = base64_encode($test_string);
    $_ = base64_decode($_);
}
$time_end = microtime(true);
$time = $time_end - $time_start;
echo $time;
die();
# END

kinetic (already fixed):
0.15323901176453

jammy:
1.0264580249786

jammy (with proposed patch):
0.15348196029663

focal:
1.023166179657

focal (with proposed patch):
0.1554548740387

This shows that base64 operations benefit from the patch both in jammy (php 8.1) and focal (php 7.4).

Next, we ran the upstream benchmark scripts for kinetic, jammy, jammy (with fix), focal, and focal (with fix).

The test consisted of running the 3 upstream shipped benchmark scripts

- php-src/Zend/bench.php
- php-src/Zend/micro_bench.php
- php-src/ext/hash/bench.php

30 times for each of the tested series and calculating the median time for each benchmark test.

This is the script used for the experiment:

# BEGIN
#!/bin/bash
set -ex

if [[ "$1" = fix ]]; then
 add-apt-repository -y ppa:athos-ribeiro/lp1882279-php-performance
fi

apt update && apt install -y php vim git python3
mkdir -p benchmark/hash benchmark/zend benchmark/zend_micro

cat > benchmark/hash/gen.sh <<EOF
#!/bin/bash
set -ex
for param in \$(cat hash_bench_1.out | cut -d ' ' -f1); do
 cat * | grep "^\${param} " | sed "s|\${param} *||" | python3 -c "import statistics; arr=[float(input()) for i in range(30)]; print('{:14s} {}'.format('\${param}',statistics.median(arr)))"
done
EOF

cat > benchmark/zend/gen.sh <<EOF
#!/bin/bash
set -ex
cat * | grep '^Total' | sed 's/Total *//' | python3 -c 'import statistics; arr=[float(input()) for i in range(30)]; print(statistics.median(arr))'
EOF

cp benchmark/zend/gen.sh benchmark/zend_micro/gen.sh

chmod +x benchmark/zend/gen.sh
chmod +x benchmark/zend_micro/gen.sh
chmod +x benchmark/hash/gen.sh

git clone https://github.com/php/php-src

for i in {1..30}; do php php-src/ext/hash/bench.php | sort > benchmark/hash/hash_bench_${i}.out; done
for i in {1..30}; do php php-src/Zend/bench.php > benchmark/zend/zend_bench_${i}.out; done
for i in {1..30}; do php php-src/Zend/micro_bench.php > benchmark/zend_micro/zend_micro_bench_${i}.out; done

for benchmark_dir in zend zend_micro hash; do
 pushd benchmark/${benchmark_dir}
 ./gen.sh > median.out
 popd
 cp benchmark/${benchmark_dir}/median.out ${benchmark_dir}_median.out
done
# END

For these tests, the results for the Zend benchmarks showed no performance improvements. The values varied between 0.41 and 0.4235 for php-src/Zend/bench.php, and 2.3 and 2.377 for php-src/Zend/micro_...

[Robie Basak (racb)]

Accepting, but please adjust the Test Plan to:

1) Verify that on a CPU without the required support, the fallback works as expected. I think you already did this while working on it in comment 11; I think it needs doing during SRU verification too, please.

2) Verify that in the fallback test above, the fallback code is actually active, and not accidentally using the higher performing path by accident (eg. to catch if testing in a VM and manually removing higher performing hardware support, but that manual removal didn't take full effect for some reason, such as a feature flag test not working as expected).

[Robie Basak (racb)]

> Verify that in the fallback test above, the fallback code is actually active, and not accidentally using the higher performing path by accident

Eg. by using gdb, bpf, perf, or just by explicitly looking at the performance difference as a heuristic of which code path actually ran.

[Robie Basak (racb)]

Hello Dustin, or anyone else affected,

Accepted php8.1 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php8.1/8.1.2-1ubuntu2.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello Dustin, or anyone else affected,

Accepted php7.4 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php7.4/7.4.3-4ubuntu2.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted php7.4 (7.4.3-4ubuntu2.13) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

php-luasandbox/3.0.3-2build2 (armhf, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#php7.4

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted php8.1 (8.1.2-1ubuntu2.4) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

php-luasandbox/4.0.2-3build1 (arm64, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#php8.1

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Athos Ribeiro (athos-ribeiro)]

Thanks, Robie.

I updated the test plan as requested.

Regarding the autopkgtest regressions above, they were both due to flaky tests, which were re-triggered and passed for this second run.

[Athos Ribeiro (athos-ribeiro)]

Verification steps:

on a clean focal/jammy installation, in a modern x86_64 (with v2/v3 capabilities) VM:
- installed php (without the proposed fix)
- ran the test script

focal: ~1.027
jammy: ~1.012

- installed php from the proposed pocket
- ran the test script

focal: ~0.135
jammy: ~0.136

on a clean focal/jammy installation, in a v1 x86_64 (no v2/v3 capabilities) VM:
- installed php (without the proposed fix)
- ran the test script

focal: ~1.037
jammy: ~1.020

- installed php from the proposed pocket
- ran the test script

focal: ~1.019
jammy: ~1.033

This shows that the fix improves the performance on v2/v3 capable machines without breaking php on v1 machines (covering test steps 1 and 2).

Next, we should show that the resolve_base64_decode path does not trigger calls to v2/v3 instructions on a v1 machine:

With gdb, for non-patched environments (jammy/focal), we verified that resolve_base64_decode, which is the function that resolves to the v2/v3 specific code, is not defined.

for patched environments (jammy/focal), we ran the test script with a break point in
resolve_base64_decode, which resolves to one of:

php_base64_dexode_ex_default;
php_base64_dexode_ex_avx2; or
php_base64_dexode_ex_ssse3.

We also set breakpoints to each of these functions.

We verified that in v2/v3 capable environments, resolve_base64_decode returns php_base64_dexode_ex_avx2 (v3). and php_base64_dexode_ex_avx2 is the only function called out of the three above.

For v1 environments, resolve_base64_decode returns php_base64_dexode_ex_default and the avx2/ssse3 functions are not called.

As far as we could verify with the data above, the fix is verified and introduce no regressions.

[Simon Quigley (tsimonq2)]

With 100 runs on a system before and after using a bash loop, I'm seeing a significant difference.

Without the fix averages: 2.17
With the fix averages: 0.18

That's about 2 seconds faster. Incredible.

Additionally, I tested this fix with OpenEMR, which is an open source PHP-based Electronic Medical Records site. It feels much faster and I'm seeing no issues.

From a pure usability standpoint, I can attest that this update does not have any immediate regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package php8.1 - 8.1.2-1ubuntu2.4

---------------
php8.1 (8.1.2-1ubuntu2.4) jammy; urgency=medium

  * d/p/0047-Update-gcc-func-attr-macro.patch: fix detection of unknown gcc
    function attributes. (LP: #1882279)

 -- Athos Ribeiro <email address hidden> Wed, 17 Aug 2022 10:08:39 -0300

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for php8.1 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package php7.4 - 7.4.3-4ubuntu2.13

---------------
php7.4 (7.4.3-4ubuntu2.13) focal; urgency=medium

  * d/p/0047-Update-gcc-func-attr-macro.patch: fix detection of unknown gcc
    function attributes. (LP: #1882279)

 -- Athos Ribeiro <email address hidden> Wed, 17 Aug 2022 10:29:56 -0300