OpenSSL 3 performance regression

[Seth Arnold]

Hello, it sounds like there's some significant performance regressions in OpenSSL 3:

https://github.com/openssl/openssl/issues/20286#issuecomment-1438826816

Some we might be able to address with:
https://github.com/openssl/openssl/pull/18151

Some of the performance differences may be subject to ongoing work.

Thanks

[Adrien Nader (adrien)]

We discussed this elsewhere but it's worth updating this ticket too.

22.04 and 24.04 will most likely both be on openssl 3.0. It seems unlikely there is another LTS in time for 24.04.

There are several tickets upstream about performance regression and I'm not aware of a few specific and isolated ones that reclaim that performance (I'm not even sure HEAD has reclaimed it yet). The more time passes by, the more difficult backports will be, especially since some of the changes seem to be related to the architecture.

Overall I'm not very confident we can tackle this.

[Bryce Harrington (bryce)]

Upstream has an umbrella bug that covers this and other associated performance related problems:

https://github.com/openssl/openssl/issues/17627

[Adrien Nader (adrien)]

I'm switching to bug to Confirmed but I still don't really know if we're going to be able to address this while on 3.0.

I'll try to discuss with at least one person involved in September but I also really wish there's a new LTS for 24.04.

[Rafael Lopez (rafael.lopez)]

The perf regression is easy to demonstrate using the 'main.py' script attached to this bug, also available here: https://pastebin.ubuntu.com/p/XzV766BpHd/

Some results -

ubuntu@focal-vm:~$ /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 20.04.6 LTS
Python Version: 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0]
OpenSSL Version: OpenSSL 1.1.1f 31 Mar 2020
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 1.65 s

root@jammy-vm:~# /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 22.04.2 LTS
Python Version: 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
OpenSSL Version: OpenSSL 3.0.2 15 Mar 2022
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 5.91 s

Both the above tests were done on a 1core/1G VM. Interestingly, when adding resources it exacerbates the problem, example using 8core/4G VMs (same hardware):

ubuntu@sslfocal-vm1:~$ /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 20.04.6 LTS
Python Version: 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0]
OpenSSL Version: OpenSSL 1.1.1f 31 Mar 2020
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 1.77 s

ubuntu@ssljammy-vm1:~$ /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 22.04.2 LTS
Python Version: 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
OpenSSL Version: OpenSSL 3.0.2 15 Mar 2022
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 12.69 s

[Rafael Lopez (rafael.lopez)]

Also, when replacing the distro libssl and libcrypto with ones built from upstream master, there is a massive improvement, close to 1.1 perf:

1core/1G:
root@jammy-vm:~# /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 22.04.2 LTS
Python Version: 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
OpenSSL Version: OpenSSL 3.2.0-dev
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 2.15 s
root@jammy-vm:~#

8core/4G:
root@ssljammy-vm1:~# /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 22.04.2 LTS
Python Version: 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
OpenSSL Version: OpenSSL 3.2.0-dev
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 1.87 s
root@ssljammy-vm1:~#

[Rafael Lopez (rafael.lopez)]

Script for testing OpenSSL performance, 100 requests using 30 workers.

[Adrien Nader (adrien)]

Rafael, could you do these tests on Lunar or Mantic too?

[Rafael Lopez (rafael.lopez)]

Hi Adrien, see results below - lunar and mantic show similar performance, both a significant improvement over jammy. All tests were run using same underlying hardware, 8vcpu/4G VMs.

ubuntu@openssl-jammy:~/openssl_python_ubuntu$ time python3 main.py
Distro: Ubuntu 22.04.3 LTS
Python Version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
OpenSSL Version: OpenSSL 3.0.2 15 Mar 2022
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
real 0m12.917s
user 0m21.726s
sys 0m10.047s
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 13.53 User: 22.79
Real: 13.59 User: 22.64
Real: 12.62 User: 22.84
Real: 12.48 User: 21.97
Real: 11.96 User: 21.54
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

ubuntu@openssl-lunar:~/openssl_python_ubuntu$ time python3 main.py
Distro: Ubuntu 23.04
Python Version: 3.11.2 (main, May 30 2023, 17:45:26) [GCC 12.2.0]
OpenSSL Version: OpenSSL 3.0.8 7 Feb 2023
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
real 0m2.492s
user 0m13.588s
sys 0m0.150s
ubuntu@openssl-lunar:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 2.36 User: 13.56
Real: 2.68 User: 14.76
Real: 2.55 User: 13.60
Real: 2.81 User: 15.37
Real: 2.51 User: 13.16
ubuntu@openssl-lunar:~/openssl_python_ubuntu$

ubuntu@openssl-mantic:~/openssl_python_ubuntu$ time python3 main.py
Distro: Ubuntu Mantic Minotaur (development branch)
Python Version: 3.11.4 (main, Jun 7 2023, 10:13:09) [GCC 12.3.0]
OpenSSL Version: OpenSSL 3.0.10 1 Aug 2023
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
real 0m2.324s
user 0m14.769s
sys 0m0.147s
ubuntu@openssl-mantic:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 2.63 User: 14.47
Real: 2.57 User: 14.82
Real: 2.27 User: 13.22
Real: 2.39 User: 12.82
Real: 2.71 User: 15.10
ubuntu@openssl-mantic:~/openssl_python_ubuntu$

[Rafael Lopez (rafael.lopez)]

Tested some additional patches, including [1] (PR linked in bug desc) plus [2] and [3]. PPAs for each at [4], [5] and [6]. All tests on same hardware and VM 8vcpu/4G.

Baseline:
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 13.62 User: 24.44
Real: 13.84 User: 22.89
Real: 14.12 User: 23.71
Real: 13.72 User: 23.94
Real: 13.42 User: 24.31
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

PR#18151[1]
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 3.20 User: 13.89
Real: 2.43 User: 15.29
Real: 2.72 User: 14.88
Real: 2.77 User: 14.56
Real: 2.43 User: 15.46
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

PR#18151[1] + PR#17881[2]
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 2.02 User: 7.92
Real: 1.93 User: 7.87
Real: 1.89 User: 7.88
Real: 2.09 User: 8.46
Real: 1.89 User: 7.80
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

PR#18151[1] + PR#17921[3]
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 1.64 User: 5.30
Real: 1.74 User: 6.08
Real: 1.74 User: 5.80
Real: 1.69 User: 6.50
Real: 1.70 User: 5.68
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

PR#18151[1] + PR#17881[2] + PR#17921[3]
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 1.55 User: 3.79
Real: 1.64 User: 3.78
Real: 1.86 User: 4.36
Real: 1.73 User: 3.81
Real: 1.81 User: 3.72
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

[1] https://github.com/openssl/openssl/pull/18151
[2] https://github.com/openssl/openssl/pull/17881
[3] https://github.com/openssl/openssl/pull/17921
[4] https://launchpad.net/~rafael.lopez/+archive/ubuntu/lp2009544
[5] https://launchpad.net/~rafael.lopez/+archive/ubuntu/lp2009544-2
[6] https://launchpad.net/~rafael.lopez/+archive/ubuntu/lp2009544-3

[David Leadbeater (launchpad-net-dgl)]

We came across this and noticed the CA certificates parsing itself is one of the biggest slowdowns. One of the reasons OpenSSL ends up doing that parsing is ca-certificates puts the same certificates in /etc/ssl/certs/*.pem ("CApath") and /etc/ssl/certs/ca-certificates.crt ("CAfile"):

$ wc -l /etc/ssl/certs/ca-certificates.crt <(cat /etc/ssl/certs/*.pem)
   3431 /etc/ssl/certs/ca-certificates.crt
   3431 /dev/fd/63

On a mantic system, as a baseline:

$ time python3.11 main.py
Distro: Ubuntu 23.10
Python Version: 3.11.6 (main, Oct 8 2023, 05:06:43) [GCC 13.2.0]
OpenSSL Version: OpenSSL 3.0.10 1 Aug 2023
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
real 0m2.768s
user 0m9.666s
sys 0m0.124s

If I replace /etc/ssl/certs/ca-certificates.crt with a random one of the certificates (sadly an empty file results in errors), so there's much less parsing to be done, things are much faster:

$ sudo sh -c 'cat /etc/ssl/certs/002c0b4f.0 > /etc/ssl/certs/ca-certificates.crt'
$ time python3.11 main.py
Distro: Ubuntu 23.10
Python Version: 3.11.6 (main, Oct 8 2023, 05:06:43) [GCC 13.2.0]
OpenSSL Version: OpenSSL 3.0.10 1 Aug 2023
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
real 0m0.675s
user 0m0.781s
sys 0m0.059s

This also reproduces as a visible difference in user CPU time when simply running curl against a HTTPS site. Both Python and curl happily validate peers using the files from CApath.

I do wonder if somehow it would be possible to disable the CA file and only rely on the CA path. Likely easier said than done, particularly with the API changes in 3.x (splitting out the file loading into SSL_CTX_load_verify_file).

There's also some Python specific discussion in https://github.com/python/cpython/issues/95031 too, I did try setting requests.get(..., verify='/etc/ssl/certs/') in the test script to attempt to get requests to read only the directory per[1] but that didn't seem to work.

[1]: https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification

[Antoine (nuuantoine)]

Not easilly reproducible for others but one of our dotnet application uses openSSL to compute some cryptographic hash and here's the ratio of performances:
Ubuntu 20.04 + openSSL 1.1 : 100 %
Ubuntu 22.04 + openSSL 3.0 : 5.1%
Ubuntu 22.04 + openSSL 3.2 (custom compilation not at all realistic) : 50%
Those were run with latest available version availalble the 22 of march 2024 in official repo.
Those performance number are quite concerning I hope future release of openSSL 3.0 will address those performance issue which makes Ubuntu 22.04 (potentialy 24.04 as well) not usable for any openSSL heavy reliant workload.

[Adrien Nader (adrien)]

Due to openssl's release schedule, 24.04 Noble Numbat will still use 3.0. It will be 3.0.13 unless a 3.0.14 is released very soon.

After Noble Numbat is released, I will work on openssl 3.3 for the subsequent Ubuntu release. It is not yet released but will be soon so I might start with beta/RC. The openssl release schedule dictates that there will be another openssl LTS release by the time Ubuntu's next release (26.04) enters development.

Unfortunately there is little way around this due to openssl's own schedule and the need to have very long support.

[Adrien Nader (adrien)]

I'm going to target this to 24.10 as it's the first time it will be possible to "solve" it. As far as I understand, there will probably be performance loss with 3.3 compared to 1.1 but it's going to be a long tail rather than a few big changes which have been included in 3.1, 3.2 and 3.3.

Btw, Antoine, are you able to test with 3.3 beta? I'd like to know where we'll stand and if we should take additional steps. I'm also not opposed to performance backports for 22.04 but I must make it clear that these take time to author, test and validate, and also require calendar time (at which point the next release might very well be out).