Ubuntu 18.04.1 - OpenSSL RSA connection rate performance degradation using ibmca engine (openssl-ibmca)

[bugproxy]

[Impact]

 * hw accelerated crypto suffers from poor performance under certain configurations.

[Test Case]

---Steps to Reproduce---
 Server; openssl s_server -cert benchcert.pem -quiet -WWW -engine ibmca -accept 8050
Client: openssl s_time -key benchcert.pem -www /2k.html -time 90 -cipher AES256-SHA -new -bugs -connect 10.14.1.254:8050 -elapsed

By scaling the number of processes, the issue becomes more and more visible.

requires installing and configuring openssl-ibmca, on systems with hw crypto enabled on the LPAR in the HMC.

[Regression Potential]

 * There will be a change in crypto performance, when openssl-ibmca is installed and configured to be used; hopefully for the better.

[Other Info]

 * Original bug report:

---Problem Description---
Recent performance evaluation has shown significant degradation in the TLS connections per second rate using the OpenSSL s_time benchmark with Ubuntu 18.04.1.
While doing RSA sign/verify operations, the engine would preffer doing RSA-ME instead of RSA-CRT which is significantly better in terms of performance.

Baseline for this comparison are measurements executed with another distro.
Both measurements have been made on LPAR native, using the CEX6A adapter.

Crypto stack on the host:
OpenSSL ver: 1.1.0g
IBMCA ver: 1.4.1.-0
Libica ver: 3.2.1

Problem present under following condititions:
1. IBMCA ver >= 2.0.0
2. OpenSSL version >= 1.1.0 && IBMCA ver >= 1.3.1.

---uname output---
Linux m42lp01 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:42:24 UTC 2018 s390x s390x s390x GNU/Linux

Machine Type = Type/Model:3906-M04 LPAR

---Steps to Reproduce---
 Server; openssl s_server -cert benchcert.pem -quiet -WWW -engine ibmca -accept 8050
Client: openssl s_time -key benchcert.pem -www /2k.html -time 90 -cipher AES256-SHA -new -bugs -connect 10.14.1.254:8050 -elapsed

By scaling the number of processes, the issue becomes more and more visible.

Userspace tool common name: openssl-ibmca

The userspace tool has the following bit modes: 64

Userspace package: openssl-ibmca-1.4.1-0ubuntu1.s390x

The attached patch is generated from the commit available here:
https://github.com/opencryptoki/openssl-ibmca/commit/a0e23d4063bf897dd9136c491d2201de5fbba653

Generated with:
git format-patch -1 a0e23d4063bf897dd9136c491d2201de5fbba653

To be applied with:
patch /openssl-ibmca/src/ibmca_rsa.c ~/0001-Fix-doing-rsa-me-altough-rsa-crt-would-be-possible.patch

Fix applies smoothly and shows expected performance improvement as visible on the chart.

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-12-07 03:42 EDT-------
A sniff test showed that the upstream package version is not sufficient, the following patch (commit Id) is needed on top - please include it

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl-ibmca - 2.0.2-0ubuntu1

---------------
openssl-ibmca (2.0.2-0ubuntu1) disco; urgency=medium

  * New upstream release LP: #1804233 LP: #1806483
  * Drop dlopen-soname.patch, applied upstream.
  * Update watch file to github.com.

 -- Dimitri John Ledkov <email address hidden> Mon, 10 Dec 2018 11:21:56 +1100

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-12-10 08:36 EDT-------
Fix verified on package openssl-ibmca - 2.0.2-0ubuntu1.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-01-16 04:51 EDT-------
Fix Released for Disco.
Due to -> Problem present under following condititions:
1. IBMCA ver >= 2.0.0
2. OpenSSL version >= 1.1.0 && IBMCA ver >= 1.3.1.

which is correct with Bionic and later, no Xenial backport required.
The Xenial release can be deleted from this request

[Brian Murray (brian-murray)]

Hello bugproxy, or anyone else affected,

Accepted openssl-ibmca into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl-ibmca/2.0.0-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello bugproxy, or anyone else affected,

Accepted openssl-ibmca into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl-ibmca/1.4.1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-01 05:01 EDT-------
Confirmation by IBM that the fix included in the proposed openssl-ibmca_1.4.1-0ubuntu1.1_s390x.deb package applies and resolves the issue described within this bugzilla.

Distro version (openssl-ibmca/bionic,now 1.4.1-0ubuntu1) achieves 2373 conn/s having 16 clients connecting in parallel during 90 sec runtime.

Same test run with the proposed version ( openssl-ibmca_1.4.1-0ubuntu1.1) shows a 7380 conn/s rate.

[Frank Heimes (fheimes)]

Adjusting verification tags according to comment #9

[Łukasz Zemczak (sil2100)]

@Frank I only see verification of the bionic packages in comment #9 while you seem to have switched the tags to verification-done-cosmic. We still need to get the cosmic ones verified from what I see? Unswitching the tag.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-07 10:18 EDT-------
Hi,

openssl-ibmca 2.0.0-0ubuntu2.1 package from cosmic-proposed successfully verified. Same positive results like on bionic.

[Frank Heimes (fheimes)]

Thx Danijel - now (more carefully) adjusting the verification tags again according to comment #12 ...

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for openssl-ibmca has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl-ibmca - 2.0.0-0ubuntu2.1

---------------
openssl-ibmca (2.0.0-0ubuntu2.1) cosmic; urgency=medium

  * Cherrypick upstream hw accelerated crypto perfomance fix to prefer
    RSA-CRT, instead of RSA-ME. LP: #1806483

 -- Dimitri John Ledkov <email address hidden> Tue, 12 Feb 2019 13:49:47 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl-ibmca - 1.4.1-0ubuntu1.1

---------------
openssl-ibmca (1.4.1-0ubuntu1.1) bionic; urgency=medium

  * Cherrypick upstream hw accelerated crypto perfomance fix to prefer
    RSA-CRT, instead of RSA-ME. LP: #1806483

 -- Dimitri John Ledkov <email address hidden> Tue, 12 Feb 2019 13:56:35 +0000

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-15 06:20 EDT-------
IBM Bugzilla status -> closed. Fix Released for all requested distros