Ubuntu 18.04.1 - OpenSSL RSA connection rate performance degradation using ibmca engine (openssl-ibmca)

Bug #1806483 reported by bugproxy on 2018-12-03
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
High
Canonical Foundations Team
openssl-ibmca (Ubuntu)
Undecided
Skipper Bug Screeners
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned
Disco
Undecided
Skipper Bug Screeners

Bug Description

[Impact]

 * hw accelerated crypto suffers from poor performance under certain configurations.

[Test Case]

---Steps to Reproduce---
 Server; openssl s_server -cert benchcert.pem -quiet -WWW -engine ibmca -accept 8050
Client: openssl s_time -key benchcert.pem -www /2k.html -time 90 -cipher AES256-SHA -new -bugs -connect 10.14.1.254:8050 -elapsed

By scaling the number of processes, the issue becomes more and more visible.

requires installing and configuring openssl-ibmca, on systems with hw crypto enabled on the LPAR in the HMC.

[Regression Potential]

 * There will be a change in crypto performance, when openssl-ibmca is installed and configured to be used; hopefully for the better.

[Other Info]

 * Original bug report:

---Problem Description---
Recent performance evaluation has shown significant degradation in the TLS connections per second rate using the OpenSSL s_time benchmark with Ubuntu 18.04.1.
While doing RSA sign/verify operations, the engine would preffer doing RSA-ME instead of RSA-CRT which is significantly better in terms of performance.

Baseline for this comparison are measurements executed with another distro.
Both measurements have been made on LPAR native, using the CEX6A adapter.

Crypto stack on the host:
OpenSSL ver: 1.1.0g
IBMCA ver: 1.4.1.-0
Libica ver: 3.2.1

Problem present under following condititions:
1. IBMCA ver >= 2.0.0
2. OpenSSL version >= 1.1.0 && IBMCA ver >= 1.3.1.

---uname output---
Linux m42lp01 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:42:24 UTC 2018 s390x s390x s390x GNU/Linux

Machine Type = Type/Model:3906-M04 LPAR

---Steps to Reproduce---
 Server; openssl s_server -cert benchcert.pem -quiet -WWW -engine ibmca -accept 8050
Client: openssl s_time -key benchcert.pem -www /2k.html -time 90 -cipher AES256-SHA -new -bugs -connect 10.14.1.254:8050 -elapsed

By scaling the number of processes, the issue becomes more and more visible.

Userspace tool common name: openssl-ibmca

The userspace tool has the following bit modes: 64

Userspace package: openssl-ibmca-1.4.1-0ubuntu1.s390x

The attached patch is generated from the commit available here:
https://github.com/opencryptoki/openssl-ibmca/commit/a0e23d4063bf897dd9136c491d2201de5fbba653

Generated with:
git format-patch -1 a0e23d4063bf897dd9136c491d2201de5fbba653

To be applied with:
patch /openssl-ibmca/src/ibmca_rsa.c ~/0001-Fix-doing-rsa-me-altough-rsa-crt-would-be-possible.patch

Fix applies smoothly and shows expected performance improvement as visible on the chart.

Default Comment by Bridge

tags: added: architecture-s39064 bugnameltc-173745 severity-high targetmilestone-inin18043

Default Comment by Bridge

Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → openssl098 (Ubuntu)
Luciano Chavez (lnx1138) on 2018-12-03
affects: openssl098 (Ubuntu) → openssl-ibmca (Ubuntu)
Changed in ubuntu-z-systems:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Canonical Foundations Team (canonical-foundations)

------- Comment From <email address hidden> 2018-12-07 03:42 EDT-------
A sniff test showed that the upstream package version is not sufficient, the following patch (commit Id) is needed on top - please include it

tags: added: id-5c093a07c31aed1587ec4ab7
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssl-ibmca - 2.0.2-0ubuntu1

---------------
openssl-ibmca (2.0.2-0ubuntu1) disco; urgency=medium

  * New upstream release LP: #1804233 LP: #1806483
  * Drop dlopen-soname.patch, applied upstream.
  * Update watch file to github.com.

 -- Dimitri John Ledkov <email address hidden> Mon, 10 Dec 2018 11:21:56 +1100

Changed in openssl-ibmca (Ubuntu Disco):
status: New → Fix Released
Changed in ubuntu-z-systems:
status: Triaged → In Progress
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-12-10 08:36 EDT-------
Fix verified on package openssl-ibmca - 2.0.2-0ubuntu1.

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2019-01-16 04:51 EDT-------
Fix Released for Disco.
Due to -> Problem present under following condititions:
1. IBMCA ver >= 2.0.0
2. OpenSSL version >= 1.1.0 && IBMCA ver >= 1.3.1.

which is correct with Bionic and later, no Xenial backport required.
The Xenial release can be deleted from this request

Changed in openssl-ibmca (Ubuntu Xenial):
status: New → Invalid
Changed in openssl-ibmca (Ubuntu Cosmic):
status: New → In Progress
description: updated
Changed in openssl-ibmca (Ubuntu Bionic):
status: New → In Progress
tags: added: id-5c62d27150a5637232131cdf

Hello bugproxy, or anyone else affected,

Accepted openssl-ibmca into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl-ibmca/2.0.0-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openssl-ibmca (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Changed in openssl-ibmca (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Brian Murray (brian-murray) wrote :

Hello bugproxy, or anyone else affected,

Accepted openssl-ibmca into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl-ibmca/1.4.1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ubuntu-z-systems:
status: In Progress → Fix Committed

------- Comment From <email address hidden> 2019-03-01 05:01 EDT-------
Confirmation by IBM that the fix included in the proposed openssl-ibmca_1.4.1-0ubuntu1.1_s390x.deb package applies and resolves the issue described within this bugzilla.

Distro version (openssl-ibmca/bionic,now 1.4.1-0ubuntu1) achieves 2373 conn/s having 16 clients connecting in parallel during 90 sec runtime.

Same test run with the proposed version ( openssl-ibmca_1.4.1-0ubuntu1.1) shows a 7380 conn/s rate.

Frank Heimes (frank-heimes) wrote :

Adjusting verification tags according to comment #9

tags: added: verification-done verification-done-bionic verification-done-cosmic
removed: verification-needed verification-needed-bionic verification-needed-cosmic
Łukasz Zemczak (sil2100) wrote :

@Frank I only see verification of the bionic packages in comment #9 while you seem to have switched the tags to verification-done-cosmic. We still need to get the cosmic ones verified from what I see? Unswitching the tag.

tags: added: verification-needed verification-needed-cosmic
removed: verification-done verification-done-cosmic
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2019-03-07 10:18 EDT-------
Hi,

openssl-ibmca 2.0.0-0ubuntu2.1 package from cosmic-proposed successfully verified. Same positive results like on bionic.

Frank Heimes (frank-heimes) wrote :

Thx Danijel - now (more carefully) adjusting the verification tags again according to comment #12 ...

tags: added: verification-done-cosmic
removed: verification-needed-cosmic
tags: added: verification-done
removed: verification-needed

The verification of the Stable Release Update for openssl-ibmca has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssl-ibmca - 2.0.0-0ubuntu2.1

---------------
openssl-ibmca (2.0.0-0ubuntu2.1) cosmic; urgency=medium

  * Cherrypick upstream hw accelerated crypto perfomance fix to prefer
    RSA-CRT, instead of RSA-ME. LP: #1806483

 -- Dimitri John Ledkov <email address hidden> Tue, 12 Feb 2019 13:49:47 +0000

Changed in openssl-ibmca (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssl-ibmca - 1.4.1-0ubuntu1.1

---------------
openssl-ibmca (1.4.1-0ubuntu1.1) bionic; urgency=medium

  * Cherrypick upstream hw accelerated crypto perfomance fix to prefer
    RSA-CRT, instead of RSA-ME. LP: #1806483

 -- Dimitri John Ledkov <email address hidden> Tue, 12 Feb 2019 13:56:35 +0000

Changed in openssl-ibmca (Ubuntu Bionic):
status: Fix Committed → Fix Released
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released

------- Comment From <email address hidden> 2019-03-15 06:20 EDT-------
IBM Bugzilla status -> closed. Fix Released for all requested distros

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers