Openssl libcrypto performance issue

[bugproxy]

== Comment: #0 - Bastian Pfeifer <email address hidden> - 2016-04-22 03:37:03 ==
---Problem Description---
Performance problem with s390x assembly code in the openssl libcrypto library:
CPACF functions such as SHA, AES ... queries the CPACF facility bits too often.

Problematic code can be found here:
https://github.com/openssl/openssl/blob/master/crypto/sha/asm/sha1-s390x.pl

What happens is that for every e.g SHA1 call the code first tests if the HW function is available. That's the case for all the CPACF functions.

However what the lib should do is to query only once, safe the value and then use the function. The problem is that the Hipervisor in certain scenarios is required to intercept the query instructions, which makes this really expensive.

Contact Information = <email address hidden>

---uname output---
4.4.0-18-generic #34-Ubuntu SMP

Machine Type = 2964, 701 NC9

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 n/a

Userspace tool common name: OpenSSL

The userspace tool has the following bit modes: 64-bit

Userspace rpm: OpenSSL 1.0.2g 1 Mar 2016

Userspace tool obtained from project website: na

*Additional Instructions for <email address hidden>:
-Attach ltrace and strace of userspace application.

== Comment: #8 - Bastian Pfeifer <email address hidden> - 2016-06-02 07:05:00 ==
We performed tests on the new SHA,AES and GHASH code and report performance improvements especially for SHA (up to 20%).

Here are the links to the new s390x assembly code which should be used to create patches for the UBUNTU specific openssl versions.

1)
https://github.com/openssl/openssl/blob/master/crypto/s390xcpuid.S
2)
https://github.com/openssl/openssl/blob/master/crypto/s390xcap.c
3)
https://github.com/openssl/openssl/blob/master/crypto/sha/asm/sha1-s390x.pl
4)
https://github.com/openssl/openssl/blob/master/crypto/sha/asm/sha512-s390x.pl
5)
https://github.com/openssl/openssl/blob/master/crypto/aes/asm/aes-s390x.pl
6)
https://github.com/openssl/openssl/blob/master/crypto/modes/asm/ghash-s390x.pl

In case of AES I was forced to change the following code in aes-s390x.pl

.globl AES_set_decrypt_key
.type AES_set_decrypt_key,\@function

goes to

.globl private_AES_set_decrypt_key
.type private_AES_set_decrypt_key,\@function

This was done for 'AES_set_encrypt_key as well; to be consistent with the openssl code which comes with UBUNTU. For my performance tests this worked properly and I checked the CPACF counter with the tool 'cpacfstats'.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Dimitri John Ledkov (xnox)]

Linking to individual files is not very practical to understand introduced changes. Could you please link to upstream commits that implement desired functionality?

[Frank Heimes (fheimes)]

Some add. IBM info:

---Problem Description---
The openssl s390x assembly code is not available, causing a fallback to openssl internal C-code. The performance degradation is up to a factor 4 for asymmetric (RSA, ...) and a factor >10 for symmetric cipher like SHA & AES.

Contact Information = <email address hidden>

---uname output---
4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:14:29 UTC 2016 s390x s390x s390x GNU/Linux

Machine Type = 2964, 701

---Debugger---
A debugger is not configured

---Steps to Reproduce---
#Run a benchmark
openssl speed sha1

#Check if CPACF was used
cpacfstatsd
cpacfstats

Userspace tool common name: OpenSSL 1.0.2g-fips 1 Mar 2016

The userspace tool has the following bit modes: 64-bit

Userspace rpm: OpenSSL 1.0.2g-fips 1 Mar 2016

Userspace tool obtained from project website: na

*Additional Instructions for <email address hidden>:
-Attach ltrace and strace of userspace application.

To me it seems the following compiler flags

-DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DAES_CTR_ASM -DAES_XTS_ASM -DGHASH_ASM

are completely missing. Probably openssl was configured with the 'no-asm' option. This option, for example, is applied for s390 as defined in the openssl Configure file.

"debian-s390","gcc:-DB_ENDIAN ${debian_cflags}::-D_REENTRANT::-ldl:RC4_CHAR RC4_CHUNK DES_INT DES_UNROLL:${no_asm}:dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR)",

"debian-s390x","gcc:-DB_ENDIAN ${debian_cflags}::-D_REENTRANT::-ldl:SIXTY_FOUR_BIT_LONG RC4_CHAR RC4_CHUNK DES_INT DES_UNROLL:${no_asm}:dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR)",
-------
Its the same code as in bug IBM140645(LP1601836), right, where I report performance improvements using modified s390x code already upstream on openssl github. Tested by a locally installed openssl source package.

The issue reported here says that the s390x CPACF assembly code is not configured into the openssl build of Ubuntu at all and thus is not available in openssl client/server workloads; a fallback to very slow C-code is the result.

Probably, just some compiler flags are missing ...

[Frank Heimes (fheimes)]

This ticket is related to https://bugs.launchpad.net/ubuntu-z-systems/+bug/1602655

[Dimitri John Ledkov (xnox)]

After searching more, this has been backported upstream to the 1.0.2 stable series. Thus at most we could pick up and test https://github.com/openssl/openssl/commit/0b48a24ce993d1a4409d7bde26295f6df0d173cb

If I understand the original request correctly. Please confirm.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-08-02 06:02 EDT-------
Yes, that is the s390x assembly package with the fixes included.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 1.0.2g-1ubuntu7

---------------
openssl (1.0.2g-1ubuntu7) yakkety; urgency=medium

  * Cherry-pick s390x assembly pack bugfix to cache capability query
    results for improved performance. LP: #1601836.

 -- Dimitri John Ledkov <email address hidden> Mon, 01 Aug 2016 16:58:01 +0100

[Brian Murray (brian-murray)]

Hello bugproxy, or anyone else affected,

Accepted openssl into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-08-12 08:05 EDT-------
I tested for:

1) Ubuntu Yakkety Yak (development branch) (GNU/Linux 4.4.0-33-generic s390x)
Version: 1.0.2g-1ubuntu7

2)Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-18-generic s390x)
Version: 1.0.2g-1ubuntu4.2

The BUG is fixed in both cases.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 1.0.2g-1ubuntu4.2

---------------
openssl (1.0.2g-1ubuntu4.2) xenial; urgency=medium

  * Cherry-pick s390x assembly pack bugfix to cache capability query
    results for improved performance. LP: #1601836.
  * Enable asm optimisations on s390x. LP: #1602655.

 -- Dimitri John Ledkov <email address hidden> Thu, 28 Jul 2016 15:37:07 +0300

[Martin Pitt (pitti)]

The verification of the Stable Release Update for openssl has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.