[FFe] [UBUNTU 22.04] ibmca engine with libica = libica.so.4 - sshd dumps core (openssl-ibmca)

[bugproxy]

Dear Ubuntu Release Team, please consider the acceptance this feature freeze exception about a new openssl-ibmca-2.2.3-0ubuntu1 package for jammy, because:

Rationale for the exception:
----------------------------
- a severe issue is fixed (sshd core dump while using hw crypto)
 e91e179 ("PKEY: Fix usage of ECX keys")

- another potential segmentation fault is fixed by:
  6563dd2 ("use correct libica for ibmca_mechaList_test")

- and since these are the only two fixes between 2.2.2 and 2.2.3:
  93a12d3 (tag: v2.2.3) Update to version 2.2.3
  6563dd2 use correct libica for ibmca_mechaList_test
  e91e179 PKEY: Fix usage of ECX keys
  fae4490 (tag: v2.2.2) Update to version 2.2.2
  the version 2.2.3 is a bug-fix only release,
  and could be acceptable for a FFe
  (according to https://wiki.ubuntu.com/FreezeExceptionProcess)

- but to get the new version build (esp. with e91e179) a backport of
  e59cce5 ("Fix compilation for OpenSSL 3.0")
  was needed on top.

- To me it wouldn't make sense to add the 3 commits above to v2.2.2,
  since it would end up in a package version that's a super-set
  of upstream 2.2.3 anyway )but named 2.2.2-0ubuntu1,
  hence asking for the FFe.

- The package now also ships a sample config
  (as well as the script to generate it,
   in case one wants/needs to re-generate it).

upstream ChangeLog diff:
------------------------
$ diff -u openssl-ibmca-2.2.2/ChangeLog openssl-ibmca-2.2.3/ChangeLog
--- openssl-ibmca-2.2.2/ChangeLog 2022-01-27 17:23:55.000000000 +0100
+++ openssl-ibmca-2.2.3/ChangeLog 2022-03-31 10:32:20.935374435 +0200
@@ -1,3 +1,6 @@
+* openssl-ibmca 2.2.3
+- Fix PKEY segfault with OpenSSL 3.0
+
 * openssl-ibmca 2.2.2
 - Fix tests with OpenSSL 3.0
 - Build against libica 4.0

News:
-----
There is no upstream NEWS file (or suchlike - the README.md is unchanged).

build log:
----------
https://launchpadlibrarian.net/594086046/buildlog_ubuntu-jammy-s390x.openssl-ibmca_2.2.3-0ubuntu1_BUILDING.txt.gz

Install log:
------------
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1967141/+attachment/5575230/+files/install-log_and_virification.txt

Package upgrade log:
--------------------
https://bugs.launchpad.net/ubuntu-z-systems/+bug/1967141/+attachment/5575247/+files/package_upgrade_test.txt

Testing:
--------
The previous link also includes the testing and verification that I did,
hence I can confirm that the reported problem is solved.
On top a testsuite is executed when the package is build.
Local build:
========================================================================
Testsuite summary for openssl-ibmca 2.2.3
========================================================================
# TOTAL: 34
# PASS: 28
# SKIP: 6
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
========================================================================
PPA:
========================================================================
Testsuite summary for openssl-ibmca 2.2.3
========================================================================
# TOTAL: 34
# PASS: 26
# SKIP: 8
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
========================================================================
(Two more tests are (auto-)skipped when running a PPA build, because the builder does not have access to the s390x crypto hardware.
(The other skipped tests are skipped by upstream, since they are known to cause issues on openssl 3 systems).

description of proposed changes:
--------------------------------

"PKEY: Fix usage of ECX keys" - the usage of ECX keys was fixed by using proper missing set/get methods for opaque types.
New file src/openssl-compat.h introduced that holds the specific ossl_ecx* function.
All this only effects ED25519, ED448, X25519 and X448 using s390x hardware crypto.

"use correct libica for ibmca_mechaList_test" - the Makefile for ibmca_mechaList_test is now generated during the configure run, to make sure it links with the same libica variant as used by the ibmca.so module.

"Fix compilation for OpenSSL 3.0" - the API used in the above commit is not public (and introduces a line "include <crypto/evp.h>" that cannot be resolved all the time - only compile if the OpenSSL 3.0 source tree is present). Hence the defines to be used are now copied over, which makes the engine compile even without OpenSSL 3.0 source tree.

In addition a sample config that allows to be simply copied over is added to the package, as well as the script that generates it. That is done for convenience reasons and reduces the configuration time to just seconds.

The openssl-ibmca package is a universe package that is available for s390x only.

Dependencies:
-------------
openssl-ibmca has no reverse dependencies:
$ apt-cache depends openssl-ibmca
openssl-ibmca
  Depends: libica4
  Depends: libc6
  Depends: libssl3
$ apt-cache rdepends openssl-ibmca
openssl-ibmca
Reverse Depends:

_________________________

---Problem Description---
Summary
=======

New IBM HW with Crypto Accelerator cards attached
Kernel level: 5.14
Core dump when configuring the ibmca engine with libica = libica.so.4 in the openssl.cnf file in the engine section.
The problem only occurs with OpenSSL 3.0 and is immediately reproducible.

Details
=======
HINT: To be able to receive core dump files at all it is needed to change the /etc/systemd/system.conf file entry DefaultLimitCORE=0:infinity to read
      DefaultLimitCORE=infinity:infinity

On a system with ibmca engine configured system wide, when trying to use the libica.so.4 to support the ibmca engine the sshd daemon dumps core upon the first login attempt.

# openssl engine -c
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
 [RSA, DSA, DH]

Debug Data
==========
core dump file in the attachments.

Contact Information = <email address hidden>

---uname output---
Linux system 5.14.

---Debugger---
A debugger is not configured

---Steps to Reproduce---
1.) Edit /etc/systemd/system.conf file to allow core dumps:
    Change the line DefaultLimitCORE=0:infinity
    to read DefaultLimitCORE=infinity:infinity
2.) run: systemctl daemon-reload
         systemctl restart systemd-coredump.socket
3.) Run the /usr/share/doc/openssl-ibmca/ibmca-engine-opensslconfig perl script
4.) Edit the /etc/pki/tls file near the end to contain the line
    to back the ibmca engine by the libica.so.4 library as outlined in the
    /usr/share/doc/openssl-ibmca/README.md file
5.) Run: openssl engine -c
6.) Keep the current session open for subsequently stepping back to the
    original openssl.cnf!
7.) Open up a new ssh session to the system under test
    and watch the login to fail with broken pipe
8.) On the remaining session, run
    coreumpctl list / coredumpctl dump

Userspace tool common name: openssl-ibmca

Userspace rpm: openssl-ibmca-2.2.2-1.el9.s390x

The userspace tool has the following bit modes: 64bit

Userspace tool obtained from project website: na

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2022-03-30 10:26 EDT-------
This problem has been fixed with with openssl-ibmca 2.2.3:

* openssl-ibmca 2.2.3
- Fix PKEY segfault with OpenSSL 3.0

This 2.2.3 release has been made specifically to fix this bug, so you can choose between the following commit
https://github.com/opencryptoki/openssl-ibmca/commit/93a12d3f3d401247c13ea3f4f47dc3d10fbb6f7b

or a package upgrade to 2.2.3 to fix this.

Thanks.

[Frank Heimes (fheimes)]

I've noticed that v2.2.3 is obviously a bugfix-only release:
...
93a12d3 (tag: v2.2.3) Update to version 2.2.3
6563dd2 use correct libica for ibmca_mechaList_test
e91e179 PKEY: Fix usage of ECX keys
fae4490 (tag: v2.2.2) Update to version 2.2.2
...
and according to the FFe definition, bugfix-only updates might be acceptable,
so I will follow that route.

I've created such a package:
openssl-ibmca (2.2.3-0ubuntu1) jammy; urgency=medium
  * New upstream release. LP: #1967141
  * The difference between 2.2.2 and 2.2.3 includes just these two fixes:
    - "PKEY: Fix usage of ECX keys"
    - "use correct libica for ibmca_mechaList_test"
    Rather than adding these as quilt patches, raising the package to the
    bugfix-only version that incl. them is preferable.
  * For "PKEY: Fix usage of ECX keys" a backport of
    "Fix compilation for OpenSSL 3.0" was needed:
    d/p/e59cce5-Fix-compilation-for-OpenSSL-3.0.patch
  * For convenience reasons a generated sample config is now included in
    the package, but also the optional configuration generator Perl script
    'ibmca-engine-opensslconfig'.

It built fine locally (on s390x - it's an s390x-only package).

In addition I've did a PPA build of this package which is available here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1967141

The build includes a run of the test suite:

local build:
============================================================================
Testsuite summary for openssl-ibmca 2.2.3
============================================================================
# TOTAL: 34
# PASS: 28
# SKIP: 6
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================

PPA:
============================================================================
Testsuite summary for openssl-ibmca 2.2.3
============================================================================
# TOTAL: 34
# PASS: 26
# SKIP: 8
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
(Two more tests are (auto-)skipped, because the builder does not have access to the s390x crypto hardware.
(The other skipped tests are skipped by upstream, since they are known to cause issues on openssl 3 systems).

On top I installed, tested ad verified the package on an s390x system:
$ uname -a
Linux s1lp15 5.15.0-23-generic #23-Ubuntu SMP Fri Mar 11 14:53:58 UTC 2022 s390x s390x s390x GNU/Linux
$ apt-cache policy openssl-ibmca
openssl-ibmca:
  Installed: 2.2.3-0ubuntu1
  Candidate: 2.2.3-0ubuntu1
  Version table:
 *** 2.2.3-0ubuntu1 100
        100 /var/lib/dpkg/status
     2.2.2-0ubuntu1 500
        500 http://ports.ubuntu.com/ubuntu-ports jammy/universe s390x Packages

And I can confirm that sshd does NOT core dump.

[Frank Heimes (fheimes)]

diff between
openssl-ibmca_2.2.2-0ubuntu1
and
openssl-ibmca_2.2.3-0ubuntu1
debdiff_openssl-ibmca_from_2.2.2-0ubuntu1_to_2.2.3-0ubuntu1.diff

[Frank Heimes (fheimes)]

Install-log and functional verification of the new package.

[Frank Heimes (fheimes)]

package upgrade test

[Łukasz Zemczak (sil2100)]

Thank you for filling in this detailed FFe - I'm actually always super happy when seeing that. As you already noticed yourself, the new upstream version looks like a bugfix-only release, so a FFe shouldn't be even required. Anyway, this is approved and please proceed with an upload o/

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "debdiff_openssl-ibmca_from_2.2.2-0ubuntu1_to_2.2.3-0ubuntu1.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Simon Chopin (schopin)]

Uploaded! I'll pile on with my own thanks as sponsor, context always makes review much easier :)

[Frank Heimes (fheimes)]

Many thx sil2100 and schopin - much appreciated!

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl-ibmca - 2.2.3-0ubuntu1

---------------
openssl-ibmca (2.2.3-0ubuntu1) jammy; urgency=medium

  * New upstream release. LP: #1967141
  * The difference between 2.2.2 and 2.2.3 includes just these two fixes:
    - "PKEY: Fix usage of ECX keys"
    - "use correct libica for ibmca_mechaList_test"
    Rather than adding these as quilt patches, raising the package to the
    bugfix-only version that incl. them is preferable.
  * For "PKEY: Fix usage of ECX keys" a backport of
    "Fix compilation for OpenSSL 3.0" was needed:
    d/p/e59cce5-Fix-compilation-for-OpenSSL-3.0.patch
  * For convenience reasons a generated sample config is now included in
    the package, but also the optional configuration generator Perl script
    'ibmca-engine-opensslconfig'.
  * d/control: add dh-autoreconf to Build-Depends to work around a Lintian
    regression on missing-build-dependency-for-dh-addon

 -- Frank Heimes <email address hidden> Wed, 30 Mar 2022 19:19:00 +0100