[UBUNTU 22.04] openssl with ibmca engine configured dumps core when creating a new certificate

[bugproxy]

=== SRU information ===

[Meta]
This bug is part of a series of three bugs for a single SRU.
The "central" bug with the global information and debdiff is http://pad.lv/2033422

[Impact]
Openssl using an engine dumps core upon certificate creation; other operations are probably affected too. Overall, engines are likely mostly unusable.

[Test plan]
- An openssl engine is req. to test the fix.
- A z13 / LinuxONE LPAR or z/VM guest is needed, with attached APQN.
- Check with 'lszcrypt -V' the availability (online) of the hw crypto resources.
- Install the needed package that allows to exploit the hw crypto resources:
  sudo apt-get install libica-utils libica? openssl-ibmca
- And copy a working sample openssf.cnf file:
  sudo cp /usr/share/doc/openssl-ibmca/examples/openssl.cnf.sample /etc/ssl/openssl.cnf
- Verify if the 'openssl engine' lists an ibmca engine,
  in addition to the dynamic engine:
  openssl engine
    (dynamic) Dynamic engine loading support
    (ibmca) Ibmca hardware engine support <===
- try to create a new certificate, using this cmd-line:
  openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
- The above command must not lead to a 'Segmentation fault (core dumped)',
  rather than create a proper certificate file.
  Also watch /var/log/syslog / journalctl for more details.
- Upgrade not only the openssl package itself,
  but also libssl3, before verification.
- The issue is fixed in openssl 3.0.8 which landed in lunar.

[Where problems could occur]
I don't pretend to understand the lifecycle of providers in openssl3 but the patch is simple and has been widely tested by now, including on ubuntu. Thus, I see little chance an unexpected problem would occur with it.

[Patches]
The patches come directly from upstream and apply cleanly.

https://github.com/openssl/openssl/issues/18578

* https://git.launchpad.net/~adrien-n/ubuntu/+source/openssl/tree/debian/patches/jammy-sru-0001-Release-the-drbg-in-the-global-default-context-befor.patch?h=jammy-sru&id=04ef023920ab08fba214817523fba897527dfff0

=== Original description ===

openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'

---Problem Description---
OpenSSL with ibmca engine configured dumps core when creating a new certificate.

# openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
# openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
Segmentation fault (core dumped)

# journalctl
Jun 07 13:06:08 SYSTEM kernel: User process fault: interruption code 003b ilc:2 in libc.so.6[3ffae080000+1ca000]
Jun 07 13:06:08 SYSTEM kernel: Failing address: 0000000000000000 TEID: 0000000000000800
Jun 07 13:06:08 SYSTEM kernel: Fault in primary space mode while using user ASCE.
Jun 07 13:06:08 SYSTEM kernel: AS:000000009c2941c7 R3:0000000000000024
Jun 07 13:06:08 SYSTEM kernel: CPU: 2 PID: 2344 Comm: openssl Kdump: loaded Not tainted 5.15.0-73-generic #80-Ubuntu
Jun 07 13:06:08 SYSTEM kernel: Hardware name: IBM 3931 A01 703 (z/VM 7.3.0)
Jun 07 13:06:08 SYSTEM kernel: User PSW : 0705000180000000 000003ffae11c708
Jun 07 13:06:08 SYSTEM kernel: R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:1 AS:0 CC:0 PM:0 RI:0 EA:3
Jun 07 13:06:08 SYSTEM kernel: User GPRS: 0000000000000007 000003ffae11c6f0 0000000000000000 000002aa3289f9d0
Jun 07 13:06:08 SYSTEM kernel: 000002aa1825980f 000002aa3289f9d0 0000000000000000 000002aa328a4300
Jun 07 13:06:08 SYSTEM kernel: 000003ffae870720 000003ffae657128 000002aa000003ff 0000000000000000
Jun 07 13:06:08 SYSTEM kernel: 000003ffae24dd10 000003ffae657120 000003ffae437c22 000003ffec2fe000
Jun 07 13:06:08 SYSTEM kernel: User Code: 000003ffae11c6fc: b90400b2 lgr %r11,%r2
                                                      000003ffae11c700: 47000000 bc 0,0
                                                     #000003ffae11c704: b24f00a0 ear %r10,%a0
                                                     >000003ffae11c708: 58102018 l %r1,24(%r2)
                                                      000003ffae11c70c: ebaa0020000d sllg %r10,%r10,32
                                                      000003ffae11c712: b24f00a1 ear %r10,%a1
                                                      000003ffae11c716: 5910a0d0 c %r1,208(%r10)
                                                      000003ffae11c71a: a7840033 brc 8,000003ffae11c780
Jun 07 13:06:08 SYSTEM kernel: Last Breaking-Event-Address:
Jun 07 13:06:08 SYSTEM kernel: [<000003ffae33242c>] 0x3ffae33242c
Jun 07 13:06:08 SYSTEM systemd[1]: Started Process Core Dump (PID 2345/UID 0).
Jun 07 13:06:08 SYSTEM systemd-coredump[2350]: Process 2344 (openssl) of user 0 dumped core.

                                                           Found module linux-vdso64.so.1 with build-id: bcfab8ac8dbd44c758c3c5494e2952db16905d2e
                                                           Found module libica.so.4 with build-id: 0cc5ace50644dfba6d0ecf4f783477cd04a55731
                                                           Found module ibmca.so with build-id: 27daaf0ed1857fdad3761c2b3db21020999eee08
                                                           Found module ld64.so.1 with build-id: 31d4856f0ba9ea058c91a34f4d684ae0fe01964c
                                                           Found module libc.so.6 with build-id: 74250317950da91d3345f258cb2dd12d22c3f2e5
                                                           Found module libcrypto.so.3 with build-id: a27f20e6cf293f214d459530ce2c0b2b52fdbdb4
                                                           Found module libssl.so.3 with build-id: e2c031c3dac06b5ce43bdea022aee7989f78dde4
                                                           Found module openssl with build-id: ed0fe325182e99d135ee6b08e6d90a9d1c42af7f
                                                           Stack trace of thread 2344:
                                                           #0 0x000003ffae11c708 __pthread_rwlock_wrlock_full64 (libc.so.6 + 0x9c708)
                                                           #1 0x000003ffae437c22 CRYPTO_THREAD_write_lock (libcrypto.so.3 + 0x1b7c22)
                                                           #2 0x000003ffae3e3472 ENGINE_finish (libcrypto.so.3 + 0x163472)
                                                           #3 0x000003ffae406844 EVP_CIPHER_CTX_reset (libcrypto.so.3 + 0x186844)
                                                           #4 0x000003ffae40688c EVP_CIPHER_CTX_free (libcrypto.so.3 + 0x18688c)
                                                           #5 0x000003ffae4f903c n/a (libcrypto.so.3 + 0x27903c)
                                                           #6 0x000003ffae40ca98 EVP_RAND_CTX_free (libcrypto.so.3 + 0x18ca98)
                                                           #7 0x000003ffae461a92 n/a (libcrypto.so.3 + 0x1e1a92)
                                                           #8 0x000003ffae430b9c CRYPTO_free_ex_data (libcrypto.so.3 + 0x1b0b9c)
                                                           #9 0x000003ffae4293ca n/a (libcrypto.so.3 + 0x1a93ca)
                                                           #10 0x000003ffae4335e8 OPENSSL_cleanup (libcrypto.so.3 + 0x1b35e8)
                                                           #11 0x000003ffae0cb6cc __run_exit_handlers (libc.so.6 + 0x4b6cc)
                                                           #12 0x000003ffae0cb790 __GI_exit (libc.so.6 + 0x4b790)
                                                           #13 0x000002aa31847c06 main (openssl + 0x47c06)
                                                           #14 0x000003ffae0aa712 __libc_start_call_main (libc.so.6 + 0x2a712)
                                                           #15 0x000003ffae0aa7f0 __libc_start_main_impl (libc.so.6 + 0x2a7f0)
                                                           #16 0x000002aa31848070 n/a (openssl + 0x48070)
Jun 07 13:06:08 SYSTEM systemd[1]: [e-mail]: Deactivated successfully.
Jun 07 13:06:31 SYSTEM kernel: User process fault: interruption code 003b ilc:2 in libc.so.6[3ffbee00000+1ca000]
Jun 07 13:06:31 SYSTEM kernel: Failing address: 0000000000000000 TEID: 0000000000000800
Jun 07 13:06:31 SYSTEM kernel: Fault in primary space mode while using user ASCE.
Jun 07 13:06:31 SYSTEM kernel: AS:000000009c2941c7 R3:0000000000000024
Jun 07 13:06:31 SYSTEM kernel: CPU: 2 PID: 2356 Comm: openssl Kdump: loaded Not tainted 5.15.0-73-generic #80-Ubuntu
Jun 07 13:06:31 SYSTEM kernel: Hardware name: IBM 3931 A01 703 (z/VM 7.3.0)
Jun 07 13:06:31 SYSTEM kernel: User PSW : 0705000180000000 000003ffbee9c708
Jun 07 13:06:31 SYSTEM kernel: R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:1 AS:0 CC:0 PM:0 RI:0 EA:3
Jun 07 13:06:31 SYSTEM kernel: User GPRS: 0000000000000007 000003ffbee9c6f0 0000000000000000 000002aa176569d0
Jun 07 13:06:31 SYSTEM kernel: 000002aa3dc356c6 000002aa176569d0 0000000000000000 000002aa1765b300
Jun 07 13:06:31 SYSTEM kernel: 000003ffbf5f0720 000003ffbf3d7128 000002aa000003ff 0000000000000000
Jun 07 13:06:31 SYSTEM kernel: 000003ffbefcdd10 000003ffbf3d7120 000003ffbf1b7c22 000003ffc4efdd40
Jun 07 13:06:31 SYSTEM kernel: User Code: 000003ffbee9c6fc: b90400b2 lgr %r11,%r2
                                                      000003ffbee9c700: 47000000 bc 0,0
                                                     #000003ffbee9c704: b24f00a0 ear %r10,%a0
                                                     >000003ffbee9c708: 58102018 l %r1,24(%r2)
                                                      000003ffbee9c70c: ebaa0020000d sllg %r10,%r10,32
                                                      000003ffbee9c712: b24f00a1 ear %r10,%a1
                                                      000003ffbee9c716: 5910a0d0 c %r1,208(%r10)
                                                      000003ffbee9c71a: a7840033 brc 8,000003ffbee9c780
Jun 07 13:06:31 SYSTEM kernel: Last Breaking-Event-Address:
Jun 07 13:06:31 SYSTEM kernel: [<000003ffbf0b242c>] 0x3ffbf0b242c
Jun 07 13:06:32 SYSTEM systemd[1]: Started Process Core Dump (PID 2357/UID 0).
Jun 07 13:06:32 SYSTEM systemd-coredump[2362]: Process 2356 (openssl) of user 0 dumped core.

                                                           Found module linux-vdso64.so.1 with build-id: bcfab8ac8dbd44c758c3c5494e2952db16905d2e
                                                           Found module libica.so.4 with build-id: 0cc5ace50644dfba6d0ecf4f783477cd04a55731
                                                           Found module ibmca.so with build-id: 27daaf0ed1857fdad3761c2b3db21020999eee08
                                                           Found module ld64.so.1 with build-id: 31d4856f0ba9ea058c91a34f4d684ae0fe01964c
                                                           Found module libc.so.6 with build-id: 74250317950da91d3345f258cb2dd12d22c3f2e5
                                                           Found module libcrypto.so.3 with build-id: a27f20e6cf293f214d459530ce2c0b2b52fdbdb4
                                                           Found module libssl.so.3 with build-id: e2c031c3dac06b5ce43bdea022aee7989f78dde4
                                                           Found module openssl with build-id: ed0fe325182e99d135ee6b08e6d90a9d1c42af7f
                                                           Stack trace of thread 2356:
                                                           #0 0x000003ffbee9c708 __pthread_rwlock_wrlock_full64 (libc.so.6 + 0x9c708)
                                                           #1 0x000003ffbf1b7c22 CRYPTO_THREAD_write_lock (libcrypto.so.3 + 0x1b7c22)
                                                           #2 0x000003ffbf163472 ENGINE_finish (libcrypto.so.3 + 0x163472)
                                                           #3 0x000003ffbf186844 EVP_CIPHER_CTX_reset (libcrypto.so.3 + 0x186844)
                                                           #4 0x000003ffbf18688c EVP_CIPHER_CTX_free (libcrypto.so.3 + 0x18688c)
                                                           #5 0x000003ffbf27903c n/a (libcrypto.so.3 + 0x27903c)
                                                           #6 0x000003ffbf18ca98 EVP_RAND_CTX_free (libcrypto.so.3 + 0x18ca98)
                                                           #7 0x000003ffbf1e1a92 n/a (libcrypto.so.3 + 0x1e1a92)
                                                           #8 0x000003ffbf1b0b9c CRYPTO_free_ex_data (libcrypto.so.3 + 0x1b0b9c)
                                                           #9 0x000003ffbf1a93ca n/a (libcrypto.so.3 + 0x1a93ca)
                                                           #10 0x000003ffbf1b35e8 OPENSSL_cleanup (libcrypto.so.3 + 0x1b35e8)
                                                           #11 0x000003ffbee4b6cc __run_exit_handlers (libc.so.6 + 0x4b6cc)
                                                           #12 0x000003ffbee4b790 __GI_exit (libc.so.6 + 0x4b790)
                                                           #13 0x000002aa161c7c06 main (openssl + 0x47c06)
                                                           #14 0x000003ffbee2a712 __libc_start_call_main (libc.so.6 + 0x2a712)
                                                           #15 0x000003ffbee2a7f0 __libc_start_main_impl (libc.so.6 + 0x2a7f0)
                                                           #16 0x000002aa161c8070 n/a (openssl + 0x48070)

Contact Information = <email address hidden> <email address hidden>

---uname output---
Linux SYSTEM 5.15.0-73-generic #80-Ubuntu SMP Mon May 15 15:23:03 UTC 2023 s390x s390x s390x GNU/Linux

Machine Type = Manufacturer: IBM Type: 3931 Model: 703 A01

---Steps to Reproduce---
1. Configure openssl to be used with ibmca engine.
2. run the following command:
    # openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
3. Check the syslog

Userspace tool common name: openssl

The userspace tool has the following bit modes: 64

Userspace rpm: openssl 3.0.2-0ubuntu1.10 s390x

Userspace tool obtained from project website: na

== Comment: #1 - Grgo ===============================================
Further investigations point to this open issue on openssl repository https://github.com/openssl/openssl/issues/18578

The default_algorithms declaration of RAND and others has no effect on the behaviour of this problem.

== Comment: #2 - Ingo - 2023-06-12 06:05:20 =========================
The OpenSSL fix for the mentioned issue https://github.com/openssl/openssl/issues/18578 is commit
https://github.com/openssl/openssl/commit/a88e97fcace01ecf557b207f04328a72df5110df in the master branch.
The corresponding commit for the openssl-3.0 branch is
https://github.com/openssl/openssl/commit/d0f8056c47f7aea40a34815fe459404f14501e81
This commit is included in OpenSSL 3.0.8.

Please include this commit into the OpenSSL package shipped with 22.04 (and later releases).

[Adrien Nader (adrien)]

I've created a PPA for Jammy that incorporates the fix mentionned. The details are available at https://launchpad.net/~adrien-n/+archive/ubuntu/openssl-jammy-sru . Could you test it and confirm your issue is solved?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-09-14 03:15 EDT-------
First, reproduced the behaviour of the problem as described and encountered the segmentation fault situation as described.
Wrote out the coredump with coredumpctl dump and saw from the call stack the reported problem was reproduced.

The installed libssl3 and openssl versions were:
# dpkg -l libssl3 openssl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================-============-====================================================
ii libssl3:s390x 3.0.2-0ubuntu1.10 s390x Secure Sockets Layer toolkit - shared libraries
ii openssl 3.0.2-0ubuntu1.10 s390x Secure Sockets Layer toolkit - cryptographic utility

Next, added the ppa repository as instructed.

# apt list --upgradable
Listing... Done
libssl-dev/jammy 3.0.2-0ubuntu1.11~ppa2 s390x [upgradable from: 3.0.2-0ubuntu1.10]
libssl3/jammy 3.0.2-0ubuntu1.11~ppa2 s390x [upgradable from: 3.0.2-0ubuntu1.10]
openssl/jammy 3.0.2-0ubuntu1.11~ppa2 s390x [upgradable from: 3.0.2-0ubuntu1.10]

Successfully updated the three required packages to the version provided by the ppa package:
# apt upgrade libssl3 openssl libssl-dev
The following packages will be upgraded:
libssl-dev libssl3 openssl
3 upgraded, 0 newly installed, 0 to remove and 15 not upgraded.
Need to get 5,038 kB of archives.
After this operation, 5,120 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
...

Cross-checked the libssl3 and openssl packages had been successfully upgraded:
# dpkg -l libssl3 openssl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-======================-============-====================================================
ii libssl3:s390x 3.0.2-0ubuntu1.11~ppa2 s390x Secure Sockets Layer toolkit - shared libraries
ii openssl 3.0.2-0ubuntu1.11~ppa2 s390x Secure Sockets Layer toolkit - cryptographic utility

Next checked the ibmca engine is still enabled, and subsequently was able to run the key generation request without encountering the segmentation fault.
Thanks for providing the fix.
Please integrate.

Thanks.

# openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
# openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
# openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
..+..+.+......+...+.....+.+..+...+.........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+..+.+.....+.......+........+...+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.....................+.........+...+..+...+......+..........+.....+......+.+..+......+.+....................+...+...+.............+....

[Adrien Nader (adrien)]

Thanks a lot for taking the time to test and provide feedback. I'll continue with the SRU process; landing will probably take a few weeks.

[Lucas Kanashiro (lucaskanashiro)]

Hi Adrien,

You subscribed ubuntu-sponsors, do you have any debdiff or MP to be reviewed? Sorry, I did not find anything ready to review. Without that I believe the best way is to unsubscribe ubuntu-sponsors until there is something ready to be uploaded.

[Adrien Nader (adrien)]

Hi Lucas,

Sorry, this is part of an SRU with 4 patches but that we've decided to hold back for a bit (a few days after the current release). I've removed ubuntu-sponsors from the "main" LP bug (link near the top of the bug report) but not from the others. I'll do it now and I think maybe it's better to only add ~ubuntu-sponsors to that main ticket.

[Simon Chopin (schopin)]

A version containing a fix for this has been uploaded to the Jammy queue to be processed by the SRU team. Thanks, Adrien :)

[Andreas Hasenack (ahasenack)]

Adrien, we do have openssl-ibmca in the archive, for the s390x arch:
$ rmadison openssl-ibmca
 openssl-ibmca | 1.3.0-0ubuntu2 | xenial/universe | source, s390x
 openssl-ibmca | 1.3.0-0ubuntu2.16.04.2 | xenial-updates/universe | source, s390x
 openssl-ibmca | 1.4.1-0ubuntu1 | bionic/universe | source, s390x
 openssl-ibmca | 1.4.1-0ubuntu1.2 | bionic-security/universe | source, s390x
 openssl-ibmca | 1.4.1-0ubuntu1.2 | bionic-updates/universe | source, s390x
 openssl-ibmca | 2.1.0-0ubuntu1 | focal/universe | source, s390x
 openssl-ibmca | 2.1.0-0ubuntu1.20.04.2 | focal-security/universe | source, s390x
 openssl-ibmca | 2.1.0-0ubuntu1.20.04.2 | focal-updates/universe | source, s390x
 openssl-ibmca | 2.2.3-0ubuntu1 | jammy/universe | source, s390x
 openssl-ibmca | 2.2.3-0ubuntu1.1 | jammy-security/universe | source, s390x
 openssl-ibmca | 2.2.3-0ubuntu1.1 | jammy-updates/universe | source, s390x
 openssl-ibmca | 2.4.0-0ubuntu1 | lunar/universe | source, s390x
 openssl-ibmca | 2.4.0-0ubuntu2 | mantic/universe | source, s390x
 openssl-ibmca | 2.4.0-0ubuntu2 | noble/universe | source, s390x

It should be possible to come up with a test case that mimics what was reported. But if not, we can rely on their testing of the proposed packages, since this particular test case requires specialized hardware.

One caveat, though. I fear we might find a bug in our openssl-ibmca packaging. The package in mantic, for example, installs the engine in an openssl 1.1 directory, and there is no openssl 1.1 in mantic:

ubuntu@s1lp5:~$ dpkg -L openssl-ibmca|grep engine
/usr/bin/ibmca-engine-opensslconfig
/usr/lib/s390x-linux-gnu/engines-1.1
/usr/lib/s390x-linux-gnu/engines-1.1/ibmca.so

I suppose IBM built the engine from source for their case? But if they used openssl-ibmca from jammy, then it should be easier for us to test.

As to possible regressions in other engines, we do have some in the archive, although for openssl-3 they are deprecated, and providers are favored. But I have a script that tests openvpn with an openssl engine that we could use here, I used it in another SRU[1]:
"""
[Jammy specific test plan]
Given that in jammy we had to re-enable openssl3 engine support, here is a test for that to make sure it keeps working.

In a jammy *VM*:
- Download test script:
wget https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/2004676/+attachment/5714185/+files/vpn-setup-with-pkcs11

- Install test dependencies
sudo apt install openvpn easy-rsa openssl opensc-pkcs11 gnutls-bin softhsm2 dpkg-dev libengine-pkcs11-openssl expect

- Run test:
sudo ./vpn-setup-with-pkcs11

It runs with set -e, so expected result is "## All good, stopping client and server services" and "## Done." This result is expected both with the current jammy package, and the proposed update.
"""

In that script, the bit that exercises the engine is this, right after having configured openssl.cnf to use the engine:
OPENS...

[Adrien Nader (adrien)]

As you mention, it's difficult to test with this reproducer specifically since it's specialized hardware and I've largely had to rely on testing from the proxied persons who also have interests and duties in this working well. The issue also appears without the specific hardware when using providers for some functions but openssl 3.0 providers are recent and not very widely used so there aren't many one that fit either and the verification/setup is correspondingly high. On the bright side, the potential for damage is low due to the small userbase. One last thing: I don't know if this could work less well than right now since they get crashes.

The "engines-1.1" is not necessarily a concern: libengine-gost-openssl/libengine-gost-openssl1.1 was/were putting files in a similar place without issue IIRC. I don't have a good example to show because the package currently for jammy puts it in / directly... . In any case, the path should be configured and the actual location isn't really an issue. You can see the configuration on https://sysos.ru/archives/589 (russian, search for "openssl_def"; I have seen non-russian links too but I can't find them again).

The package for the SRU is already in -proposed so it should be possible to test already. It's (very) late here though so I'll come back to this and the tests tomorrow. Thanks for the review, comments, and testcase.

[Frank Heimes (fheimes)]

Excuse me for chiming in so late, but we can test (and even recreate) the situation by ourselves on our system (and we have systems with attached crypto hw to it).

I just tried it on a jammy z/VM guest:

$ lszcrypt -V
CARD.DOM TYPE MODE STATUS REQUESTS PENDING HWTYPE QDEPTH FUNCTIONS DRIVER
--------------------------------------------------------------------------------------------
02 CEX5C CCA-Coproc online 1 0 11 08 S--D--N-- cex4card
02.0011 CEX5C CCA-Coproc online 1 0 11 08 S--D--N-- cex4queue
$ sudo apt-get install libica-utils libica? openssl-ibmca
$ sudo cp /usr/share/doc/openssl-ibmca/examples/openssl.cnf.sample /etc/ssl/openssl.cnf
$ openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
$ openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
..+....+........+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+........+...+.+........+.+......+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......................+......+......+...+...+........+.........................+......+........+...+...+..........+..+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+..+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..+...+....+...+...+........+.+.....+...+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+.....................+..+....+...+..+......+...+......+.............+..+.+.....+...................+........+.+.....+......+.........+.......+..+.+.....+....+...+.....+....+......+.....+.......+...............+......+.........+...+........+...............+............+....+.....+.+..+..........+...............+......+.........+.....+.+.....+.+..............+..........+............+...+......+...........+...+......+....+............+..+.+......+.....+...+.............+.................+.+.....+.........+.+.....+.+.....+.+........+....+..+......+..........+..+.+......+...+..+......+....+.........+..+..................+......+...+.+.................+...............+.......+.....+.+......+...+......+......+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
Segmentation fault (core dumped)
$

[Steve Langasek (vorlon)]

An upload of openssl to jammy-proposed has been rejected from the upload queue for the following reason: "Please reupload without debian/patches/lp1994165/0002-Handle-SMIME_crlf_copy-return-code.patch".

[Steve Langasek (vorlon)]

> Excuse me for chiming in so late, but we can test (and even recreate)
> the situation by ourselves on our system (and we have systems with
> attached crypto hw to it).

Someone please update the bug description accordingly with a complete description of the SRU test case.

[Frank Heimes (fheimes)]

I had the update of the bug description on my todo list - it's done now.

[Steve Langasek (vorlon)]

Hello bugproxy, or anyone else affected,

Accepted openssl into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/3.0.2-0ubuntu1.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openssl (3.0.2-0ubuntu1.13) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

diffoscope/205 (armhf)
dotnet6/6.0.126-0ubuntu1~22.04.1 (amd64, arm64)
ganeti/3.0.2-1ubuntu1 (armhf)
linux-aws-5.19/5.19.0-1029.30~22.04.1 (arm64)
linux-aws-6.5/6.5.0-1011.11~22.04.1 (arm64)
linux-azure-6.2/6.2.0-1018.18~22.04.1 (arm64)
linux-gcp-6.2/6.2.0-1019.21~22.04.1 (arm64)
linux-gke/5.15.0-1048.53 (arm64)
linux-lowlatency-hwe-6.2/6.2.0-1018.18~22.04.1 (arm64)
linux-lowlatency-hwe-6.5/6.5.0-14.14.1~22.04.1 (arm64)
linux-mtk/5.15.0-1029.33 (arm64)
linux-nvidia-6.5/6.5.0-1010.10 (arm64)
linux-oracle-6.2/6.2.0-1017.18~22.04.1 (arm64)
puma/5.5.2-2ubuntu2 (arm64)
python-bonsai/1.3.0+ds-3build1 (armhf)
ruby3.0/3.0.2-7ubuntu2.4 (amd64, arm64, i386)
seqkit/2.1.0+ds-1 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#openssl

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Frank Heimes (fheimes)]

verification on jammy was successful

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-01-23 03:44 EDT-------
The problem is still visible after installing the -proposed package:

root:~# dpkg -l | grep openssl
ii openssl 3.0.2-0ubuntu1.13 s390x Secure Sockets Layer toolkit - cryptographic utility
ii openssl-ibmca 2.2.3-0ubuntu1.1 s390x libica based hardware acceleration engine for OpenSSL
ii python3-openssl 21.0.0-1 all Python 3 wrapper around the OpenSSL library
root:~# openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support

By running with gdb the following BT can be observed:
(gdb) r
Starting program: /usr/bin/openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj /CN=US
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".
.............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.+.....+..........+............+...+...........+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+.....+.+..........................+...+.+......+.....+...+.........+.+.........+......+..+.......+....................+.+...............+..............+...+...+......+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+.....+............+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----

Program received signal SIGSEGV, Segmentation fault.
0x000003fffd89c738 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0x0) at pthread_rwlock_common.c:603
603 pthread_rwlock_common.c: No such file or directory.
(gdb) bt
#0 0x000003fffd89c738 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0x0) at pthread_rwlock_common.c:603
#1 ___pthread_rwlock_wrlock (rwlock=0x0) at pthread_rwlock_wrlock.c:26
#2 0x000003fffdbb7812 in CRYPTO_THREAD_write_lock () from /lib/s390x-linux-gnu/libcrypto.so.3
#3 0x000003fffdb634f2 in ENGINE_finish () from /lib/s390x-linux-gnu/libcrypto.so.3
#4 0x000003fffdb86314 in EVP_CIPHER_CTX_reset () from /lib/s390x-linux-gnu/libcrypto.so.3
#5 0x000003fffdb8635c in EVP_CIPHER_CTX_free () from /lib/s390x-linux-gnu/libcrypto.so.3
#6 0x000003fffdc791ac in ?? () from /lib/s390x-linux-gnu/libcrypto.so.3
#7 0x000003fffdb8ca88 in EVP_RAND_CTX_free () from /lib/s390x-linux-gnu/libcrypto.so.3
#8 0x000003fffdbe4bc2 in ?? () from /lib/s390x-linux-gnu/libcrypto.so.3
#9 0x000003fffdbb0c6c in CRYPTO_free_ex_data () from /lib/s390x-linux-gnu/libcrypto.so.3
#10 0x000003fffdba8dfa in ?? () from /lib/s390x-linux-gnu/libcrypto.so.3
#11 0x000003fffdbb36b8 in OPENSSL_cleanup () from /lib/s390x-linux-gnu/libcrypto.so.3
#12 0x000003fffd84b6ec in __run_exit_handlers (status=<optimized out>, listp=0x3fffd9ce618 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true,
run_dtors=run_dtors@entry=...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-01-23 03:53 EDT-------
After upgrading to Ubuntu 22.04.3 the solution is working and the problem is not visible anymore.

Explicit update of the fix is not valid on the older version (22.04.1).

[Frank Heimes (fheimes)]

Hi Grgo, did you also updated libssl3?
see test plan in Description:
"
- Upgrade not only the openssl package itself,
  but also libssl3, before verification.
"
With that I could successfully verify.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-01-23 05:09 EDT-------
# apt-cache policy libssl3 openssl
libssl3:
Installed: 3.0.2-0ubuntu1.13
Candidate: 3.0.2-0ubuntu1.13
Version table:
*** 3.0.2-0ubuntu1.13 400
400 http://ports.ubuntu.com/ubuntu-ports jammy-proposed/main s390x Packages
100 /var/lib/dpkg/status
3.0.2-0ubuntu1.12 500
500 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main s390x Packages
500 http://ports.ubuntu.com/ubuntu-ports jammy-security/main s390x Packages
3.0.2-0ubuntu1 500
500 http://ports.ubuntu.com/ubuntu-ports jammy/main s390x Packages
openssl:
Installed: 3.0.2-0ubuntu1.13
Candidate: 3.0.2-0ubuntu1.13
Version table:
*** 3.0.2-0ubuntu1.13 400
400 http://ports.ubuntu.com/ubuntu-ports jammy-proposed/main s390x Packages
100 /var/lib/dpkg/status
3.0.2-0ubuntu1.12 500
500 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main s390x Packages
500 http://ports.ubuntu.com/ubuntu-ports jammy-security/main s390x Packages
3.0.2-0ubuntu1 500
500 http://ports.ubuntu.com/ubuntu-ports jammy/main s390x Packages

Successfully verified. The problem is not visible any more. Thank you!

[Adrien Nader (adrien)]

Frank and Grgo, thanks for the verification. That was very helpful.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 3.0.2-0ubuntu1.13

---------------
openssl (3.0.2-0ubuntu1.13) jammy; urgency=medium

  * Fix (upstream): crash when using an engine for ciphers used by DRBG
    (LP: #2023545)
    - lp2023545/0001-Release-the-drbg-in-the-global-default-context-befor.patch
  * Fix (upstream): do not ignore return values for S/MIME signature
    (LP: #1994165)
    - lp1994165/0001-REGRESSION-CMS_final-do-not-ignore-CMS_dataFinal-res.patch
  * Perf (upstream): don't empty method stores and provider synchronization
    records when flushing the query cache (LP: #2033422)
    - lp2033422/0001-Drop-ossl_provider_clear_all_operation_bits-and-all-.patch
    - lp2033422/0002-Refactor-method-construction-pre-and-post-condition.patch
    - lp2033422/0003-Don-t-empty-the-method-store-when-flushing-the-query.patch
    - lp2033422/0004-Make-it-possible-to-remove-methods-by-the-provider-t.patch
    - lp2033422/0005-Complete-the-cleanup-of-an-algorithm-in-OSSL_METHOD_.patch
    - lp2033422/0006-For-child-libctx-provider-don-t-count-self-reference.patch
    - lp2033422/0007-Add-method-store-cache-flush-and-method-removal-to-n.patch

 -- Adrien Nader <email address hidden> Tue, 09 Jan 2024 11:42:50 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for openssl has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-01-25 07:28 EDT-------
Thanks everyone for your work.
With the fix being released to -updates, I will close this bug as soon as the status in Launchpad changes to "fix released"

[Frank Heimes (fheimes)]

Since this bug is fixed with openssl 3.0.8 and newer,
I'm changing the status of the current devel. release to Fix Released too (since we are there on 3.0.10).
And with that the affected project status can be set to Fix Released, too.

Thx!