Comment 30 for bug 2050017

------- Comment From <email address hidden> 2024-03-07 10:36 EDT-------
I have downloaded the packages and installed them on my 23.10. An upgrade to 24.04 did not work, do-release-upgrade downloaded something and then just exited without doing anything....

Anyway, I got the packages installed on 23.10 as well.

# apache2 -v
Server version: Apache/2.4.58 (Ubuntu)
Server built: 2024-03-06T14:36:12

I did build opencryptoki and the pkcs11-provider from source to be sure to have the latest.

Here is what I did to set it up:

1.) Add the pkcs11-providetr to openssl.cnf:

[pkcs11_sect]
module = /usr/local/lib/ossl-modules/pkcs11.so
pkcs11-module-path = /usr/local/lib/opencryptoki/libopencryptoki.so
pkcs11-module-load-behavior = early
activate = 1

# openssl list -providers
Providers:
default
name: OpenSSL Default Provider
version: 3.0.10
status: active
pkcs11
name: PKCS#11 Provider
version: 3.0.10
status: active

2.) Generate a server key using p11sak:
# p11sak generate-key rsa 2048 --label "httpd server key" --id "0011223344556677" --slot 3
Successfully generated a RSA key pair with labels "httpd server key:pub":"httpd server key:prv".

Its important that the key has an CKA_ID attribute (--id option), this is required by the pkcs#11 provider to identify the key.

3.) List the key to get the URI:
# p11sak list-key all --slot 3 --long

The URI of the private key is
pkcs11:manufacturer=IBM;model=Soft;token=soft;id=%00%11%22%33%44%55%66%77;object=httpd%20server%20key:prv;type=private

4.) Export the public key:
# p11sak export-key rsa --slot 3 --pin yaq12wsx --file /etc/ssl/httpd.pub --label "httpd server key:pub"
Are you sure you want to export public RSA 2048 key object "httpd server key:pub" [y/n/a/c]? y
Successfully exported public RSA 2048 key object "httpd server key:pub" to file '/etc/ssl/httpd.pub'.
1 key object(s) exported.

5.) Generate a self signed certificate with the private key:

# openssl req -new -x509 -out /etc/ssl/httpd.crt -key "pkcs11:manufacturer=IBM;model=Soft;token=soft;id=%00%11%22%33%44%55%66%77;object=httpd%20server%20key:prv;type=private"
Enter pass phrase for PKCS#11 Token (Slot 3 - Linux):
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:DE
State or Province Name (full name) [Some-State]:BW
Locality Name (eg, city) []:BB
Organization Name (eg, company) [Internet Widgits Pty Ltd]:IBM
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:www.example.com
Email Address []:<email address hidden>

6.) Enabled mod-ssl for apache:

# a2enmod ssl
Considering dependency mime for ssl:
Module mime already enabled
Considering dependency socache_shmcb for ssl:
Enabling module socache_shmcb.
Enabling module ssl.
See /usr/share/doc/apache2/README.Debian.gz on how to configure SSL and create self-signed certificates.
To activate the new configuration, you need to run:
systemctl restart apache2

7.) Added the following to /etc/apache2/mods-available/ssl.conf

<VirtualHost *:443>
ServerName www.example.com
SSLEngine on
SSLCertificateFile "/etc/ssl/httpd.crt"
SSLCertificateKeyFile "pkcs11:manufacturer=IBM;model=Soft;token=soft;id=%00%11%22%33%44%55%66%77;object=httpd%20server%20key:prv;type=private"
</VirtualHost>

8.) Start apache2 with -X (forground):

# source /etc/apache2/envvars
# apache2 -X

No PIN prompt..... no connection to port 443 possible.

Running in gdb shows that it seems to hang in the pkcs11-provider in p11prov_slot_fork_prepare()....

__futex_abstimed_wait_common (futex_word=futex_word@entry=0x2aa00105d60, expected=expected@entry=2, clockid=clockid@entry=0,
abstime=abstime@entry=0x0, private=private@entry=0, cancel=false) at futex-internal.c:103
103 futex-internal.c: No such file or directory.
(gdb) bt
#0 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x2aa00105d60, expected=expected@entry=2, clockid=clockid@entry=0,
abstime=abstime@entry=0x0, private=private@entry=0, cancel=false) at futex-internal.c:103
#1 0x000003fff7a98918 in __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0x2aa00105d60, expected=expected@entry=2,
clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:128
#2 0x000003fff7aa2964 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0x2aa00105d58) at pthread_rwlock_common.c:829
#3 ___pthread_rwlock_wrlock (rwlock=0x2aa00105d58) at pthread_rwlock_wrlock.c:26
#4 0x000003fff6db0396 in p11prov_slot_fork_prepare (sctx=<optimized out>) at slot.c:243
#5 0x000003fff6da362e in fork_prepare () at provider.c:80
#6 0x000003fff7b09290 in __run_prefork_handlers (do_locking=do_locking@entry=true) at register-atfork.c:141
#7 0x000003fff7aeb93e in __libc_fork () at fork.c:51
#8 0x000003fff7d31a76 in apr_proc_create () from /lib/s390x-linux-gnu/libapr-1.so.0
#9 0x000003fff6fac440 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#10 0x000003fff6faf83a in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#11 0x000003fff6fa26ce in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#12 0x000003fff6b2bda2 in UI_process () from /lib/s390x-linux-gnu/libcrypto.so.3
#13 0x000003fff6acee56 in ?? () from /lib/s390x-linux-gnu/libcrypto.so.3
#14 0x000003fff6acf37c in ?? () from /lib/s390x-linux-gnu/libcrypto.so.3
#15 0x000003fff6da972c in token_login (session=session@entry=0x2aa0014f5d0, uri=uri@entry=0x2aa000f13d0, pw_cb=0x3fff6acf360,
pw_cb@entry=<error reading variable: value has been optimized out>, pw_cbarg=0x2aa0014f598,
pw_cbarg@entry=<error reading variable: value has been optimized out>, slot=slot@entry=0x2aa001064d0, user_type=1)
at session.c:475
#16 0x000003fff6daafa0 in slot_login (slot=<optimized out>, slot@entry=0x2aa001064d0, uri=uri@entry=0x2aa000f13d0,
pw_cb=<optimized out>, pw_cb@entry=0x3fff6acf360, pw_cbarg=<optimized out>, pw_cbarg@entry=0x2aa0014f598,
reqlogin=reqlogin@entry=true, _session=0x0) at session.c:781
#17 0x000003fff6dab4fe in p11prov_get_session (provctx=0x2aa000f6860, slotid=slotid@entry=0x3ffffff96c0,
next_slotid=next_slotid@entry=0x3ffffff96c8, uri=0x2aa000f13d0, mechtype=mechtype@entry=18446744073709551615,
pw_cb=0x3fff6acf360, pw_cbarg=0x2aa0014f598, reqlogin=true, rw=false, _session=0x2aa0014f4a8) at session.c:915
#18 0x000003fff6db21a8 in store_fetch (pw_cbarg=0x2aa0014f598, pw_cb=0x3fff6acf360, ctx=0x2aa0014f420) at store.c:122
#19 p11prov_store_load (pctx=0x2aa0014f420, object_cb=0x3fff6b20430, object_cbarg=<optimized out>, pw_cb=0x3fff6acf360,
pw_cbarg=0x2aa0014f598) at store.c:253
#20 0x000003fff6b1ebf8 in OSSL_STORE_load () from /lib/s390x-linux-gnu/libcrypto.so.3
#21 0x000003fff6fab692 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#22 0x000003fff6f96988 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#23 0x000003fff6f999c2 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#24 0x000002aa00062590 in ap_run_post_config ()
#25 0x000002aa0003b32e in main ()

However, in #20 you can see that mod-sll is doing OSSL_STORE_load to load the key from the URI. This calls the PKCS#11 provider's p11prov_store_load() and this goes up to token_login() which is about to prompt for the pin, which calls back into OpenSSL and finally mod-ssl. This then leads to a call to apr_proc_create() which performs a fork (why ??????), and this leads to a deadlock in fork_prepare().

I got around this by changing SSLPassPhraseDialog from 'exec:/usr/share/apache2/ask-for-passphrase' to 'builtin' in /etc/apache2/mods-available/ssl.conf. With that it does prompt for the PIN.

And with that it accepts an SSL connection:

# wget https://localhost --no-check-certificate
--2024-03-07 15:28:19-- https://localhost/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:443... connected.
WARNING: cannot verify localhost's certificate, issued by ?<email address hidden>,CN=www.example.com,O=IBM,L=BB,ST=BW,C=DE?:
Self-signed certificate encountered.
WARNING: certificate common name ?www.example.com? doesn't match requested host name ?localhost?.
HTTP request sent, awaiting response... 200 OK
Length: 10671 (10K) [text/html]
Saving to: ?index.html.1?

index.html.1 100%[============================================================>] 10.42K --.-KB/s in 0s

2024-03-07 15:28:19 (513 MB/s) - ?index.html.1? saved [10671/10671]

With '# pkcsstats --slot 3' I can see that for each connection, the counter for CKM_SHA256_RSA_PKCS_PSS gets increased. So the PKCS#11 private key is used.

Note: You can omit the interactive PIN prompt by appending '?pin-value=<pin>' or '?pin-source=</file/containing/pin>'. This would probably also help for resolving the deadlock.....