[24.04 FEAT] [SEC2339] HSM protected signing support for Apache httpd for openSSL 3.0 with PKCS #11 provider

[bugproxy]

The driver for this is the need to update mod_ssl in Apache2 to support
openssl 3.x providers, since engines are deprecated in openssl 3.x.

This new functionality (openssl provider support) is required for the
use case that one wants to protect the private key of a httpd server
by using a PKCS#11 based (HSM based) private key for the server
instead of using a clear key.

This would subsequently open business opportunity esp. on the s390x platform.

The diff/delta in the 2.5.x/trunk CHANGES file (https://github.com/apache/httpd/blob/trunk/CHANGES) is:
"
  *) mod_ssl: Support loading certificates and private keys from the
     PKCS#11 OpenSSL engine. [Anderson Sasaki <ansasaki redhat.com>,
     Joe Orton]
"

In addition the reference to Revision 1914365 seems to be useful reference,
that provides further details:
https://svn.apache.org/viewvc?view=revision&revision=1914365

Once backports for 2.4.x are available:
- a test build in PPA will be done (and a build log can be provided)
- install and upgrade tests will be done (and an install log can be provided)

The new package should not break any other packages that depend on it,
since there are no changes in the dependencies (or package meta data in general) expected.

A description of a sample setup, incl. all affected components, can be taken from here:
https://www.ibm.com/docs/en/linux-on-z?topic=linuxone-libp11-engine
(The sample is based on RHEL, but except the patches discussed here,
 this generally applies to other distributions as well).
'Figure - 1' provides a graphical representation of the overall use case setup.

The above sample setup does incl. test steps;
look for 'Testing' --> 'Test with Apache web server'
(Test uses "httpd -X" and "openssl s_client".)

Once an Ubuntu based Apache 2.4.x test build for noble is available,
and the logs (see above are available)
the 'ubuntu-release' team can finally be subscribed.

__________

Enable an E2E use case that allows to configure an Apache webserver to protect its private keys with an HSM that is addressable via an PKCS #11 (signing) provider configured for an openSSL 3.0 library.

Accepted for httpd > 2.4.58, see
https://svn.apache.org/viewvc?view=revision&revision=1914365

[Christian Ehrhardt  (paelzer)]

Hi @Ingo - do you already happen to know if there are other changes >.58 that are needed for this to work well?

Furthermore if we'd build a .58+change for Ubuntu noble could you test if that works on s390x as intended or do we go the inferior "compiles -> done" path :-P ?

[Christian Ehrhardt  (paelzer)]

Thanks Frank for fixing up the state

Incomplete for now, but once we know the answers from Ingo we can decide how we shall proceed with it for Noble.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-01-22 08:28 EDT-------
Besides the one commit that adds support for this (https://github.com/apache/httpd/commit/cc796e269d7c4f8d105fa46b590c9301c2a55329) I do see the following commits that are related:

https://github.com/apache/httpd/commit/2412f20b176ff54538b67088a9e643ffed6e87ae and https://github.com/apache/httpd/commit/d3a970420f04f9304e202bd1bdc04cbace9bbbd1 - French translation of related docu.

https://github.com/apache/httpd/commit/f5bf0869c7cfd6723f53115c8483737fce53fd2a
- small fix to compile with OPENSSL_NO_ENGINE environment.

None of those commits are absolutely required, but yo might want to pick them anyway. I am not aware of anything further that would be required to make this work.

I can certainly test a .58+change build for noble once a 24.04 build is available in Tessia (our IBM internal test system installer). Alternatively, if I could install it on 23.10 then it would be even easier.

I can also provide a description how to setup such an environment, it only needs httpd (.58+change), a current opencryptoki (I guess we will get this in 24.04 anyway) and the pkcs11-provider from https://github.com/latchset/pkcs11-provider (must be built from source I guess since not available in Ubuntu distribution), together with a few config file updates to set it up. I guess you can even test it on non-s390x using opencryptoki's soft token.

[Bryce Harrington (bryce)]

Thanks for flagging this commit as desired. From the provided link I do see it has landed on the 2.5.x trunk but am not spotting it in the 2.4.x backport branch? That gives me some pause. I would want to better understand the use case or requirement, and how this patch solves them.

The patch doesn't look like it will apply cleanly, although it's not large and doesn't look hard to backport. But I'm more concerned if it has other dependencies from 2.5.x that aren't present in 2.4. Ideally, this would be first backported upstream to the 2.4.x and we (or Debian) could cherrypick it.

Apache2 typically releases every few months but last release was in October, and the CHANGES file looks awfully full, so I'm suspecting we might see a new release. If 2.4.59 were released within a couple weeks and lands in Debian, it might be possible to make it in for the LTS release. Lacking a 2.4.59 release, cherrypicking a patch from 2.4.x is easy prior to feature freeze at the end of February. Other than that, it's less easy and I'd like to first better understand the need.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-02-16 08:39 EDT-------
> I would want to better understand the use case or requirement, and how this patch solves them.

The use case is that you want to protect the private key of a httpd server by using a PKCS#11 based (HSM based) private key for the server instead of a clear key.

The private key of the server is the 'identity' of the server.
If an attacker can steal the private key of the server, he can take over that server and the clients will not notice, because the server's public key and the server certificate do not change. So even with certificate/pubic-key pinning, the client won't see any changes.

A more secure way of protecting the server key is by using a HSM based key for that. An HSM based key is a so called secure key, and it is encrypted by the HSMs master key. The HSM master key will never ever leave the HSM, and thus is physically bound to the HSM. A secure key is encrypted by the HSM master key, and any crypto operation with such a key must happen inside the HSM. So even if a secure key gets stolen, it is totally useless without having the HSM, too.

PKCS#11 is a crypto API that is often used for HSM based crypto systems.
There are different PKCS#11 implementations available, of of it is Opencryptoki, which offers true HSM based crypto operations with IBM Crypto Express adapters in CCA or EP11 mode.

Unfortunately, only very little applications do support PKCS#11 directly. Many of them (including Apache httpd's mod_ssl) do use the OpenSSL crypto API to perform (clear key) crypto operations. To allow such applications to transparently use PKCS#11, there exists a so called PKCS#11 engine (libp11 engine - https://github.com/OpenSC/libp11).

Apache httpd mod_ssl since long time has support for using this engine to work with a PKCS#11 implementation for the server key and certificate. The httpd mod_ssl has special configuration keywords to enable the engine, and to specify PKCS#11 URIs for the key and the certificate, so that it uses the PKCS#11 based key instead of a clear key.

See this article about how to configure such a setup:
https://www.ibm.com/docs/en/linux-on-z?topic=linuxone-libp11-engine

With OpenSSL 3.0 the OpenSSL engines got deprecated and OpenSSL is moving to a new plugin concept, called providers. So the libp11 engine usage will end at some point in time.

Since some time, there exist two PKCS#11 providers that can kind of replace the libp11 Engine.
PKCS#11 provider: https://github.com/latchset/pkcs11-provider
PKCS#11 sign provider: https://github.com/opencryptoki/openssl-pkcs11-sign-provider

Unfortunately, the way httpd 's mod_ssl is implemented, it need some adaptions in order to use these PKCS#11 providers instead if a PKCS#11 engine. There is a lot of engine specific code in mod_ssl, that needs to be adjusted for working with providers.

This is exactly what the patch of this feature does.

So with that support, one can configure the Apache httpd with a PKCS#11 provider to use a PKCS#11 based server key and certificate, the same way as it was possible with the libp11 engine.

[Bryce Harrington (bryce)]

Hi ifranzki,

Thank you for the detailed description of the feature in question.

Given this is an important security feature, would it make more sense to get this backported by Apache to stable, or is your preference that we do the backport in Ubuntu? As I mentioned, this did land in the development trunk as you pointed out, but is not present on the Apache 2.4.x stable tree yet, and I'm not seeing it mentioned in their STATUS file (where they plan which patches to backport). Would love to hear your thoughts on this before handling of it is determined.

[Frank Heimes (fheimes)]

Since time is progressing and 24.04 FF is coming soon,
let me sum this up a bit and let me extract a few concerns/questions that arose out of this discussion:

a) In such cases Canonical/Ubuntu can take (stable) patches that are upstream accepted.
   However, in this particular case the patch(es) have landed in the 2.5.x devel trunk.
   But support for an LTS release might be granted for up to 12 years these days.
   Hence in this particular case we would need IBMs commitment on the help supporting these
   patches, and especially the ongoing push to get them into stable (which would ease the
   maintenance burden a lot - and is btw. what we usually would require).
b) Even if we talk about 'just' 4 commits (if I got things right):
    https://github.com/apache/httpd/commit/cc796e269d7c4f8d105fa46b590c9301c2a55329
    https://github.com/apache/httpd/commit/2412f20b176ff54538b67088a9e643ffed6e87ae
    https://github.com/apache/httpd/commit/d3a970420f04f9304e202bd1bdc04cbace9bbbd1
    https://github.com/apache/httpd/commit/f5bf0869c7cfd6723f53115c8483737fce53fd2a
   the changes (esp. of cc796e2) are significant.
   1) Can we be sure that no further changes that have made it over time into 2.5.x
      are needed on top (means any add. internal code dependencies) ?
   2) And has this ever be tried with 2.4.x (which would mitigate the above) ?
   3) And would you share the backported version of the commit(s) for the 2.4.x trunk?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-02-21 04:09 EDT-------
> ... would it make more sense to get this backported by Apache to stable, or is your preference that we do the backport in Ubuntu?

I am fine with both. It would certainly be good to have this is stable, since it would make things much easier. However, I have no clue how the Apache community handles such things.

I would guess that backporting these changes should not be a too huge effort, since I would guess that mod_ssl did not change much in the areas affected by the patches since quite long time. However, I have not tried to backport them.

> 1) Can we be sure that no further changes that have made it over time into 2.5.x
are needed on top (means any add. internal code dependencies) ?

I don't think so, but I am also not that deep into mod_ssl that I can answer that.

> 2) And has this ever be tried with 2.4.x (which would mitigate the above) ?

No, I have not tried.

> 3) And would you share the backported version of the commit(s) for the 2.4.x trunk?

I don't have any.

[Andreas Hasenack (ahasenack)]

I'll take a look at this.

[Andreas Hasenack (ahasenack)]

Ok, so in essence this is about updating mod_ssl to support openssl 3 providers, since engines are deprecated in openssl 3.

[Frank Heimes (fheimes)]

@ahasenack Yes, that is the main driver for this.
Let me pick this and put it also into the bug description.

[Andreas Hasenack (ahasenack)]

Working on this today.

[Andreas Hasenack (ahasenack)]

The main patch[1] depends on these other two:
- improve compatibility with openssl 3[2]
- fix typo[3]

With all three, the main[1] one appplies with some offset:
Applying patch mod_ssl_support_pkcs11_provider_for_ossl3.patch
patching file modules/ssl/ssl_engine_init.c
Hunk #1 succeeded at 1411 (offset -65 lines).
Hunk #2 succeeded at 1425 (offset -65 lines).
patching file modules/ssl/ssl_engine_pphrase.c
Hunk #2 succeeded at 611 (offset 32 lines).
Hunk #3 succeeded at 829 (offset 33 lines).
Hunk #4 succeeded at 910 (offset 33 lines).
patching file modules/ssl/ssl_private.h
patching file modules/ssl/ssl_util.c
Hunk #1 succeeded at 476 (offset -24 lines).

Now at patch mod_ssl_support_pkcs11_provider_for_ossl3.patch

Testing a build...

1. https://github.com/apache/httpd/commit/cc796e269d7c4f8d105fa46b590c9301c2a55329
2. https://github.com/apache/httpd/commit/28f6fc01c379282b647758c68ab59074dc4533df
3. https://github.com/apache/httpd/commit/43f7bc4508cc3750ee3a0c01a73d21f23fa2eee2

[Andreas Hasenack (ahasenack)]

I will also note that the main patch[1] has not yet been proposed for 2.4, according to the STATUS[2] file in the 2.4.x branch.

1. https://github.com/apache/httpd/commit/cc796e269d7c4f8d105fa46b590c9301c2a55329
2. https://github.com/apache/httpd/blob/2.4.x/STATUS

[Andreas Hasenack (ahasenack)]

I have builds going on in this ppa for amd64, arm64, and s390x:

https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/

Completely untested. Once the builds are published, I'll trigger the autopkgtests.

[Andreas Hasenack (ahasenack)]

> the pkcs11-provider from https://github.com/latchset/pkcs11-provider (must be built from source I guess
> since not available in Ubuntu distribution)

That package is available in noble, btw: https://launchpad.net/ubuntu/+source/pkcs11-provider

[Frank Heimes (fheimes)]

Hi Andreas, thanks for your patched Apache 2.4 build, great!
(I'm curious about the autopkgtest results.)

Regarding the provider, there are different implementations out there.
This one - available in noble - is supposed to work with this Apache2 modification:
https://launchpad.net/ubuntu/+source/openssl-pkcs11-sign-provider
 (we picked the most promising and future-oriented one, backed and maintained by IBM)

[Andreas Hasenack (ahasenack)]

I may have hit a bug elsewhere first, though. I'm following what I did for a pkcs11 engine test[1], but with the pkcs11-provider package. I'm able to create the RSA key in the softhsm2 token, and even generate a certificate request with it using openssl -provider pkcs11. But when I sign the request with the same key (nonsense, but technically valid), it does sign it, but core dumps at the end:

# openssl x509 -provider pkcs11 -signkey "pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=f4561bbe1b739173;token=apache2-hsm-token;id=%BD%06%9A%2E%16%D0%03%85%AE%AF%12%DE%81%0C%DA%3A%56%F2%51%42;object=apache2-hsm-key;type=private" -in apache2-hsm-key.req -out foo
Enter pass phrase for PKCS#11 Token (Slot 460558707 - SoftHSM slot ID 0x1b739173):
Segmentation fault (core dumped)

# cat foo
-----BEGIN CERTIFICATE-----
MIICrzCCAZcCFHRHl/ehMDanzecCjxubJu2fKX5KMA0GCSqGSIb3DQEBCwUAMBQx
EjAQBgNVBAMMCW4taHNtLmx4ZDAeFw0yNDAzMDYxOTAxNDVaFw0yNDA0MDUxOTAx
NDVaMBQxEjAQBgNVBAMMCW4taHNtLmx4ZDCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAMCkFhFN9NtpzchsT1SlrTDHANe9d5L1NR7FNpXJjCZAkEEkDmP9
4CE5dWp7X2RlGeZ9Ge+EMC84ud2Kx6O5MFoEBi/h8fmy6FPtRBlLyx+wExGLAeRR
puyBE6GpYWDmUlYG1XvwiUmESVZ4U8QJiQgF/0euu8ldbIyqa8zi20dqI+T9HiuL
TDcFnD+95xUbQ6Lsz8F5zbEps5BQZmV7MjsAHNnG24CCwjRpQr244tVeYLDhtVE8
/m7y8HGK29eto9cBypedUxAzzwofLTN3UShb2PX5ffOmT/n0ifxbBC7LpmbbEWT6
47mWvtru/9eME3y3UBs5d928cSxL8d+kGisCAwEAATANBgkqhkiG9w0BAQsFAAOC
AQEACgeFvFFyugMKJtfT5Jgpfk4aBFtcYhJonQ/woqmEi50KU4bbZMQeFXWnNdVx
ktdWRzbxJgIITllUVnSaP7Iyef+6qI35FgrIefqWLr98tT6X2kMuZn1mJU5HuMco
HL4ibOcNNb2PKCJkTVXhJkIQSiEYQoGgevKrxOyjOUIg6OxibWvqATQgWG/9THHF
VnAnaSSkCRO9D5FD4RvLlwTiUS5g/TZJwcbj5bxtuNjTBXY5NdqMATlTdVSbmS1E
SRX06Fsk8mwD1I26/eIRxoD7iGdrvGjA2YXb1OehLTc/rI8eaHEqbfyNliwiJCOA
wuSORq/F24ydDjVYsvbtDV5VkQ==
-----END CERTIFICATE-----

The certificate looks ok, and a quick gdb on the core dump shows it was at shutdown time. But I'm also getting a core dump in apache now when configured to use this cert and hsm key. But also at shutdown. And while running, apache ssl isn't working. Still, it could be because softhsm2 usually requires root access, but I straced it and didn't see any EACCESS errors, and I also added the www-data user to the softhsm group.

Still, the segfault isn't good, and seems to be in either softhsm2 or pkcs11-provider, not apache itself.

1. https://git.launchpad.net/ubuntu/+source/libp11/tree/debian/tests/engine

[Andreas Hasenack (ahasenack)]

> (I'm curious about the autopkgtest results.)

They passed for amd64 already:
Results: (from http://autopkgtest.ubuntu.com/results/autopkgtest-noble-ahasenack-apache2-modssl-provider-support/?format=plain)
  apache2 @ amd64:
    06.03.24 17:05:50 Log 🗒️ ✅ Triggers: apache2/2.4.58-1ubuntu4~ppa1

log: https://autopkgtest.ubuntu.com/results/autopkgtest-noble-ahasenack-apache2-modssl-provider-support/noble/amd64/a/apache2/20240306_170550_0cb4e@/log.gz

but failed for s390x all over the place:
  apache2 @ s390x:
    06.03.24 17:54:30 Log 🗒️ ❌ Triggers: apache2/2.4.58-1ubuntu4~ppa1
      1617s run-test-suite FAIL 🟥
      1617s duplicate-module-load FAIL 🟥
      1617s default-mods FAIL 🟥
      1617s htcacheclean FAIL 🟥
      1617s ssl-passphrase FAIL 🟥
      1617s check-http2 FAIL 🟥
      1617s check-ubuntu-branding FAIL 🟥
      1617s chroot FAIL 🟥

Log: https://autopkgtest.ubuntu.com/results/autopkgtest-noble-ahasenack-apache2-modssl-provider-support/noble/s390x/a/apache2/20240306_175430_7cb94@/log.gz

It's a dependency problem while installing packages. The archive is still in a lot of flux due to the time_t 64bit changes, which mean a lot of rebuilds. I'll retry later.

[Andreas Hasenack (ahasenack)]

let me also try with openssl-pkcs11-sign-provider

[Andreas Hasenack (ahasenack)]

I'm having difficulties with openssl-pkcs11-sign-provider. I'm getting a sequence of errors, a segfault, and it looks like it's trying to load the rdrand.so *engine*, which we are not shipping (might not even exist anymore?)

# openssl req -provider pkcs11sign -new -key "pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=148c784165ed428b;token=test-token;id=%96%7F%20%F2%98%18%D7%15%3D%AF%87%AB%EC%09%25%C5%14%51%2E%E1;object=test-key;type=private" -out test-key.req -text -x509 -subj "/CN=n-hsm.lxd"
Could not open file or uri for loading private key from pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=148c784165ed428b;token=test-token;id=%96%7F%20%F2%98%18%D7%15%3D%AF%87%AB%EC%09%25%C5%14%51%2E%E1;object=test-key;type=private
4067AC93797F0000:error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file
4067AC93797F0000:error:80000002:system library:file_open:No such file or directory:../providers/implementations/storemgmt/file_store.c:267:calling stat(pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=148c784165ed428b;token=test-token;id=%96%7F%20%F2%98%18%D7%15%3D%AF%87%AB%EC%09%25%C5%14%51%2E%E1;object=test-key;type=private)
4067AC93797F0000:error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=pkcs11
4067AC93797F0000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:../crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/x86_64-linux-gnu/engines-3/rdrand.so): /usr/lib/x86_64-linux-gnu/engines-3/rdrand.so: cannot open shared object file: No such file or directory
4067AC93797F0000:error:12800067:DSO support routines:DSO_load:could not load the shared library:../crypto/dso/dso_lib.c:152:
4067AC93797F0000:error:13000084:engine routines:dynamic_load:dso not found:../crypto/engine/eng_dyn.c:442:
4067AC93797F0000:error:13000074:engine routines:ENGINE_by_id:no such engine:../crypto/engine/eng_list.c:430:id=rdrand
Segmentation fault (core dumped)

The openssl config is (abbreviated, and note I have disabled the pcks11 provider for this test):

[openssl_init]
providers = provider_sect
alg_section = evp_properties

[evp_properties]
default_properties = ?provider=pkcs11sign

[provider_sect]
default = default_sect
#pkcs11 = pkcs11_sect
pkcs11sign = pkcs11sign_sect

[pkcs11sign_sect]
module = /usr/lib/x86_64-linux-gnu/ossl-modules/pkcs11sign.so
identity = pkcs11sign
pkcs11sign-forward = provider=default
pkcs11sign-module-path = /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so
activate = 1

[pkcs11_sect]
module = /usr/lib/x86_64-linux-gnu/ossl-modules/pkcs11.so
pkcs11-module-path = /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so
#pkcs11-module-token-pin = file:/etc/apache2/pin.txt
activate = 0

[default_sect]
activate = 1

The pkcs11sign provider is recognized:

root@n-hsm:~# openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.0.10
    status: active
  pkcs11sign...

[Andreas Hasenack (ahasenack)]

It's really trying to open the pkcs11 URI as a file... :/

newfstatat(AT_FDCWD, "pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=148c784165ed428b;token=test-token;id=%96%7F%20%F2%98%18%D7%15%3D%AF%87%AB%EC%09%25%C5%14%51%2E%E1;object=test-key;type=private;pin-value=1234", 0x7ffca135a450, 0) = -1 ENOENT (No such file or directory)

I think something in my setup broke, because it's also failing with the pkcs11 module which worked before(tm).

UPDATE: hm, having only one pkcs11 provider loaded at once seems better. It kind of worked when I commented out (removed) pkcs11sign from ssl.cnf. I still get a core dump, but the request file is generated:

# openssl req -provider pkcs11 -new -key "pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=148c784165ed428b;token=test-token;id=%96%7F%20%F2%98%18%D7%15%3D%AF%87%AB%EC%09%25%C5%14%51%2E%E1;object=test-key;type=private;pin-value=1234" -out test-key.req -text -x509 -subj "/CN=n-hsm.lxd"
Segmentation fault (core dumped)

# l test-key.req
-rw-r--r-- 1 root root 4.3K Mar 6 20:18 test-key.req

Still, with just pkcs11sign, and no pkcs11, it still didn't work.

UPDATE2: the stat on the pkcs11 URI as a file also happens with the pkcs11 provider, so it's probably unrelated (like an attempt: is it a file? No? Ok, is it something else? And so on)

[Andreas Hasenack (ahasenack)]

This comment[1] suggested a quirk, and after I added it, it got rid of the segfault with the pkcs11-provider.

1. https://github.com/latchset/pkcs11-provider/issues/310#issuecomment-1821547394

[Andreas Hasenack (ahasenack)]

And signing the request file now also works without a segfault (still using the pkcs11-provider, not pkcs11sign):

# openssl x509 -provider pkcs11 -signkey "pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=148c784165ed428b;token=test-token;id=%96%7F%20%F2%98%18%D7%15%3D%AF%87%AB%EC%09%25%C5%14%51%2E%E1;object=test-key;type=private;pin-value=1234" -in test-key.req -out test-key.crt

I'll try with apache again next, tomorrow.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-07 02:42 EDT-------
You can also try with Opencryptoki instead of SoftHSM. Opencryptoki provides a soft token, wich also can be used on non-s390x platforms. Please make sure you use Opencryptoki 3.23.0. This release should be in 24.04 anyway.

Regarding the pkcs11-sign-provider: Did you upgrade it to the 1.0.1 release?
https://github.com/opencryptoki/openssl-pkcs11-sign-provider/releases/tag/v1.0.1
This includes some important fixes regarding fork support (required for Apache).

Note: I would NOT recommend to use 'openssl -provider xxxx', but configure the provider in the OpenSSL config file (needed anyway), and thus have the provider loaded automatically. When using 'openssl -provider xxxx' it might happen that algos that are not provided by the specified provider are not available. You really want to use the PKCS#11 provider ONLY for operations with the signing key, but not for anything else.

Can the s390x package from https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/ be installed on a 23.10 as well? If so, I can give it a try myself, too.

[Frank Heimes (fheimes)]

Just a side note,
we should have the right package versions in 24.04 as of today:
$ rmadison --suite=noble,noble-proposed openssl-pkcs11-sign-provider opencryptoki
 openssl-pkcs11-sign-provider | 1.0.1-0ubuntu1 | noble/universe | source, amd64, arm64, armhf, ppc64el, s390x
 openssl-pkcs11-sign-provider | 1.0.1-0ubuntu2 | noble-proposed/universe | source, amd64, arm64, ppc64el, s390x
 opencryptoki | 3.23.0+dfsg-0ubuntu1 | noble/universe | source, amd64, arm64, armhf, ppc64el, s390x
 opencryptoki | 3.23.0+dfsg-0ubuntu2 | noble-proposed/universe | source, amd64, arm64, ppc64el, s390x
(Notice that the "0ubuntu2" versions of these packages that are currently in -proposed are not urgently needed in this case, since they are 'no-change rebuilds' against libssl3t64, which is only relevant for arm.)

And it looks like the patches apache packages are installable on 23.10 as well - easiest is probably:
Quickly wget them:
wget https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/apache2-bin_2.4.58-1ubuntu4~ppa1_s390x.deb https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/apache2-data_2.4.58-1ubuntu4~ppa1_all.deb https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/apache2-dev_2.4.58-1ubuntu4~ppa1_s390x.deb https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/apache2-doc_2.4.58-1ubuntu4~ppa1_all.deb https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/apache2-ssl-dev_2.4.58-1ubuntu4~ppa1_s390x.deb https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/apache2-suexec-custom_2.4.58-1ubuntu4~ppa1_s390x.deb https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/apache2-suexec-pristine_2.4.58-1ubuntu4~ppa1_s390x.deb https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/apache2-utils_2.4.58-1ubuntu4~ppa1_s390x.deb https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/apache2_2.4.58-1ubuntu4~ppa1_s390x.deb https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/libapache2-mod-md_2.4.58-1ubuntu4~ppa1_s390x.deb https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/+files/libapache2-mod-proxy-uwsgi_2.4.58-1ubuntu4~ppa1_s390x.deb
And install from local:
$ sudo apt install ./apache2_2.4.58-1ubuntu4~ppa1_s390x.deb ./apache2-bin_2.4.58-1ubuntu4~ppa1_s390x.deb ./apache2-data_2.4.58-1ubuntu4~ppa1_all.deb ./apache2-utils_2.4.58-1ubuntu4~ppa1_s390x.deb ./libapache2-mod-md_2.4.58-1ubuntu4~ppa1_s390x.deb ssl-cert

However, you can also quickly upgrade your 23.10 system to 24.04, which is probably recommended in this case, to get real 24.04 test results (of course only if your system is test/dev system and not a production system). These are the steps:
1) ensure you have all the latest updates installed on your 23.10 system:
   sudo apt -q -y update && sudo apt -q -y full-upgrade
   (and in case you got a new kernel, you may reboot your system)
2) then...

[Andreas Hasenack (ahasenack)]

> Regarding the pkcs11-sign-provider: Did you upgrade it to the 1.0.1 release?

Yes, I was using 1.0.1 from noble:

openssl-pkcs11-sign-provider 1.0.1-0ubuntu1

And pkcs11-provider 0.3-1.

> Note: I would NOT recommend to use 'openssl -provider xxxx', but configure the provider in the OpenSSL
> config file

It's what I did. openssl list -providers works without further options, indicating the system-wide openssl config file is loading the module:

$ openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.0.10
    status: active
  pkcs11sign
    name: PKCS11 signing key provider
    version: 1.0.1
    status: active

I think apache is not even trying, or not able, to load the private key from softhsm2. When I start it in the foreground with -X, it doesn't prompt for the pin. And it doesn't change if I give the pin-value in the pkcs11 URI or not. More investigation/testing is needed. This setup is somewhat complex, involving multiple libraries from different source packages, it's quite possible I did something wrong.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-07 09:29 EDT-------
Can I see the source tree somewhere? I could then check if the relevant hunks are correct.

[Andreas Hasenack (ahasenack)]

Sure. For any package in ubuntu, the git repository is in:

https://code.launchpad.net/ubuntu/+source/<package>

So, for apache2 for example, it would be:

https://code.launchpad.net/ubuntu/+source/apache2

Then, the branches. Quick intro:

ubuntu/devel: TIP. Currently this points at 24.04 noble
ubuntu/<release>-devel: is the TIP for that release. For example, ubuntu/jammy-devel would be TIP for 22.04 jammy. This includes packages currently in the -proposed repositories.

applied/<whatever>: same as above, but with all patches from debian/patches/* applied to the source tree.

In the case of unreleased packages, like my apache2 PPA packages, then it's in my own namespace. In the https://code.launchpad.net/ubuntu/+source/apache2 page, scroll down until "Other repositories", and there will be https://code.launchpad.net/~ahasenack/ubuntu/+source/apache2/+git/apache2, and you can usually locate the correct branch name in there.

So for pkcs11-provider, the repo is https://code.launchpad.net/ubuntu/+source/pkcs11-provider

And for openssl-pkcs11-sign-provider, the repo is https://code.launchpad.net/ubuntu/+source/openssl-pkcs11-sign-provider

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-07 10:36 EDT-------
I have downloaded the packages and installed them on my 23.10. An upgrade to 24.04 did not work, do-release-upgrade downloaded something and then just exited without doing anything....

Anyway, I got the packages installed on 23.10 as well.

# apache2 -v
Server version: Apache/2.4.58 (Ubuntu)
Server built: 2024-03-06T14:36:12

I did build opencryptoki and the pkcs11-provider from source to be sure to have the latest.

Here is what I did to set it up:

1.) Add the pkcs11-providetr to openssl.cnf:

[pkcs11_sect]
module = /usr/local/lib/ossl-modules/pkcs11.so
pkcs11-module-path = /usr/local/lib/opencryptoki/libopencryptoki.so
pkcs11-module-load-behavior = early
activate = 1

# openssl list -providers
Providers:
default
name: OpenSSL Default Provider
version: 3.0.10
status: active
pkcs11
name: PKCS#11 Provider
version: 3.0.10
status: active

2.) Generate a server key using p11sak:
# p11sak generate-key rsa 2048 --label "httpd server key" --id "0011223344556677" --slot 3
Successfully generated a RSA key pair with labels "httpd server key:pub":"httpd server key:prv".

Its important that the key has an CKA_ID attribute (--id option), this is required by the pkcs#11 provider to identify the key.

3.) List the key to get the URI:
# p11sak list-key all --slot 3 --long

The URI of the private key is
pkcs11:manufacturer=IBM;model=Soft;token=soft;id=%00%11%22%33%44%55%66%77;object=httpd%20server%20key:prv;type=private

4.) Export the public key:
# p11sak export-key rsa --slot 3 --pin yaq12wsx --file /etc/ssl/httpd.pub --label "httpd server key:pub"
Are you sure you want to export public RSA 2048 key object "httpd server key:pub" [y/n/a/c]? y
Successfully exported public RSA 2048 key object "httpd server key:pub" to file '/etc/ssl/httpd.pub'.
1 key object(s) exported.

5.) Generate a self signed certificate with the private key:

# openssl req -new -x509 -out /etc/ssl/httpd.crt -key "pkcs11:manufacturer=IBM;model=Soft;token=soft;id=%00%11%22%33%44%55%66%77;object=httpd%20server%20key:prv;type=private"
Enter pass phrase for PKCS#11 Token (Slot 3 - Linux):
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:DE
State or Province Name (full name) [Some-State]:BW
Locality Name (eg, city) []:BB
Organization Name (eg, company) [Internet Widgits Pty Ltd]:IBM
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:www.example.com
Email Address []:<email address hidden>

6.) Enabled mod-ssl for apache:

# a2enmod ssl
Considering dependency mime for ssl:
Module mime already enabled
Considering dependency socache_shmcb for ssl:
Enabling module socache_shmcb.
Enabling module ssl.
See /usr/share/doc/apache2/README.Debian.gz on how to configure SSL and create self-signed certificates.
To activate the new configuration, you need to run:
systemctl restart ap...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-07 10:53 EDT-------
FYI: Reported the deadlock via https://github.com/latchset/pkcs11-provider/issues/355 . Not sure if its anything that they can prevent though.

[Andreas Hasenack (ahasenack)]

The deadlock in pkcs11-provider was fixed by upstream via https://github.com/latchset/pkcs11-provider/pull/356/

[Frank Heimes (fheimes)]

The IBM team agreed upon the proposal to let's go with the PPA solution for now, until upstream accepted (and reconsider in this case).

(So I think I'm updating the status of this ticket to 'Opinion'.)

[Andreas Hasenack (ahasenack)]

Which PPA do you mean, https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/ ? Or another one to be created?

[Frank Heimes (fheimes)]

Yes, I thought about https://launchpad.net/~ahasenack/+archive/ubuntu/apache2-modssl-provider-support/

But we can also copy it to another PPA or I can re-create it based on your, if you prefer not to leave yours.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-07-02 09:03 EDT-------
Thanks everyone for your work!
With having the solution available as PPA, we can close this bug.

[Andreas Hasenack (ahasenack)]

This is the official PPA for this effort:

https://launchpad.net/~canonical-server/+archive/ubuntu/lp-2050017-apache2-modssl-provider-support

I just copied the packages over from my personal one, and they should be published in the next few hours.