[FFe] [24.04 FEAT] [SEC2339] HSM protected signing support for Apache httpd for openSSL 3.0 with PKCS #11 provider

Bug #2050017 reported by bugproxy
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Triaged
High
Skipper Bug Screeners
apache2 (Ubuntu)
Triaged
Undecided
Andreas Hasenack

Bug Description

Feature Freeze Exception (FFe):
-------------------------------

Since this may take a little longer now and noble's FF is coming up soon,
I'm pro-actively transferring this request into a feature freeze exception (FFe).

The main reason for this request is a new functionality and the use case that one wants to protect the private key of a httpd server by using a PKCS#11 based (HSM based) private key for the server instead of using a clear key.
Which would subsequently open business opportunity esp. on the s390x platform.

The diff/delta in the 2.5.x/trunk CHANGES file (https://github.com/apache/httpd/blob/trunk/CHANGES) is:
"
  *) mod_ssl: Support loading certificates and private keys from the
     PKCS#11 OpenSSL engine. [Anderson Sasaki <ansasaki redhat.com>,
     Joe Orton]
"

In addition a reference to Revision 1914365 seems to be useful, that provides further details:
https://svn.apache.org/viewvc?view=revision&revision=1914365

Once backports for 2.4.x are available:
- a test build in PPA will be done (and a build log can be provided)
- install and upgrade tests will be done (and an install log can be provided)

The new package should not break any other packages that depend on it,
since there are no changes in the dependencies (or package meta data in general) expected.

A description of a sample setup, incl. all affected components, can be taken from here:
https://www.ibm.com/docs/en/linux-on-z?topic=linuxone-libp11-engine
(The sample is based on RHEL, but except the patches discussed here,
 this generally applies to other distributions as well).
'Figure - 1' provides a graphical representation of the overall use case setup.

The above sample setup does incl. test steps;
look for 'Testing' --> 'Test with Apache web server'
(Test uses "httpd -X" and "openssl s_client".)

Once an Ubuntu based Apache 2.4.x test build for noble is available,
and the logs (see above are available)
the 'ubuntu-release' team can finally be subscribed.

__________

Enable an E2E use case that allows to configure an Apache webserver to protect its private keys with an HSM that is addressable via an PKCS #11 (signing) provider configured for an openSSL 3.0 library.

Accepted for httpd > 2.4.58, see
https://svn.apache.org/viewvc?view=revision&revision=1914365

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-204743 severity-high targetmilestone-inin2404
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
information type: Private → Public
affects: linux (Ubuntu) → apache2 (Ubuntu)
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi @Ingo - do you already happen to know if there are other changes >.58 that are needed for this to work well?

Furthermore if we'd build a .58+change for Ubuntu noble could you test if that works on s390x as intended or do we go the inferior "compiles -> done" path :-P ?

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Changed in apache2 (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → nobody
Changed in ubuntu-z-systems:
importance: Undecided → High
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks Frank for fixing up the state

Incomplete for now, but once we know the answers from Ingo we can decide how we shall proceed with it for Noble.

Changed in apache2 (Ubuntu):
status: New → Incomplete
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2024-01-22 08:28 EDT-------
Besides the one commit that adds support for this (https://github.com/apache/httpd/commit/cc796e269d7c4f8d105fa46b590c9301c2a55329) I do see the following commits that are related:

https://github.com/apache/httpd/commit/2412f20b176ff54538b67088a9e643ffed6e87ae and https://github.com/apache/httpd/commit/d3a970420f04f9304e202bd1bdc04cbace9bbbd1 - French translation of related docu.

https://github.com/apache/httpd/commit/f5bf0869c7cfd6723f53115c8483737fce53fd2a
- small fix to compile with OPENSSL_NO_ENGINE environment.

None of those commits are absolutely required, but yo might want to pick them anyway. I am not aware of anything further that would be required to make this work.

I can certainly test a .58+change build for noble once a 24.04 build is available in Tessia (our IBM internal test system installer). Alternatively, if I could install it on 23.10 then it would be even easier.

I can also provide a description how to setup such an environment, it only needs httpd (.58+change), a current opencryptoki (I guess we will get this in 24.04 anyway) and the pkcs11-provider from https://github.com/latchset/pkcs11-provider (must be built from source I guess since not available in Ubuntu distribution), together with a few config file updates to set it up. I guess you can even test it on non-s390x using opencryptoki's soft token.

Frank Heimes (fheimes)
Changed in apache2 (Ubuntu):
status: Incomplete → Triaged
Changed in ubuntu-z-systems:
status: New → Triaged
tags: added: server-todo
Revision history for this message
Bryce Harrington (bryce) wrote : Re: [24.04 FEAT] [SEC2339] HSM protected signing support for Apache httpd for openSSL 3.0 with PKCS #11 provider

Thanks for flagging this commit as desired. From the provided link I do see it has landed on the 2.5.x trunk but am not spotting it in the 2.4.x backport branch? That gives me some pause. I would want to better understand the use case or requirement, and how this patch solves them.

The patch doesn't look like it will apply cleanly, although it's not large and doesn't look hard to backport. But I'm more concerned if it has other dependencies from 2.5.x that aren't present in 2.4. Ideally, this would be first backported upstream to the 2.4.x and we (or Debian) could cherrypick it.

Apache2 typically releases every few months but last release was in October, and the CHANGES file looks awfully full, so I'm suspecting we might see a new release. If 2.4.59 were released within a couple weeks and lands in Debian, it might be possible to make it in for the LTS release. Lacking a 2.4.59 release, cherrypicking a patch from 2.4.x is easy prior to feature freeze at the end of February. Other than that, it's less easy and I'd like to first better understand the need.

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2024-02-16 08:39 EDT-------
> I would want to better understand the use case or requirement, and how this patch solves them.

The use case is that you want to protect the private key of a httpd server by using a PKCS#11 based (HSM based) private key for the server instead of a clear key.

The private key of the server is the 'identity' of the server.
If an attacker can steal the private key of the server, he can take over that server and the clients will not notice, because the server's public key and the server certificate do not change. So even with certificate/pubic-key pinning, the client won't see any changes.

A more secure way of protecting the server key is by using a HSM based key for that. An HSM based key is a so called secure key, and it is encrypted by the HSMs master key. The HSM master key will never ever leave the HSM, and thus is physically bound to the HSM. A secure key is encrypted by the HSM master key, and any crypto operation with such a key must happen inside the HSM. So even if a secure key gets stolen, it is totally useless without having the HSM, too.

PKCS#11 is a crypto API that is often used for HSM based crypto systems.
There are different PKCS#11 implementations available, of of it is Opencryptoki, which offers true HSM based crypto operations with IBM Crypto Express adapters in CCA or EP11 mode.

Unfortunately, only very little applications do support PKCS#11 directly. Many of them (including Apache httpd's mod_ssl) do use the OpenSSL crypto API to perform (clear key) crypto operations. To allow such applications to transparently use PKCS#11, there exists a so called PKCS#11 engine (libp11 engine - https://github.com/OpenSC/libp11).

Apache httpd mod_ssl since long time has support for using this engine to work with a PKCS#11 implementation for the server key and certificate. The httpd mod_ssl has special configuration keywords to enable the engine, and to specify PKCS#11 URIs for the key and the certificate, so that it uses the PKCS#11 based key instead of a clear key.

See this article about how to configure such a setup:
https://www.ibm.com/docs/en/linux-on-z?topic=linuxone-libp11-engine

With OpenSSL 3.0 the OpenSSL engines got deprecated and OpenSSL is moving to a new plugin concept, called providers. So the libp11 engine usage will end at some point in time.

Since some time, there exist two PKCS#11 providers that can kind of replace the libp11 Engine.
PKCS#11 provider: https://github.com/latchset/pkcs11-provider
PKCS#11 sign provider: https://github.com/opencryptoki/openssl-pkcs11-sign-provider

Unfortunately, the way httpd 's mod_ssl is implemented, it need some adaptions in order to use these PKCS#11 providers instead if a PKCS#11 engine. There is a lot of engine specific code in mod_ssl, that needs to be adjusted for working with providers.

This is exactly what the patch of this feature does.

So with that support, one can configure the Apache httpd with a PKCS#11 provider to use a PKCS#11 based server key and certificate, the same way as it was possible with the libp11 engine.

Revision history for this message
Bryce Harrington (bryce) wrote : Re: [24.04 FEAT] [SEC2339] HSM protected signing support for Apache httpd for openSSL 3.0 with PKCS #11 provider

Hi ifranzki,

Thank you for the detailed description of the feature in question.

Given this is an important security feature, would it make more sense to get this backported by Apache to stable, or is your preference that we do the backport in Ubuntu? As I mentioned, this did land in the development trunk as you pointed out, but is not present on the Apache 2.4.x stable tree yet, and I'm not seeing it mentioned in their STATUS file (where they plan which patches to backport). Would love to hear your thoughts on this before handling of it is determined.

Revision history for this message
Frank Heimes (fheimes) wrote :

Since time is progressing and 24.04 FF is coming soon,
let me sum this up a bit and let me extract a few concerns/questions that arose out of this discussion:

a) In such cases Canonical/Ubuntu can take (stable) patches that are upstream accepted.
   However, in this particular case the patch(es) have landed in the 2.5.x devel trunk.
   But support for an LTS release might be granted for up to 12 years these days.
   Hence in this particular case we would need IBMs commitment on the help supporting these
   patches, and especially the ongoing push to get them into stable (which would ease the
   maintenance burden a lot - and is btw. what we usually would require).
b) Even if we talk about 'just' 4 commits (if I got things right):
    https://github.com/apache/httpd/commit/cc796e269d7c4f8d105fa46b590c9301c2a55329
    https://github.com/apache/httpd/commit/2412f20b176ff54538b67088a9e643ffed6e87ae
    https://github.com/apache/httpd/commit/d3a970420f04f9304e202bd1bdc04cbace9bbbd1
    https://github.com/apache/httpd/commit/f5bf0869c7cfd6723f53115c8483737fce53fd2a
   the changes (esp. of cc796e2) are significant.
   1) Can we be sure that no further changes that have made it over time into 2.5.x
      are needed on top (means any add. internal code dependencies) ?
   2) And has this ever be tried with 2.4.x (which would mitigate the above) ?
   3) And would you share the backported version of the commit(s) for the 2.4.x trunk?

Frank Heimes (fheimes)
description: updated
summary: - [24.04 FEAT] [SEC2339] HSM protected signing support for Apache httpd
- for openSSL 3.0 with PKCS #11 provider
+ [FFe] [24.04 FEAT] [SEC2339] HSM protected signing support for Apache
+ httpd for openSSL 3.0 with PKCS #11 provider
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2024-02-21 04:09 EDT-------
> ... would it make more sense to get this backported by Apache to stable, or is your preference that we do the backport in Ubuntu?

I am fine with both. It would certainly be good to have this is stable, since it would make things much easier. However, I have no clue how the Apache community handles such things.

I would guess that backporting these changes should not be a too huge effort, since I would guess that mod_ssl did not change much in the areas affected by the patches since quite long time. However, I have not tried to backport them.

> 1) Can we be sure that no further changes that have made it over time into 2.5.x
are needed on top (means any add. internal code dependencies) ?

I don't think so, but I am also not that deep into mod_ssl that I can answer that.

> 2) And has this ever be tried with 2.4.x (which would mitigate the above) ?

No, I have not tried.

> 3) And would you share the backported version of the commit(s) for the 2.4.x trunk?

I don't have any.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I'll take a look at this.

Changed in apache2 (Ubuntu):
assignee: nobody → Andreas Hasenack (ahasenack)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.