Ubuntu 18.04: slow page loads with client cert auth after upgrade to openssl 1.1.1

[Andreas Amann]

After upgrade to openssl 1.1.1 on Ubuntu 18.04 i encountered slow page loads (>15 sec delay for each GET) when client cert auth is used.

Apache logs show delays like this (LogLevel debug):

[Fri Jun 21 11:36:13.760861 2019] [socache_shmcb:debug] [pid 14032] mod_socache_shmcb.c(516): AH00834: leaving socache_shmcb_store successfully
[Fri Jun 21 11:36:30.229486 2019] [authz_core:debug] [pid 14032] mod_authz_core.c(835): [client XXX.XXX.XXX.XXX:XXXX] AH01628: authorization result: granted (no directives)

This appears to be a problem in apache that has been triggered with Openssl 1.1.1 and was fixed in 2.4.34, see

https://bz.apache.org/bugzilla/show_bug.cgi?id=62691

The workaround mentioned there worked for me, so after moving the "SSLVerifyClient require" part out of an LocationMatch block into the containing VirtualHost stopped the delays instantly.

Thanks a lot!
Andreas

Ubuntu 18.04
apache2 2.4.29-1ubuntu4.6
openssl 1.1.1-1ubuntu2.1~18.04.3

[Wmperry (wmperry)]

Working on a project and need an LTS version of OpenSSL, which is soon to be 1.1.1. Recompiled apache 2.4.34 against it and trying to use client certificates shows a 1 minute delay between the handshake completing and the 0-byte SSL_peek() returning in ssl_engine_kernel.c:1033 returns.

Working just fine with OpenSSL 1.1.0 or 1.0.2, also appears to work without a delay using the 1.1.1 openssl s_server command.

Minimal configuration file is:

------------
LoadModule ssl_module /usr/lib/apache2/modules/mod_ssl.so
LoadModule mpm_event_module /usr/lib/apache2/modules/mod_mpm_event.so
LoadModule unixd_module /usr/lib/apache2/modules/mod_unixd.so
LoadModule mime_module /usr/lib/apache2/modules/mod_mime.so
LoadModule authz_core_module /usr/lib/apache2/modules/mod_authz_core.so

SSLPassPhraseDialog "exec:......"

LogLevel trace5
ErrorLog /tmp/client-certificates.log
ServerName my.test.com
DocumentRoot /var/www

<Location />
   SSLVerifyClient require
   Require ssl-verify-client
</Location>

Listen 1443
<VirtualHost *:1443>
    SSLEngine on
    SSLCertificateKeyFile "/etc/xxxx.key"
    SSLCertificateFile "/etc/xxxx.cert"
    SSLCertificateChainFile "/etc/xxxx.cert"
    SSLCACertificateFile "/etc/backendca.cert"
</VirtualHost>
------------

The logs from apache itself that shows the delay are:

[Tue Sep 04 18:58:14.886205 2018] [ssl:debug] [pid 2571:tid 140532252661504] ssl_engine_kernel.c(2082): [client 172.16.1.101:53414] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
---- delay here ----
[Tue Sep 04 18:59:14.944591 2018] [ssl:trace4] [pid 2571:tid 140532252661504] ssl_engine_io.c(2204): [client 172.16.1.101:53414] OpenSSL: read 0/5 bytes from BIO#7fd014002a10 [mem: 7fd014002d43] (BIO dump follows)

There is no delay before the web browser / client prompts for a certificate to use - just between the ssl re-handshake completing and the peek() returning no bytes.

[Steffn (steffn)]

We are currently adding support for TLSv1.3 for the next 2.4.x release. If you like, you can test your setup against

https://svn.apache.org/repos/asf/httpd/httpd/branches/tlsv1.3-for-2.4.x

to see if this delay has been eliminated by the recent changes in renegotiation/client cert handling. Would be good to know. Thanks!

[Wmperry (wmperry)]

Created attachment 36161
Trace5 logs of the renegotiation

I downloaded the latest on that branch as of a few days ago (, and the behaviour is still the same. Full renegotiation is done, certificate is verified, and then a timeout of 1 minute while the SSL_peek() fails. Attaching logs @ trace5

[Wmperry (wmperry)]

Status back to 'new' - please let me know if you need any additional info.

[Rainer Jung (rainer-jung-kippdata)]

Thanks for the logs. I can't currently promise to work on it myself, but some info:

- the branch tlsv1.3-for-2.4.x has been merged just 3 days ago into the normal 2.4.x branch. So any further tests can be done against the normal branch, unless something needs to get tested which might only exist in trunk, in which case that would b the right source. The tlsv1.3-for-2.4.x should probably considered stale starting now.

- what was the client you tested with?

- just to make sure: the hang only happens when "RE"negotation occurs due to the VHost having different TLS config than the URI on that VHost that you are trying to access (eg. the access to the VHost is not protected by client certs, but some URLs are). To phrase it differently: when all of the vhost uses the same ssl config including client certs, ciphers etc., then no hang occurs?

- As far as I understand this situation, previously handled by renegotiation, gets handled in OpenSSL 1.1.1 by a post-handshake-authentication (PHA) extension and support for PHA is not yet clear for all clients. Especially clients using OpenSSL as their client TLS stack need to explicitly turn on PHA when using OpenSSL 1.1.1. But that's only what I think I understood from other discussions.

Thanks and regards,

Rainer

[Wmperry (wmperry)]

Moving "SSLVerifyClient require" outside of the <Location> block instantly returns the document. So it does appear to be ONLY the renegotiation case.

[Jorton-9 (jorton-9)]

Note this is testing with TLSv1.2 not 1.3 - I will try to reproduce as well.

Can you stick "#warning foo" on a line directly before:

    SSL_CTX_clear_mode(ctx, SSL_MODE_AUTO_RETRY);

in ssl_engine_init.c to make sure that is being enabled when you build? Preferable to test against the 2.4.x branch now.

[Jorton-9 (jorton-9)]

I can't reproduce with 2.4.34+the TLS v1.3 patch from the branch, linked against Fedora 29's OpenSSL 1.1.1.

Can you also test with the OpenSSL 1.1.1 final release - the original post was dated before 1.1.1 was tagged - so I guess you were you testing with a -pre release?

[Wmperry (wmperry)]

Using the released OpenSSL 1.1.1 and the latest from 2.4.x seems to work without any noticeable delay. Guess this one can go to resolved.

I was using 1.1.1-pre9. thanks.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Tilman Sandig (ssp297)]

Yesterday (2019-06-24) I updated Ubuntu 18.04 and got OpenSSL 1.1.1; after that, every GET-request with a client certificate causes a delay > 15sec.
I use the source-download of apache 2.4.39 (the latest version), TLS 1.3 is not configured.
Because only access to parts of the virtual host should be protected with

  SSLVerifyClient require

I think I have to use it in a LocationMatch block like

    <Location /mylocation>
            SSLVerifyClient require
            SSLVerifyDepth 1
            SSLOptions +StdEnvVars +ExportCertData +OptRenegotiate
    </Location>

Behind apache, tomcat 8.5 is used via AJP.

I tried to downgrade OpenSSL to the prior version (with no errors), but the apache log still shows using OpenSSL 1.1.1 (?) and the delay remains.

Because this is a production server, any quick help/workaround is appreciated...

[Tilman Sandig (ssp297)]

Have solved my problem by simply rebuilding the apache 2.4.39 from same sources.

[J Manzano (jmanzano)]

I'm seeing this problem on my production environment. I'm dependent on TLS renegotiation. Downgrading OpenSSL to 1.1.0 solves the problem. Upgrading both OpenSSL to 1.1.1 and Apache to 2.4.39 and disabling TLSv1.3 solves it too.

I'm using mod_reqtimeout to let users authenticate with client certificates stored in slower smart cards.

I'm using this mod_reqtimeout configuration:

RequestReadTimeout header=10 body=30

I'm seeing that the wait between the client certificate selection and the authentication takes effect, is equal to the body timeout setting.

Hope it helps throwing some light at the issue.

[Paride Legovini (paride)]

Thanks for reporting this issue and your findings. I believe this report is a duplicate of this other bug, filed just a few days earlier:

  https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1833039

As it is easier to have all the discussion in one place I'm marking this bug as a duplicate. If you believe this is not the case please explain why you believe this is a different issue and remove the duplicate marking. Thank you!