apache 2.4.29-1ubuntu4.12 authentication with client certificate broken

[Riho Kalbus]

Ubuntu 18.04.4 LTS, after update from apache 2.4.29-1ubuntu4.11 to apache 2.4.29-1ubuntu4.12 authentication with client certificate stopped working. No certificate is requested from client browser and apahce log has error:

[Tue Mar 03 16:03:34.964389 2020] [ssl:debug] [pid 12384:tid 139853354215168] ssl_engine_kernel.c(2217): AH02041: Protocol: TLSv1.3, Cipher: TLS_AES_256_GCM_SHA384 (256/256 bits)
[Tue Mar 03 16:03:36.499614 2020] [ssl:debug] [pid 12383:tid 139853481088768] ssl_engine_io.c(1106): AH02001: Connection closed to child 1 with standard shutdown
[Tue Mar 03 16:03:37.714744 2020] [ssl:debug] [pid 12384:tid 139853481088768] ssl_engine_kernel.c(383): AH02034: Initial (No.1) HTTPS request received for child 65 (server devel.liisi.ee:443), referer: https://devel.liisi.ee:8950/accounts/login/
[Tue Mar 03 16:03:37.714941 2020] [ssl:error] [pid 12384:tid 139853481088768] AH: verify client post handshake, referer: https://devel.liisi.ee:8950/accounts/login/

A temporary workaround is to disable the whole TLSv1.3 protocol in the vhost configuration.
---
ProblemType: Bug
Apache2ConfdDirListing: False
Apache2Modules:
 AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.20.4.138. Set the 'ServerName' directive globally to suppress this message
 httpd (pid 13567) already running
ApportVersion: 2.20.9-0ubuntu7.11
Architecture: amd64
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2010-05-21 (3576 days ago)
InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
Package: apache2 2.4.29-1ubuntu4.12
PackageArchitecture: amd64
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 4.15.0-88.88-generic 4.15.18
Tags: bionic
Uname: Linux 4.15.0-88-generic x86_64
UpgradeStatus: Upgraded to bionic on 2018-10-16 (505 days ago)
UserGroups:

_MarkForUpload: True
error.log:
 [Thu Mar 05 06:25:05.942445 2020] [ssl:warn] [pid 13567:tid 140475868056512] AH01909: klient.liisi.ee:443:0 server certificate does NOT include an ID which matches the server name
 [Thu Mar 05 06:25:05.945212 2020] [mpm_worker:notice] [pid 13567:tid 140475868056512] AH00292: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
 [Thu Mar 05 06:25:05.945234 2020] [core:notice] [pid 13567:tid 140475868056512] AH00094: Command line: '/usr/sbin/apache2'
modified.conffile..etc.apache2.mods-available.reqtimeout.conf: [modified]
modified.conffile..etc.apache2.ports.conf: [modified]
modified.conffile..etc.apache2.sites-available.000-default.conf: [modified]
mtime.conffile..etc.apache2.mods-available.reqtimeout.conf: 2020-03-03T16:33:43.294515
mtime.conffile..etc.apache2.ports.conf: 2014-10-22T16:31:31.217125
mtime.conffile..etc.apache2.sites-available.000-default.conf: 2019-10-16T13:29:08.811073

[Bryce Harrington (bryce)]

Hi Riho,

Thank you for taking the time to report this bug. I've mentioned this on bug LP: #1845263 as a possible regression related to the
2.4.29-1ubuntu4.12 update that backported the TLSv1.3 support to bionic. That update indicated some expectation that certain environments might be adversely affected when the new protocol is added, so it would be helpful to understand in more detail about your particular setup. That may help identify what went wrong precisely in this case.

Please execute the following command, as it will automatically gather debugging information, in a terminal:

  apport-collect 1865900

Alternatively, if you want to manually attach things (e.g. so you can remove any sensitive information), the files this collects includes:

/etc/apache2/apache2.conf
/etc/apache2/sites-enabled/*
/etc/apache2/conf.d
/var/log/apache2/error.log
`/usr/sbin/apachectl -D DUMP_MODULES`

Obviously the piece that'll need more examination is the client certificate configuration, so if there are other config files or logs of relevance to that you're aware of, those details could be useful as well.

[Marc Deslauriers (mdeslaur)]

This is likely a dupe of bug 1834671...

[Ken Dreyer (Red Hat) (kdreyer-redhat)]

I can confirm this as well. I have a CI job that uses python-requests to contact Apache with SSL x590 client authentication. This job passed with apache 2.4.29-1ubuntu4.11 and it fails with apache 2.4.29-1ubuntu4.12.

Passing: https://travis-ci.org/ktdreyer/koji-ansible/builds/655568368
Failing: https://travis-ci.org/ktdreyer/koji-ansible/builds/657818117

[Riho Kalbus (rihokalbus)]

apport information

[Riho Kalbus (rihokalbus)]

apport information

[Riho Kalbus (rihokalbus)]

apport information

[Riho Kalbus (rihokalbus)]

apport information

[Marc Deslauriers (mdeslaur)]

Most clients don't support post handshake authentication, hence can't use client side certificates with TLSv1.3.

In environments where client side certificates are used, TLSv1.3 has to be disabled in the Apache configuration until browsers and other clients support post handshake authentication.

[Andreas Hasenack (ahasenack)]

Bug #1834671 also has this possible workaround:
"""
Another workaround is to move the SSLVerifyClient config to the vhost level. It it applied to the whole vhost, and there are no exceptions in specific blocks, then a re-negotiation isn't triggered and the problem doesn't happen.
"""

i.e., it's the change in ssl configuration inside a vhost that triggers the PHA, from my understanding.

[Andreas Hasenack (ahasenack)]

> I can confirm this as well. I have a CI job that uses python-requests to contact
> Apache with SSL x590 client authentication. This job passed with
> apache 2.4.29-1ubuntu4.11 and it fails with apache 2.4.29-1ubuntu4.12.

Is this a case where python or python-requests could be updated to handle PHA? Just like firefox was updated in bionic to handle PHA.

[Marc Deslauriers (mdeslaur)]

Firefox in bionic added an option to handle PHA, but it's disabled by default because it conflicts with http2.

I'm not aware if there's an equivalent "fix" for python-requests.

[Marc Deslauriers (mdeslaur)]

Perhaps this: https://github.com/urllib3/urllib3/issues/1634

[Ken Dreyer (Red Hat) (kdreyer-redhat)]

From https://bugzilla.redhat.com/show_bug.cgi?id=1761403:
"The fix is available in urllib3 1.25.4. The fix requires Python 3.7.4 or newer with fix https://bugs.python.org/issue37428 ."

I upgraded urllib3 and requests to the Disco versions:

Unpacking python3-urllib3 (1.24.1-1ubuntu0.1) over (1.22-1ubuntu0.18.04.1) ...
Unpacking python3-requests (2.21.0-1) over (2.18.4-2ubuntu0.1) ...

I still get "HTTPError: 403 Client Error: Forbidden for url: https://localhost/kojihub/ssllogin" in my Bionic VM when I try those versions.

[Ken Dreyer (Red Hat) (kdreyer-redhat)]

With Bionic's apache 2.4.29-1ubuntu4.12:

"SSLProtocol TLSv1.3 TLSv1.2" - works
"SSLProtocol TLSv1.3 +TLSv1.2" - does not work
"SSLProtocol all -SSLv3" - does not work

[Andreas Hasenack (ahasenack)]

There also seems to be https://bugs.python.org/issue37440

In any case, I think this bug is not about apache, other than it's a change introduced there that made tls v1.3 available for clients to use. The clients need to be updated now.

[Ken Dreyer (Red Hat) (kdreyer-redhat)]

"SSLProtocol all -SSLv3" is in the default /etc/apache2/mods-enabled/ssl.conf. Why does the behavior change when I set "SSLProtocol TLSv1.3 TLSv1.2"?

[Andreas Hasenack (ahasenack)]

I guess depends where you change it. If you do it on a specific location or directory, it's my understanding that this is what triggers PHA.

[Riho Kalbus (rihokalbus)]

> With Bionic's apache 2.4.29-1ubuntu4.12:
>
>"SSLProtocol TLSv1.3 TLSv1.2" - works

Tried with Firefox 73.0.1 - works, but connection is established using TLS1.2 protocol
when "SSLProtocol TLSv1.3 TLSv1.2 TLSv1.1" is specified, then TLS1.1 is used.

[Robie Basak (racb)]

I think we have enough information on this report now; all that remains is some difficult decision making on what, if anything, we can do about it. Depending on the answer, we might need to assign this bug to a different package, etc.

[Christian Ehrhardt (paelzer)]

FYI there is a similar bug 1867223 which has a patch suggested at least for some cases of this.

[Marc Deslauriers (mdeslaur)]

I have uploaded an apache2 package to the security team PPA for testing here:

https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa

It includes a few fixes related to TLSv1.3.

Could environment having this issue please test that package and see if it solves the issue?

Thanks!

[Christian Ehrhardt (paelzer)]

It is quite likely that the changes fixes some, but not all of the cases so having more than one feedback for Marc's call for testing in comment #21 would be great.

[Christian Ehrhardt (paelzer)]

Lets sort out apache first for now, but related to this (might eventually be split into a different bug) the clients in Bionic need to be PHA compatible as more and mroe of the world will grow TLS v1.3.

I added tasks for src:python-urllib3 and src:requests to remind us to think about those eventually.

[Riho Kalbus (rihokalbus)]

Hello,

tested. Issue was not solved, but got relevant error message: "You don't
have permission to access this resource.Reason: Cannot perform
Post-Handshake Authentication."

ii apache2 2.4.29-1ubuntu4.13
                   amd64 Apache HTTP Server
ii apache2-bin 2.4.29-1ubuntu4.13
                   amd64 Apache HTTP Server (modules and other
binary files)
ii apache2-data 2.4.29-1ubuntu4.13
                   all Apache HTTP Server (common files)
ii apache2-utils 2.4.29-1ubuntu4.13
                   amd64 Apache HTTP Server (utility programs for
web servers)
ii libapache2-mod-wsgi-py3 4.5.17-1ubuntu1
                  amd64 Python 3 WSGI adapter module for Apache
ii libssl1.1:amd64 1.1.1-1ubuntu2.1~18.04.5
                   amd64 Secure Sockets Layer toolkit - shared
libraries

[Tue Mar 17 09:44:14.919351 2020] [mpm_worker:notice] [pid 5259:tid
140138557897664] AH00292: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1
mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Tue Mar 17 09:44:14.919385 2020] [core:notice] [pid 5259:tid
140138557897664] AH00094: Command line: '/usr/sbin/apache2'
[Tue Mar 17 09:45:49.236283 2020] [ssl:error] [pid 5704:tid
140138323629824] [client 80.235.25.20:15540] AH: verify client post
handshake, referer: https://devel.liisi.ee:8950/accounts/login/
[Tue Mar 17 09:45:49.236315 2020] [ssl:error] [pid 5704:tid
140138323629824] [client 80.235.25.20:15540] AH10158: cannot perform
post-handshake authentication, referer:
https://devel.liisi.ee:8950/accounts/login/
[Tue Mar 17 09:45:49.236336 2020] [ssl:error] [pid 5704:tid
140138323629824] SSL Library Error: error:14268117:SSL
routines:SSL_verify_client_post_handshake:extension not received

Kontakt Marc Deslauriers (<email address hidden>) kirjutas
kuupäeval E, 16. märts 2020 kell 15:15:

> I have uploaded an apache2 package to the security team PPA for testing
> here:
>
> https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa
>
> It includes a few fixes related to TLSv1.3.
>
> Could environment having this issue please test that package and see if
> it solves the issue?
>
> Thanks!
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1865900
>
> Title:
> apache 2.4.29-1ubuntu4.12 authentication with client certificate
> broken
>
> Status in apache2 package in Ubuntu:
> New
>
> Bug description:
> Ubuntu 18.04.4 LTS, after update from apache 2.4.29-1ubuntu4.11 to
> apache 2.4.29-1ubuntu4.12 authentication with client certificate
> stopped working. No certificate is requested from client browser and
> apahce log has error:
>
> [Tue Mar 03 16:03:34.964389 2020] [ssl:debug] [pid 12384:tid
> 139853354215168] ssl_engine_kernel.c(2217): AH02041: Protocol: TLSv1.3,
> Cipher: TLS_AES_256_GCM_SHA384 (256/256 bits)
> [Tue Mar 03 16:03:36.499614 2020] [ssl:debug] [pid 12383:tid
> 139853481088768] ssl_engine_io.c(1106): AH02001: Connection closed to ...

[Marc Deslauriers (mdeslaur)]

Thanks for the test. That does in fact look like the Apache side of things is now fixed as you are getting the appropriate error message when the client support is missing, which wasn't happening before.

[Vladimir Mencl (vladimir-mencl)]

Hi,

I'm afraid the fix released in 2.4.29-1ubuntu4.13 has introduced a regression.

We have just updated our servers to 2.4.29-1ubuntu4.13 and configuration that was working previously suddenly broke.

We are using
   SSLVerifyClient optional
inside a Location element.

Our configuration has:

    SSLCACertificateFile "/etc/ssl/certs/api-ca.crt"
    <Location /api>
        SSLVerifyClient optional
        RequestHeader set X509_DN "%{SSL_CLIENT_S_DN}s"
    </Location>

However, this breaks with:

[Wed Mar 25 16:08:02.648354 2020] [ssl:error] [pid 1801:tid 140236923303680] [client 2404:138:46::126:47888] AH: verify client post handshake
[Wed Mar 25 16:08:02.648403 2020] [ssl:error] [pid 1801:tid 140236923303680] [client 2404:138:46::126:47888] AH10158: cannot perform post-handshake authentication
[Wed Mar 25 16:08:02.648420 2020] [ssl:error] [pid 1801:tid 140236923303680] SSL Library Error: error:14268117:SSL routines:SSL_verify_client_post_handshake:extension not received

Removing the SSLVerifyClient optional or disabling TLSv1.3 fixes it ... but both would be deviating from our desired target configuration.

Hope this can be fixed.

Please let me know if you need any further info - or if this should be a standalone bug report.
(So far, as this is a regression caused by the fix discussed here, I thought best to post here.

Cheers,
Vlad

[Vladimir Mencl (vladimir-mencl)]

Hi,

Just clarifying on the previous comment. From the release notes I've seen in the bionic package, I understand this fix does:
> - debian/patches/tlsv1.3-support-3.patch: fail with 403 if
> SSL_verify_client_post_handshake fails in
> modules/ssl/ssl_engine_kernel.c.

However, when authentication is optional (SSLVerifyClient optional) and no client authentication is provided, it MUST NOT count as a failure and request processing should continue...

Cheers,
Vlad

[Vladimir Mencl (vladimir-mencl)]

FYI, I have just checked upstream's code-base and submitted this as a bug to upstream:

https://bz.apache.org/bugzilla/show_bug.cgi?id=64263

[Marc Deslauriers (mdeslaur)]

Thanks for reporting the regression. What client are you using to access the web server?

[Marc Deslauriers (mdeslaur)]

@vladimir-mencl: what you are seeing is actually this bug:
https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1834671

Basically, with TLSv1.3 you need a client that supports post-handshake authentication.

Some clients, such as Firefox for example, support it but it needs to be enabled, as it's disabled by default, see security.tls.enable_post_handshake_auth in about:config.

The best course of action if you don't control the clients connecting to your web server is probably to disable TLSv1.3.

[Vladimir Mencl (vladimir-mencl)]

Hi Marc,

Thanks for getting back to me.

I've been testing this with `wget` and `curl`. And it worked before 2.4.29-1ubuntu4.13 (with 2.4.29-1ubuntu4.12), even with TLSv1.3.

Note that this particular use case, I actually don't need (or want) the clients to authenticate.

I just want the server to *offer* authentication when accessing a particular URL (/api) - with "SSLVerifyClient optional".

Some API calls are authenticated, some unauthenticated. The web application behind Apache would check whether authentication is provided based on the actual call invoked.

And the clients that are breaking now are clients that would just call unauthenticated APIs and would not authenticate.

So as per my earlier post, this is an omission in the patch applied from upstream (tlsv1.3-support-3.patch) - which fails with HTTP_FORBIDDEN when authentication is not provided, forgetting to check if it was optional.

I hope I've now explained properly what I mean by the regression - please let me know if this needs any further clarification.

I have checked upstream SVN history and there is no subsequent change to ssl_engine_kernel.c that would be fixing this - not even in trunk.

So at this point, there are no further fixes to backport and this needs to be fixed upstream.

I hope my report upstream - https://bz.apache.org/bugzilla/show_bug.cgi?id=64263 - will get this sorted.

Cheers,
Vlad

[Marc Deslauriers (mdeslaur)]

I understand your reasoning, but as I understand the issue, with TLSv1.2 renegotiation was used to see if the client can provide a certificate or not, but TLSv1.3 doesn't support renegotiation, so post-handshake authentication must be used.

Thanks for opening the upstream bug, let's see what they say about it, but I suspect it's going to ultimately be a duplicate of one of the other ones, for example: https://bz.apache.org/bugzilla/show_bug.cgi?id=63368

I will, of course, update our package if upstream provides a different fix for this issue.

[Vladimir Mencl (vladimir-mencl)]

Hi Marc,

Thanks for the reply!

I have now done more extensive testing (incl. rebuilding apache2-2.4.29-1ubuntu4.12 from source).

I now understand that for essentially all HTTPS clients,
it is necessary to update SSL API calls to support TLSv1.3
post-handshake authentication.

And I have also checked with a version of curl built right off the
top of the github repo (7.70.0-DEV) - as an example of a client
capable of post-handshake authentication.

With this version of curl, both apache2-2.4.29-1ubuntu4.12 and apache2-2.4.29-1ubuntu4.13 work over TLSv1.3 for both authenticated and unauthenticated API.

But older clients (not capable of post-handshake authentication), including curl included with Ubuntu 18.04 (7.58.0) do not work with the authenticated API with neither apache2-2.4.29-1ubuntu4.12 and apache2-2.4.29-1ubuntu4.13.

The only edge-case is my use case of unauthenticated API - that used to work with the older clients (not capable of post-handshake authentication) on apache2-2.4.29-1ubuntu4.12, but breaks with apache2-2.4.29-1ubuntu4.13 (for the older clients only).

I'll add these findings to my upstream report.

I agree the main point is updating all clients to support TLSv1.3 properly, including post-handshake authentication - the question is whether to let older clients get by when authentication is not required.

Let's see what I get upstream.

Cheers,
Vlad

[Ken Dreyer (Red Hat) (kdreyer-redhat)]

Focal's curl 7.68.0-1ubuntu2.6 now supports post-handshake client authentication.

curl \
  --tls-max 1.2 \
  --cacert ~/.koji/pki/koji-ca.crt \
  --cert ~/koji-ansible/koji-tools/admin.crt \
  --key ~/koji-ansible/koji-tools/admin.key \
  https://localhost/kojihub/ssllogin

However, python-requests on Focal or Groovy still cannot do post-handshake client auth with the default SSLProtocol setting. Sample Python script:

  import requests

  r = requests.get('https://localhost/kojihub/ssllogin',
                   verify='.koji/pki/koji-ca.crt',
                   cert=('koji-ansible/koji-tools/admin.crt',
                         'koji-ansible/koji-tools/admin.key'))
  r.raise_for_status()

 The Apache error logs say:

  [ssl:error] AH10158: cannot perform post-handshake authentication
  [ssl:error] SSL Library Error: error:14268117:SSL routines:SSL_verify_client_post_handshake:extension not received

Applying https://github.com/psf/requests/commit/db47b9b4a0c5877fb97f64ac442757604c4c45cc or updating to hirsute's python3-requests_2.25.1+dfsg-2 does work.

[Christian Ehrhardt (paelzer)]

I came by this bug and tried to update the states to better reflect the situation.

We still have the case that for Focal there was a regression due to updates. But also the problem that it is hard to see if we want to revert (no as it was security improvements) or how we could fix (is it actually feature add to become compatible) things.

The update to the states show that for new releases e.g. Jammy all the fixes are in and in addition being a new releases it is kind of allowed to do new things differently.

But at the same time that means there might be fixes existing by now that make the other components compatible to the new behavior - if those could be backported that would mitigate the issue. But OTOH as we know backporting features usually isn't done in SRUs.

That evaluation will need time - Marc was assigned to this before for a revisit, and I have kept this assignment in my update.