apache 2.4.29-1ubuntu4.12 authentication with client certificate broken
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| Release Notes for Ubuntu |
Undecided
|
Unassigned | ||
| apache2 (Ubuntu) |
Undecided
|
Marc Deslauriers | ||
| python-urllib3 (Ubuntu) |
Undecided
|
Unassigned | ||
| requests (Ubuntu) |
Undecided
|
Unassigned |
Bug Description
Ubuntu 18.04.4 LTS, after update from apache 2.4.29-1ubuntu4.11 to apache 2.4.29-1ubuntu4.12 authentication with client certificate stopped working. No certificate is requested from client browser and apahce log has error:
[Tue Mar 03 16:03:34.964389 2020] [ssl:debug] [pid 12384:tid 139853354215168] ssl_engine_
[Tue Mar 03 16:03:36.499614 2020] [ssl:debug] [pid 12383:tid 139853481088768] ssl_engine_
[Tue Mar 03 16:03:37.714744 2020] [ssl:debug] [pid 12384:tid 139853481088768] ssl_engine_
[Tue Mar 03 16:03:37.714941 2020] [ssl:error] [pid 12384:tid 139853481088768] AH: verify client post handshake, referer: https:/
A temporary workaround is to disable the whole TLSv1.3 protocol in the vhost configuration.
---
ProblemType: Bug
Apache2ConfdDir
Apache2Modules:
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.20.4.138. Set the 'ServerName' directive globally to suppress this message
httpd (pid 13567) already running
ApportVersion: 2.20.9-0ubuntu7.11
Architecture: amd64
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2010-05-21 (3576 days ago)
InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
Package: apache2 2.4.29-1ubuntu4.12
PackageArchitec
ProcEnviron:
TERM=xterm-
PATH=(custom, no user)
XDG_RUNTIME_
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcVersionSign
Tags: bionic
Uname: Linux 4.15.0-88-generic x86_64
UpgradeStatus: Upgraded to bionic on 2018-10-16 (505 days ago)
UserGroups:
_MarkForUpload: True
error.log:
[Thu Mar 05 06:25:05.942445 2020] [ssl:warn] [pid 13567:tid 140475868056512] AH01909: klient.
[Thu Mar 05 06:25:05.945212 2020] [mpm_worker:notice] [pid 13567:tid 140475868056512] AH00292: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Thu Mar 05 06:25:05.945234 2020] [core:notice] [pid 13567:tid 140475868056512] AH00094: Command line: '/usr/sbin/apache2'
modified.
modified.
modified.
mtime.conffile.
mtime.conffile.
mtime.conffile.
Bryce Harrington (bryce) wrote : | #1 |
Changed in apache2 (Ubuntu): | |
status: | New → Incomplete |
tags: | added: regression-update |
Marc Deslauriers (mdeslaur) wrote : | #2 |
This is likely a dupe of bug 1834671...
I can confirm this as well. I have a CI job that uses python-requests to contact Apache with SSL x590 client authentication. This job passed with apache 2.4.29-1ubuntu4.11 and it fails with apache 2.4.29-1ubuntu4.12.
Passing: https:/
Failing: https:/
apport information
tags: | added: apport-collected bionic |
description: | updated |
apport information
apport information
apport information
Marc Deslauriers (mdeslaur) wrote : | #8 |
Most clients don't support post handshake authentication, hence can't use client side certificates with TLSv1.3.
In environments where client side certificates are used, TLSv1.3 has to be disabled in the Apache configuration until browsers and other clients support post handshake authentication.
Andreas Hasenack (ahasenack) wrote : | #9 |
Bug #1834671 also has this possible workaround:
"""
Another workaround is to move the SSLVerifyClient config to the vhost level. It it applied to the whole vhost, and there are no exceptions in specific blocks, then a re-negotiation isn't triggered and the problem doesn't happen.
"""
i.e., it's the change in ssl configuration inside a vhost that triggers the PHA, from my understanding.
Andreas Hasenack (ahasenack) wrote : | #10 |
> I can confirm this as well. I have a CI job that uses python-requests to contact
> Apache with SSL x590 client authentication. This job passed with
> apache 2.4.29-1ubuntu4.11 and it fails with apache 2.4.29-1ubuntu4.12.
Is this a case where python or python-requests could be updated to handle PHA? Just like firefox was updated in bionic to handle PHA.
tags: | added: bionic-openssl-1.1 |
Marc Deslauriers (mdeslaur) wrote : | #11 |
Firefox in bionic added an option to handle PHA, but it's disabled by default because it conflicts with http2.
I'm not aware if there's an equivalent "fix" for python-requests.
Marc Deslauriers (mdeslaur) wrote : | #12 |
Perhaps this: https:/
From https:/
"The fix is available in urllib3 1.25.4. The fix requires Python 3.7.4 or newer with fix https:/
I upgraded urllib3 and requests to the Disco versions:
Unpacking python3-urllib3 (1.24.1-1ubuntu0.1) over (1.22-1ubuntu0.
Unpacking python3-requests (2.21.0-1) over (2.18.4-2ubuntu0.1) ...
I still get "HTTPError: 403 Client Error: Forbidden for url: https:/
With Bionic's apache 2.4.29-1ubuntu4.12:
"SSLProtocol TLSv1.3 TLSv1.2" - works
"SSLProtocol TLSv1.3 +TLSv1.2" - does not work
"SSLProtocol all -SSLv3" - does not work
Andreas Hasenack (ahasenack) wrote : | #15 |
There also seems to be https:/
In any case, I think this bug is not about apache, other than it's a change introduced there that made tls v1.3 available for clients to use. The clients need to be updated now.
"SSLProtocol all -SSLv3" is in the default /etc/apache2/
Andreas Hasenack (ahasenack) wrote : | #17 |
I guess depends where you change it. If you do it on a specific location or directory, it's my understanding that this is what triggers PHA.
Riho Kalbus (rihokalbus) wrote : | #18 |
> With Bionic's apache 2.4.29-1ubuntu4.12:
>
>"SSLProtocol TLSv1.3 TLSv1.2" - works
Tried with Firefox 73.0.1 - works, but connection is established using TLS1.2 protocol
when "SSLProtocol TLSv1.3 TLSv1.2 TLSv1.1" is specified, then TLS1.1 is used.
tags: | added: server-next |
Robie Basak (racb) wrote : | #19 |
I think we have enough information on this report now; all that remains is some difficult decision making on what, if anything, we can do about it. Depending on the answer, we might need to assign this bug to a different package, etc.
Changed in apache2 (Ubuntu): | |
status: | Incomplete → New |
Christian Ehrhardt (paelzer) wrote : | #20 |
FYI there is a similar bug 1867223 which has a patch suggested at least for some cases of this.
Marc Deslauriers (mdeslaur) wrote : | #21 |
I have uploaded an apache2 package to the security team PPA for testing here:
https:/
It includes a few fixes related to TLSv1.3.
Could environment having this issue please test that package and see if it solves the issue?
Thanks!
Changed in apache2 (Ubuntu): | |
status: | New → In Progress |
assignee: | nobody → Marc Deslauriers (mdeslaur) |
Christian Ehrhardt (paelzer) wrote : | #22 |
It is quite likely that the changes fixes some, but not all of the cases so having more than one feedback for Marc's call for testing in comment #21 would be great.
Changed in requests (Ubuntu): | |
status: | New → Confirmed |
Changed in python-urllib3 (Ubuntu): | |
status: | New → Confirmed |
Changed in ubuntu-release-notes: | |
status: | New → Confirmed |
Christian Ehrhardt (paelzer) wrote : | #23 |
Lets sort out apache first for now, but related to this (might eventually be split into a different bug) the clients in Bionic need to be PHA compatible as more and mroe of the world will grow TLS v1.3.
I added tasks for src:python-urllib3 and src:requests to remind us to think about those eventually.
Riho Kalbus (rihokalbus) wrote : Re: [Bug 1865900] Re: apache 2.4.29-1ubuntu4.12 authentication with client certificate broken | #24 |
Hello,
tested. Issue was not solved, but got relevant error message: "You don't
have permission to access this resource.Reason: Cannot perform
Post-Handshake Authentication."
ii apache2 2.4.29-1ubuntu4.13
ii apache2-bin 2.4.29-1ubuntu4.13
binary files)
ii apache2-data 2.4.29-1ubuntu4.13
ii apache2-utils 2.4.29-1ubuntu4.13
web servers)
ii libapache2-
ii libssl1.1:amd64 1.1.1-1ubuntu2.
libraries
[Tue Mar 17 09:44:14.919351 2020] [mpm_worker:notice] [pid 5259:tid
140138557897664] AH00292: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1
mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Tue Mar 17 09:44:14.919385 2020] [core:notice] [pid 5259:tid
140138557897664] AH00094: Command line: '/usr/sbin/apache2'
[Tue Mar 17 09:45:49.236283 2020] [ssl:error] [pid 5704:tid
140138323629824] [client 80.235.25.20:15540] AH: verify client post
handshake, referer: https:/
[Tue Mar 17 09:45:49.236315 2020] [ssl:error] [pid 5704:tid
140138323629824] [client 80.235.25.20:15540] AH10158: cannot perform
post-handshake authentication, referer:
https:/
[Tue Mar 17 09:45:49.236336 2020] [ssl:error] [pid 5704:tid
140138323629824] SSL Library Error: error:14268117:SSL
routines:
Kontakt Marc Deslauriers (<email address hidden>) kirjutas
kuupäeval E, 16. märts 2020 kell 15:15:
> I have uploaded an apache2 package to the security team PPA for testing
> here:
>
> https:/
>
> It includes a few fixes related to TLSv1.3.
>
> Could environment having this issue please test that package and see if
> it solves the issue?
>
> Thanks!
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> apache 2.4.29-1ubuntu4.12 authentication with client certificate
> broken
>
> Status in apache2 package in Ubuntu:
> New
>
> Bug description:
> Ubuntu 18.04.4 LTS, after update from apache 2.4.29-1ubuntu4.11 to
> apache 2.4.29-1ubuntu4.12 authentication with client certificate
> stopped working. No certificate is requested from client browser and
> apahce log has error:
>
> [Tue Mar 03 16:03:34.964389 2020] [ssl:debug] [pid 12384:tid
> 139853354215168] ssl_engine_
> Cipher: TLS_AES_
> [Tue Mar 03 16:03:36.499614 2020] [ssl:debug] [pid 12383:tid
> 139853481088768] ssl_engine_
Marc Deslauriers (mdeslaur) wrote : | #25 |
Thanks for the test. That does in fact look like the Apache side of things is now fixed as you are getting the appropriate error message when the client support is missing, which wasn't happening before.
Vladimir Mencl (vladimir-mencl) wrote : | #26 |
Hi,
I'm afraid the fix released in 2.4.29-1ubuntu4.13 has introduced a regression.
We have just updated our servers to 2.4.29-1ubuntu4.13 and configuration that was working previously suddenly broke.
We are using
SSLVerifyClient optional
inside a Location element.
Our configuration has:
SSLCACertif
<Location /api>
</Location>
However, this breaks with:
[Wed Mar 25 16:08:02.648354 2020] [ssl:error] [pid 1801:tid 140236923303680] [client 2404:138:
[Wed Mar 25 16:08:02.648403 2020] [ssl:error] [pid 1801:tid 140236923303680] [client 2404:138:
[Wed Mar 25 16:08:02.648420 2020] [ssl:error] [pid 1801:tid 140236923303680] SSL Library Error: error:14268117:SSL routines:
Removing the SSLVerifyClient optional or disabling TLSv1.3 fixes it ... but both would be deviating from our desired target configuration.
Hope this can be fixed.
Please let me know if you need any further info - or if this should be a standalone bug report.
(So far, as this is a regression caused by the fix discussed here, I thought best to post here.
Cheers,
Vlad
Vladimir Mencl (vladimir-mencl) wrote : | #27 |
Hi,
Just clarifying on the previous comment. From the release notes I've seen in the bionic package, I understand this fix does:
> - debian/
> SSL_verify_
> modules/
However, when authentication is optional (SSLVerifyClient optional) and no client authentication is provided, it MUST NOT count as a failure and request processing should continue...
Cheers,
Vlad
Vladimir Mencl (vladimir-mencl) wrote : | #28 |
FYI, I have just checked upstream's code-base and submitted this as a bug to upstream:
Marc Deslauriers (mdeslaur) wrote : | #29 |
Thanks for reporting the regression. What client are you using to access the web server?
Marc Deslauriers (mdeslaur) wrote : | #30 |
@vladimir-mencl: what you are seeing is actually this bug:
https:/
Basically, with TLSv1.3 you need a client that supports post-handshake authentication.
Some clients, such as Firefox for example, support it but it needs to be enabled, as it's disabled by default, see security.
The best course of action if you don't control the clients connecting to your web server is probably to disable TLSv1.3.
Vladimir Mencl (vladimir-mencl) wrote : | #31 |
Hi Marc,
Thanks for getting back to me.
I've been testing this with `wget` and `curl`. And it worked before 2.4.29-1ubuntu4.13 (with 2.4.29-
Note that this particular use case, I actually don't need (or want) the clients to authenticate.
I just want the server to *offer* authentication when accessing a particular URL (/api) - with "SSLVerifyClient optional".
Some API calls are authenticated, some unauthenticated. The web application behind Apache would check whether authentication is provided based on the actual call invoked.
And the clients that are breaking now are clients that would just call unauthenticated APIs and would not authenticate.
So as per my earlier post, this is an omission in the patch applied from upstream (tlsv1.
I hope I've now explained properly what I mean by the regression - please let me know if this needs any further clarification.
I have checked upstream SVN history and there is no subsequent change to ssl_engine_kernel.c that would be fixing this - not even in trunk.
So at this point, there are no further fixes to backport and this needs to be fixed upstream.
I hope my report upstream - https:/
Cheers,
Vlad
Marc Deslauriers (mdeslaur) wrote : | #32 |
I understand your reasoning, but as I understand the issue, with TLSv1.2 renegotiation was used to see if the client can provide a certificate or not, but TLSv1.3 doesn't support renegotiation, so post-handshake authentication must be used.
Thanks for opening the upstream bug, let's see what they say about it, but I suspect it's going to ultimately be a duplicate of one of the other ones, for example: https:/
I will, of course, update our package if upstream provides a different fix for this issue.
Vladimir Mencl (vladimir-mencl) wrote : | #33 |
Hi Marc,
Thanks for the reply!
I have now done more extensive testing (incl. rebuilding apache2-
I now understand that for essentially all HTTPS clients,
it is necessary to update SSL API calls to support TLSv1.3
post-handshake authentication.
And I have also checked with a version of curl built right off the
top of the github repo (7.70.0-DEV) - as an example of a client
capable of post-handshake authentication.
With this version of curl, both apache2-
But older clients (not capable of post-handshake authentication), including curl included with Ubuntu 18.04 (7.58.0) do not work with the authenticated API with neither apache2-
The only edge-case is my use case of unauthenticated API - that used to work with the older clients (not capable of post-handshake authentication) on apache2-
I'll add these findings to my upstream report.
I agree the main point is updating all clients to support TLSv1.3 properly, including post-handshake authentication - the question is whether to let older clients get by when authentication is not required.
Let's see what I get upstream.
Cheers,
Vlad
tags: | removed: server-next |
Hi Riho,
Thank you for taking the time to report this bug. I've mentioned this on bug LP: #1845263 as a possible regression related to the
2.4.29-1ubuntu4.12 update that backported the TLSv1.3 support to bionic. That update indicated some expectation that certain environments might be adversely affected when the new protocol is added, so it would be helpful to understand in more detail about your particular setup. That may help identify what went wrong precisely in this case.
Please execute the following command, as it will automatically gather debugging information, in a terminal:
apport-collect 1865900
Alternatively, if you want to manually attach things (e.g. so you can remove any sensitive information), the files this collects includes:
/etc/apache2/ apache2. conf sites-enabled/ * apache2/ error.log apachectl -D DUMP_MODULES`
/etc/apache2/
/etc/apache2/conf.d
/var/log/
`/usr/sbin/
Obviously the piece that'll need more examination is the client certificate configuration, so if there are other config files or logs of relevance to that you're aware of, those details could be useful as well.