Activity log for bug #1833039

Date Who What changed Old value New value Message
2019-06-17 08:02:21 Benjamin Schneider bug added bug
2019-06-19 17:59:33 Andreas Hasenack bug added subscriber Andreas Hasenack
2019-06-25 08:48:57 Christian Ehrhardt  bug task added openssl (Ubuntu)
2019-06-25 08:49:12 Christian Ehrhardt  summary 18.04/Apache2: rejecting client initiated renegotiation 18.04/Apache2: rejecting client initiated renegotiation due to openssl 1.1.1
2019-06-25 08:49:18 Christian Ehrhardt  bug added subscriber Dimitri John Ledkov
2019-06-25 08:49:24 Christian Ehrhardt  bug added subscriber Christian Ehrhardt 
2019-06-25 12:51:37 Launchpad Janitor apache2 (Ubuntu): status New Confirmed
2019-06-25 12:51:37 Launchpad Janitor openssl (Ubuntu): status New Confirmed
2019-06-25 23:09:12 Brian Murray tags regression-update rls-bb-incoming
2019-06-27 15:31:33 Paride Legovini bug added subscriber Paride Legovini
2019-06-28 13:01:38 Petter A. Urkedal bug added subscriber Petter A. Urkedal
2019-06-28 13:59:47 Andreas Hasenack bug watch added https://bz.apache.org/bugzilla/show_bug.cgi?id=62691
2019-06-28 16:42:35 Andreas Hasenack nominated for series Ubuntu Cosmic
2019-06-28 16:42:35 Andreas Hasenack bug task added apache2 (Ubuntu Cosmic)
2019-06-28 16:42:35 Andreas Hasenack bug task added openssl (Ubuntu Cosmic)
2019-06-28 16:42:35 Andreas Hasenack nominated for series Ubuntu Bionic
2019-06-28 16:42:35 Andreas Hasenack bug task added apache2 (Ubuntu Bionic)
2019-06-28 16:42:35 Andreas Hasenack bug task added openssl (Ubuntu Bionic)
2019-06-28 16:42:53 Andreas Hasenack openssl (Ubuntu): status Confirmed Incomplete
2019-06-28 16:43:08 Andreas Hasenack bug task deleted openssl (Ubuntu Bionic)
2019-06-28 16:43:19 Andreas Hasenack bug task deleted openssl (Ubuntu Cosmic)
2019-06-28 16:44:01 Andreas Hasenack apache2 (Ubuntu Bionic): status New Confirmed
2019-06-28 16:44:06 Andreas Hasenack apache2 (Ubuntu Cosmic): status New Confirmed
2019-06-28 16:53:47 Benjamin Baumer bug added subscriber Benjamin Baumer
2019-07-01 14:37:58 Andreas Hasenack description I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2: Installiert: 2.4.29-1ubuntu4.6 Installationskandidat: 2.4.29-1ubuntu4.6 Versionstabelle: *** 2.4.29-1ubuntu4.6 500 500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages 100 /var/lib/dpkg/status 2.4.29-1ubuntu4 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl: Installiert: 1.1.1-1ubuntu2.1~18.04.2 Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2 Versionstabelle: *** 1.1.1-1ubuntu2.1~18.04.2 500 500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages 100 /var/lib/dpkg/status 1.1.0g-2ubuntu4.3 500 500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages 1.1.0g-2ubuntu4 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages [Impact] * An explanation of the effects of the bug on users and * justification for backporting the fix to the stable release. * In addition, it is helpful, but not required, to include an explanation of how the upload fixes this bug. [Test Case] * detailed instructions how to reproduce the bug * these should allow someone who is not familiar with the affected package to reproduce the bug and verify that the updated package fixes the problem. [Regression Potential] * discussion of how regressions are most likely to manifest as a result of this change. * It is assumed that any SRU candidate patch is well-tested before upload and has a low overall risk of regression, but it's important to make the effort to think about what ''could'' happen in the event of a regression. * This both shows the SRU team that the risks have been considered, and provides guidance to testers in regression-testing the SRU. [Other Info] * Anything else you think is useful to include * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board * and address these questions in advance [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
2019-07-01 14:41:02 Andreas Hasenack description [Impact] * An explanation of the effects of the bug on users and * justification for backporting the fix to the stable release. * In addition, it is helpful, but not required, to include an explanation of how the upload fixes this bug. [Test Case] * detailed instructions how to reproduce the bug * these should allow someone who is not familiar with the affected package to reproduce the bug and verify that the updated package fixes the problem. [Regression Potential] * discussion of how regressions are most likely to manifest as a result of this change. * It is assumed that any SRU candidate patch is well-tested before upload and has a low overall risk of regression, but it's important to make the effort to think about what ''could'' happen in the event of a regression. * This both shows the SRU team that the risks have been considered, and provides guidance to testers in regression-testing the SRU. [Other Info] * Anything else you think is useful to include * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board * and address these questions in advance [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages [Impact]  * An explanation of the effects of the bug on users and  * justification for backporting the fix to the stable release.  * In addition, it is helpful, but not required, to include an    explanation of how the upload fixes this bug. [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: ubuntu.pem ubuntu.key cacert.pem Download the client certificate and key files and place them in /root: client-auth.key client-auth.pem Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c> <VirtualHost _default_:443> LogLevel info ssl:warn ServerAdmin webmaster@localhost DocumentRoot /var/www/html LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl ErrorLog \${APACHE_LOG_DIR}/error.log CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl SSLEngine on SSLCertificateFile /etc/apache2/ubuntu.pem SSLCertificateKeyFile /etc/apache2/ubuntu.key SSLCACertificateFile /etc/apache2/cacert.pem <FilesMatch "\.(cgi|shtml|phtml|php)$"> SSLOptions +StdEnvVars </FilesMatch> <Directory /usr/lib/cgi-bin> SSLOptions +StdEnvVars </Directory> <Location /> SSLVerifyClient require Require ssl-verify-client </Location> </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed.   [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
2019-07-01 14:55:11 Andreas Hasenack description [Impact]  * An explanation of the effects of the bug on users and  * justification for backporting the fix to the stable release.  * In addition, it is helpful, but not required, to include an    explanation of how the upload fixes this bug. [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: ubuntu.pem ubuntu.key cacert.pem Download the client certificate and key files and place them in /root: client-auth.key client-auth.pem Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c> <VirtualHost _default_:443> LogLevel info ssl:warn ServerAdmin webmaster@localhost DocumentRoot /var/www/html LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl ErrorLog \${APACHE_LOG_DIR}/error.log CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl SSLEngine on SSLCertificateFile /etc/apache2/ubuntu.pem SSLCertificateKeyFile /etc/apache2/ubuntu.key SSLCACertificateFile /etc/apache2/cacert.pem <FilesMatch "\.(cgi|shtml|phtml|php)$"> SSLOptions +StdEnvVars </FilesMatch> <Directory /usr/lib/cgi-bin> SSLOptions +StdEnvVars </Directory> <Location /> SSLVerifyClient require Require ssl-verify-client </Location> </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed.   [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (15s or more if modreqtimeout is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: ubuntu.pem ubuntu.key cacert.pem Download the client certificate and key files and place them in /root: client-auth.key client-auth.pem Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
2019-07-01 14:57:28 Andreas Hasenack attachment added cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem
2019-07-01 14:57:46 Andreas Hasenack attachment added ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem
2019-07-01 14:58:02 Andreas Hasenack attachment added ubuntu.key https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key
2019-07-01 14:58:16 Andreas Hasenack attachment added client-auth.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274495/+files/client-auth.pem
2019-07-01 14:58:36 Andreas Hasenack attachment added client-auth.key https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274496/+files/client-auth.key
2019-07-01 15:00:30 Andreas Hasenack description [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (15s or more if modreqtimeout is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: ubuntu.pem ubuntu.key cacert.pem Download the client certificate and key files and place them in /root: client-auth.key client-auth.pem Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (15s or more if modreqtimeout is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: client-auth.key client-auth.pem Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
2019-07-01 15:06:15 Oibaf bug added subscriber Oibaf
2019-07-01 16:25:48 Andreas Hasenack description [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (15s or more if modreqtimeout is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: client-auth.key client-auth.pem Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (about 15s if modreqtimeout is enabled, more if it is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: client-auth.key client-auth.pem Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
2019-07-01 16:49:41 Andreas Hasenack description [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (about 15s if modreqtimeout is enabled, more if it is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: client-auth.key client-auth.pem Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential]  * discussion of how regressions are most likely to manifest as a result of this change.  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (about 15s if modreqtimeout is enabled, more if it is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: client-auth.key client-auth.pem Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential] This is reverting, in mod_ssl, a settings change that was made in openssl 1.1.1. It's committed upstream in mod_ssl, and I found no other follow-up commits about this. This being SSL-related, of course it can have surprises and be complicated. The openssl commit even warns that hangs could occur (which happened in this bug here), and it's expected that applications that are affected adjust accordingly. [Other Info] The upstream mod_ssl commit that made this change also changed something else in the code, but we decided to not adopt it because it's TLSv1.3 specific, and that version of the protocol is not enabled in the apache builds from bionic or cosmic. TLSv1.3 is deemed complete only in apache 2.4.37 and later (http://www.apache.org/dist/httpd/CHANGES_2.4.37) [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
2019-07-01 16:57:49 Andreas Hasenack description [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (about 15s if modreqtimeout is enabled, more if it is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: client-auth.key client-auth.pem Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential] This is reverting, in mod_ssl, a settings change that was made in openssl 1.1.1. It's committed upstream in mod_ssl, and I found no other follow-up commits about this. This being SSL-related, of course it can have surprises and be complicated. The openssl commit even warns that hangs could occur (which happened in this bug here), and it's expected that applications that are affected adjust accordingly. [Other Info] The upstream mod_ssl commit that made this change also changed something else in the code, but we decided to not adopt it because it's TLSv1.3 specific, and that version of the protocol is not enabled in the apache builds from bionic or cosmic. TLSv1.3 is deemed complete only in apache 2.4.37 and later (http://www.apache.org/dist/httpd/CHANGES_2.4.37) [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (about 15s if modreqtimeout is enabled, more if it is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: cd /root wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274495/+files/client-auth.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274496/+files/client-auth.key client-auth.key client-auth.pem Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential] This is reverting, in mod_ssl, a settings change that was made in openssl 1.1.1. It's committed upstream in mod_ssl, and I found no other follow-up commits about this. This being SSL-related, of course it can have surprises and be complicated. The openssl commit even warns that hangs could occur (which happened in this bug here), and it's expected that applications that are affected adjust accordingly. [Other Info] The upstream mod_ssl commit that made this change also changed something else in the code, but we decided to not adopt it because it's TLSv1.3 specific, and that version of the protocol is not enabled in the apache builds from bionic or cosmic. TLSv1.3 is deemed complete only in apache 2.4.37 and later (http://www.apache.org/dist/httpd/CHANGES_2.4.37) [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
2019-07-01 17:47:03 Andreas Hasenack description [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (about 15s if modreqtimeout is enabled, more if it is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: cd /root wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274495/+files/client-auth.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274496/+files/client-auth.key client-auth.key client-auth.pem Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential] This is reverting, in mod_ssl, a settings change that was made in openssl 1.1.1. It's committed upstream in mod_ssl, and I found no other follow-up commits about this. This being SSL-related, of course it can have surprises and be complicated. The openssl commit even warns that hangs could occur (which happened in this bug here), and it's expected that applications that are affected adjust accordingly. [Other Info] The upstream mod_ssl commit that made this change also changed something else in the code, but we decided to not adopt it because it's TLSv1.3 specific, and that version of the protocol is not enabled in the apache builds from bionic or cosmic. TLSv1.3 is deemed complete only in apache 2.4.37 and later (http://www.apache.org/dist/httpd/CHANGES_2.4.37) [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages [Impact] Under the following conditions, https connections using client cert authentication will suffer a long delay (about 15s if modreqtimeout is enabled, more if it is disabled): * TLSv1.2 * client certificate authentication in use * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole This was triggered by the OpenSSL 1.1.1 SRU and was caused by this openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by default: https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130 [Test Case] It helps if you have lxd up and running. Otherwise, a VM or even bare metal host also works, as long as you stick to the "ubuntu" hostname. Launch a container for the release you are testing. The command below is for bionic: $ lxc launch ubuntu-daily:bionic ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: cd /root wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274495/+files/client-auth.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274496/+files/client-auth.key Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c>     <VirtualHost _default_:443>         LogLevel info ssl:warn         ServerAdmin webmaster@localhost         DocumentRoot /var/www/html         LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl         ErrorLog \${APACHE_LOG_DIR}/error.log         CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl         SSLEngine on         SSLCertificateFile /etc/apache2/ubuntu.pem         SSLCertificateKeyFile /etc/apache2/ubuntu.key         SSLCACertificateFile /etc/apache2/cacert.pem         <FilesMatch "\.(cgi|shtml|phtml|php)$">                 SSLOptions +StdEnvVars         </FilesMatch>         <Directory /usr/lib/cgi-bin>                 SSLOptions +StdEnvVars         </Directory>         <Location />                 SSLVerifyClient require                 Require ssl-verify-client         </Location>     </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 If at this stage you try the following command, it will fail like this because no client certificate was provided: # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0 And the apache error log will confirm the reason: [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification? Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded: # rm -f index.html # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 706 0 0:00:15 0:00:15 --:--:-- 2579 # ll index.html -rw-r--r-- 1 root root 10918 Jul 1 14:15 index.html Apache will log this in the error.log file: [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout That is due to modreqtimeout kicking in. In the access.log file, we will have the request: 10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication. So it works, but takes a long time for each request. This verifies the bug. After installing the fixed apache2 packages, the download completes almost instantly: # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2   % Total % Received % Xferd Average Speed Time Time Time Current                                  Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 333k 0 --:--:-- --:--:-- --:--:-- 333k The apache access log confirms the protocol and that client certificate authentication was used: 10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth And the error log gets no new entries. This verifies the bug is fixed. [Regression Potential] This is reverting, in mod_ssl, a settings change that was made in openssl 1.1.1. It's committed upstream in mod_ssl, and I found no other follow-up commits about this. This being SSL-related, of course it can have surprises and be complicated. The openssl commit even warns that hangs could occur (which happened in this bug here), and it's expected that applications that are affected adjust accordingly. [Other Info] The upstream mod_ssl commit that made this change also changed something else in the code, but we decided to not adopt it because it's TLSv1.3 specific, and that version of the protocol is not enabled in the apache builds from bionic or cosmic. TLSv1.3 is deemed complete only in apache 2.4.37 and later (http://www.apache.org/dist/httpd/CHANGES_2.4.37) [Original Description] I am using Apache2 with client certificate authentication. Since recently (last week) and without any configuration changes, the following errors occur frequently: AH02042: rejecting client initiated renegotiation Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser. Before installation of the latest security fixes last week, this error did not occur. Could it be related to https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689? System information: Description: Ubuntu 18.04.2 LTS Release: 18.04 apache2:   Installiert: 2.4.29-1ubuntu4.6   Installationskandidat: 2.4.29-1ubuntu4.6   Versionstabelle:  *** 2.4.29-1ubuntu4.6 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages         100 /var/lib/dpkg/status      2.4.29-1ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages openssl:   Installiert: 1.1.1-1ubuntu2.1~18.04.2   Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2   Versionstabelle:  *** 1.1.1-1ubuntu2.1~18.04.2 500         500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages         100 /var/lib/dpkg/status      1.1.0g-2ubuntu4.3 500         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages      1.1.0g-2ubuntu4 500         500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
2019-07-01 17:58:24 Launchpad Janitor merge proposal linked https://code.launchpad.net/~ahasenack/ubuntu/+source/apache2/+git/apache2/+merge/369541
2019-07-01 17:59:06 Launchpad Janitor merge proposal linked https://code.launchpad.net/~ahasenack/ubuntu/+source/apache2/+git/apache2/+merge/369542
2019-07-01 18:00:01 Andreas Hasenack apache2 (Ubuntu Cosmic): assignee Andreas Hasenack (ahasenack)
2019-07-01 18:00:03 Andreas Hasenack apache2 (Ubuntu Bionic): assignee Andreas Hasenack (ahasenack)
2019-07-01 18:00:06 Andreas Hasenack apache2 (Ubuntu Bionic): importance Undecided High
2019-07-01 18:00:08 Andreas Hasenack apache2 (Ubuntu Cosmic): importance Undecided High
2019-07-01 18:00:11 Andreas Hasenack apache2 (Ubuntu): importance Undecided Critical
2019-07-01 18:00:17 Andreas Hasenack apache2 (Ubuntu): status Confirmed Fix Released
2019-07-01 18:00:20 Andreas Hasenack apache2 (Ubuntu): importance Critical High
2019-07-01 18:00:27 Andreas Hasenack apache2 (Ubuntu Bionic): status Confirmed In Progress
2019-07-01 18:00:29 Andreas Hasenack apache2 (Ubuntu Cosmic): status Confirmed In Progress
2019-07-01 18:00:31 Andreas Hasenack openssl (Ubuntu): status Incomplete Invalid
2019-07-01 18:36:42 Virsacer bug added subscriber Virsacer
2019-07-02 15:22:07 Brian Murray apache2 (Ubuntu Cosmic): status In Progress Fix Committed
2019-07-02 15:22:08 Brian Murray bug added subscriber Ubuntu Stable Release Updates Team
2019-07-02 15:22:10 Brian Murray bug added subscriber SRU Verification
2019-07-02 15:22:17 Brian Murray tags regression-update rls-bb-incoming regression-update rls-bb-incoming verification-needed verification-needed-cosmic
2019-07-02 15:23:36 Brian Murray apache2 (Ubuntu Bionic): status In Progress Fix Committed
2019-07-02 15:23:46 Brian Murray tags regression-update rls-bb-incoming verification-needed verification-needed-cosmic regression-update rls-bb-incoming verification-needed verification-needed-bionic verification-needed-cosmic
2019-07-04 18:00:38 Mathew Hodson bug task deleted openssl (Ubuntu)
2019-07-05 14:00:44 Andreas Hasenack tags regression-update rls-bb-incoming verification-needed verification-needed-bionic verification-needed-cosmic regression-update rls-bb-incoming verification-done-bionic verification-needed verification-needed-cosmic
2019-07-05 14:02:25 Andreas Hasenack tags regression-update rls-bb-incoming verification-done-bionic verification-needed verification-needed-cosmic regression-update rls-bb-incoming verification-done-bionic verification-done-cosmic verification-needed
2019-07-11 08:45:53 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2019-07-11 08:55:56 Launchpad Janitor apache2 (Ubuntu Cosmic): status Fix Committed Fix Released
2019-07-11 09:20:07 Launchpad Janitor apache2 (Ubuntu Bionic): status Fix Committed Fix Released
2019-10-08 21:14:25 Robie Basak tags regression-update rls-bb-incoming verification-done-bionic verification-done-cosmic verification-needed bionic-openssl-1.1 regression-update rls-bb-incoming verification-done-bionic verification-done-cosmic verification-needed