Ubuntu
apache2 package

  1. Bug #1930430
  2. Activity log

Activity log for bug #1930430

Date Who What changed Old value New value Message
2021-06-01 14:49:11 Horst Platz bug added bug
2021-06-02 17:36:25 Athos Ribeiro apache2 (Ubuntu): status New Triaged
2021-06-02 17:36:36 Athos Ribeiro apache2 (Ubuntu): importance Undecided Medium
2021-06-02 17:36:59 Athos Ribeiro bug added subscriber Ubuntu Server
2021-06-02 17:41:14 Athos Ribeiro bug watch added https://bz.apache.org/bugzilla/show_bug.cgi?id=63679
2021-06-02 17:43:27 Athos Ribeiro bug task added apache2
2021-06-02 17:46:56 Sergio Durigan Junior nominated for series Ubuntu Focal
2021-06-02 17:46:56 Sergio Durigan Junior bug task added apache2 (Ubuntu Focal)
2021-06-02 17:47:05 Sergio Durigan Junior apache2 (Ubuntu Focal): status New Triaged
2021-06-02 17:47:08 Sergio Durigan Junior apache2 (Ubuntu Focal): importance Undecided Medium
2021-06-02 17:47:16 Sergio Durigan Junior apache2 (Ubuntu): status Triaged Fix Released
2021-06-02 23:55:53 Bug Watch Updater apache2: status Unknown Fix Released
2021-06-02 23:55:53 Bug Watch Updater apache2: importance Unknown Medium
2021-06-30 20:23:20 Sergio Durigan Junior bug added subscriber Sergio Durigan Junior
2021-07-05 07:00:18 Christian Ehrhardt  tags server-next
2021-07-05 07:24:40 Launchpad Janitor merge proposal linked https://code.launchpad.net/~paelzer/ubuntu/+source/apache2/+git/apache2/+merge/405164
2021-07-05 07:30:37 Christian Ehrhardt  description Description: Ubuntu 20.04.2 LTS Release: 20.04 Codename: focal After dist-upgrade bionic -> focal and Apache Update from: 2.4.29-1ubuntu4.14 to: 2.4.41-4ubuntu3.1 Overall I found a hint in https://downloads.apache.org/httpd/CHANGES_2.4 [...] *) mod_ssl: OCSP does not apply to proxy mode. PR 63679. [Lubos Uhliarik <luhliari redhat.com>, Yann Ylavic] [...] https://bz.apache.org/bugzilla/show_bug.cgi?id=63679 Backported to 2.4.x (r1872226), will be in the next release. https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/ssl/ssl_engine_kernel.c?view=markup&pathrev=1872226 -> This is part of 2.4.42 <- and a overall Question is can you please also backport that Version from ssl_engine_kernel.c in your 2.4.41-4ubuntu3.1 Apache? My Further on investigation. I Create a new VM with 20.04 an compile Apache :~$ apt-get source apache2 The Only thing i do is to replace :~$ apache2-2.4.41/modules/ssl/ssl_engine_kernel.c with the downloaded Version from upstream Apache https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/ssl/ssl_engine_kernel.c?revision=1872226&view=co&pathrev=1872226 The *.deb Packages i Saved away. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Reproduce the Error Create a New VM with 20.04 :~# apt-get install apache2 :~# mkdir /etc/apache2/ssl :~# vim /etc/apache2/ssl/letsencryt.crt in letsencryt.crt has only the intermediate ans rootCA from letsencryt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - :~# vim /etc/apache2/sites-enabled/000-default.conf <VirtualHost 127.0.0.1:80> ServerAdmin web@localhorst.org ServerName localhost ProxyPreserveHost Off ProxyRequests Off SSLProxyEngine On SSLProxyVerify require SSLProxyCheckPeerName On SSLProxyCheckPeerExpire On SSLProxyVerifyDepth 2 SSLProxyCACertificateFile ssl/letsencryt.crt SSLProxyCipherSuite ECDHE-RSA-AES256-GCM-SHA384,DHE-RSA-AES256-GCM-SHA384 SSLProxyProtocol -all +TLSv1.2 ProxyPass / https://localhorst.org/ LogLevel debug CustomLog ${APACHE_LOG_DIR}/localhorst_access.log common </VirtualHost> :~# vim /etc/apache2/apache2.conf LogLevel debug :~# a2enmod proxy_http ssl :~# systemctl restart apache2 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - I Create a local Firewall for better overview Block outgoing Traffic - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - The Proxy crashed because -> connecting to OCSP responder. With the Apache Version within bionic this does not happend. There is no connection to the OCSP responder. :~# curl http://127.0.0.1:80/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Proxy Error</title> </head><body> <h1>Proxy Error</h1> The proxy server could not handle the request<p>Reason: <strong>Error during SSL Handshake with remote server</strong></p><p /> <hr> <address>Apache/2.4.41 (Ubuntu) Server at 127.0.0.1 Port 80</address> </body></html> :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:04:11.286448 2021] [authz_core:debug] [pid 6009:tid 140286852331264] mod_authz_core.c(845): [client 127.0.0.1:47958] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:04:11.286530 2021] [proxy:debug] [pid 6009:tid 140286852331264] mod_proxy.c(1253): [client 127.0.0.1:47958] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:04:11.286549 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:04:11.286588 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2379): [client 127.0.0.1:47958] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:04:11.288378 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2588): [client 127.0.0.1:47958] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:04:11.318587 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.318697 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.318726 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:04:11.368501 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:04:11.369207 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:04:11.369934 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_ocsp.c(76): [remote 94.130.99.225:443] AH01918: no OCSP responder specified in certificate and no default configured [Tue Jun 01 14:04:11.370521 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:04:11.517640 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_util_ocsp.c(96): [remote 94.130.99.225:443] AH01973: connecting to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:04:11.521410 2021] [ssl:error] [pid 6009:tid 140286852331264] (101)Network is unreachable: [remote 94.130.99.225:443] AH01974: could not connect to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:04:11.521875 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH02276: Certificate Verification: Error (50): application verification failure [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:04:11.529291 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH02003: SSL Proxy connect failed [Tue Jun 01 14:04:11.529591 2021] [ssl:info] [pid 6009:tid 140286852331264] SSL Library Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed [Tue Jun 01 14:04:11.529708 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01998: Connection closed to child 0 with abortive shutdown (server localhost:80) [Tue Jun 01 14:04:11.529999 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01997: SSL handshake failed: sending 502 [Tue Jun 01 14:04:11.530169 2021] [proxy:error] [pid 6009:tid 140286852331264] (20014)Internal error (specific information not available): [client 127.0.0.1:47958] AH01084: pass request body failed to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.530288 2021] [proxy:error] [pid 6009:tid 140286852331264] [client 127.0.0.1:47958] AH00898: Error during SSL Handshake with remote server returned by / [Tue Jun 01 14:04:11.530379 2021] [proxy_http:error] [pid 6009:tid 140286852331264] [client 127.0.0.1:47958] AH01097: pass request body failed to 94.130.99.225:443 (localhorst.org) from 127.0.0.1 () [Tue Jun 01 14:04:11.530482 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2340): AH00943: HTTPS: has released connection for (localhorst.org) :~# tail -f /var/log/ulog/syslogemu.log Jun 1 14:04:12 devubu2004 fw-net REJECT IN= OUT=enp0s3 MAC= SRC=10.0.2.15 DST=95.101.91.160 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=59096 DF PROTO=TCP SPT=52194 DPT=80 SEQ=2173056195 ACK=0 WINDOW=64240 SYN URGP=0 UID=33 GID=33 MARK=0 Jun 1 14:04:12 devubu2004 fw-net REJECT IN= OUT=enp0s3 MAC= SRC=10.0.2.15 DST=95.101.91.146 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=32240 DF PROTO=TCP SPT=40016 DPT=80 SEQ=508673920 ACK=0 WINDOW=64240 SYN URGP=0 UID=33 GID=33 MARK=0 :~$ host r3.o.lencr.org r3.o.lencr.org is an alias for o.lencr.edgesuite.net. o.lencr.edgesuite.net is an alias for a1887.dscq.akamai.net. a1887.dscq.akamai.net has address 95.101.91.160 a1887.dscq.akamai.net has address 95.101.91.146 a1887.dscq.akamai.net has IPv6 address 2a02:26f0:10c::5f65:5a12 a1887.dscq.akamai.net has IPv6 address 2a02:26f0:10c::5f65:5ac0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Try out open the local Firewall :~# vim /etc/shorewall/rules [...] ACCEPT $FW net:95.101.91.160 tcp http ACCEPT $FW net:95.101.91.146 tcp http :~# systemctl reload shorewall - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Does not help crashed with the Following Error :~$ curl http://127.0.0.1:80/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Proxy Error</title> </head><body> <h1>Proxy Error</h1> The proxy server could not handle the request<p>Reason: <strong>Error during SSL Handshake with remote server</strong></p><p /> <hr> <address>Apache/2.4.41 (Ubuntu) Server at 127.0.0.1 Port 80</address> </body></html> :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:08:02.137740 2021] [authz_core:debug] [pid 6009:tid 140286835545856] mod_authz_core.c(845): [client 127.0.0.1:47974] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:08:02.137793 2021] [proxy:debug] [pid 6009:tid 140286835545856] mod_proxy.c(1253): [client 127.0.0.1:47974] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:08:02.137803 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:08:02.137810 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2379): [client 127.0.0.1:47974] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:08:02.137817 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2588): [client 127.0.0.1:47974] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:08:02.167485 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.168160 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.168655 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:08:02.216198 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:08:02.217565 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:08:02.218976 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_ocsp.c(76): [remote 94.130.99.225:443] AH01918: no OCSP responder specified in certificate and no default configured [Tue Jun 01 14:08:02.219265 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:08:02.358471 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(96): [remote 94.130.99.225:443] AH01973: connecting to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:08:02.386985 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(124): [remote 94.130.99.225:443] AH01975: sending request to OCSP responder [Tue Jun 01 14:08:02.579215 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Server: nginx [Tue Jun 01 14:08:02.581036 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Content-Type: application/ocsp-response [Tue Jun 01 14:08:02.581749 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Content-Length: 503 [Tue Jun 01 14:08:02.581822 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: ETag: "17C919F5E6C36BB41BEAF2C8A1BD012BBFDC3157CAC59588FBFDAE973D089853" [Tue Jun 01 14:08:02.581843 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Last-Modified: Mon, 31 May 2021 09:00:00 UTC [Tue Jun 01 14:08:02.581859 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Cache-Control: public, no-transform, must-revalidate, max-age=43160 [Tue Jun 01 14:08:02.581875 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Expires: Wed, 02 Jun 2021 02:07:22 GMT [Tue Jun 01 14:08:02.581891 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Date: Tue, 01 Jun 2021 14:08:02 GMT [Tue Jun 01 14:08:02.581906 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Connection: close [Tue Jun 01 14:08:02.581922 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(282): [remote 94.130.99.225:443] AH01987: OCSP response: got 503 bytes, 503 total [Tue Jun 01 14:08:02.583980 2021] [ssl:error] [pid 6009:tid 140286835545856] AH01924: Bad OCSP responder answer (bad nonce) [Tue Jun 01 14:08:02.585222 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH02276: Certificate Verification: Error (50): application verification failure [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:08:02.586201 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH02003: SSL Proxy connect failed [Tue Jun 01 14:08:02.587160 2021] [ssl:info] [pid 6009:tid 140286835545856] SSL Library Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed [Tue Jun 01 14:08:02.587226 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01998: Connection closed to child 0 with abortive shutdown (server localhost:80) [Tue Jun 01 14:08:02.587272 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01997: SSL handshake failed: sending 502 [Tue Jun 01 14:08:02.587354 2021] [proxy:error] [pid 6009:tid 140286835545856] (20014)Internal error (specific information not available): [client 127.0.0.1:47974] AH01084: pass request body failed to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.587391 2021] [proxy:error] [pid 6009:tid 140286835545856] [client 127.0.0.1:47974] AH00898: Error during SSL Handshake with remote server returned by / [Tue Jun 01 14:08:02.587407 2021] [proxy_http:error] [pid 6009:tid 140286835545856] [client 127.0.0.1:47974] AH01097: pass request body failed to 94.130.99.225:443 (localhorst.org) from 127.0.0.1 () [Tue Jun 01 14:08:02.587424 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2340): AH00943: HTTPS: has released connection for (localhorst.org) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Close the Firewall Again :~# vim /etc/shorewall/rules [...] #ACCEPT $FW net:95.101.91.160 tcp http #ACCEPT $FW net:95.101.91.146 tcp http :~# systemctl reload shorewall - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Installed the self compiled apache Version withe the Pateched ssl_engine_kernel.c Version :~# cd /home/vagrant/deb/ :~# dpkg -i apache2_2.4.41-4ubuntu3.1_amd64.deb apache2-bin_2.4.41-4ubuntu3.1_amd64.deb apache2-data_2.4.41-4ubuntu3.1_all.deb apache2-utils_2.4.41-4ubuntu3.1_amd64.deb :~# systemctl stop apache2 :~# systemctl start apache2 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Apache Proxy is working again as expected :~# curl http://127.0.0.1:80/ -> webite is comming :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:11:47.953485 2021] [authz_core:debug] [pid 7437:tid 140452002883328] mod_authz_core.c(845): [client 127.0.0.1:47980] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:11:47.953554 2021] [proxy:debug] [pid 7437:tid 140452002883328] mod_proxy.c(1253): [client 127.0.0.1:47980] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:11:47.953570 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:11:47.953576 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2379): [client 127.0.0.1:47980] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:11:47.955415 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2588): [client 127.0.0.1:47980] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:11:47.985343 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:11:47.985479 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:11:47.985505 2021] [ssl:info] [pid 7437:tid 140452002883328] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:11:48.034945 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:11:48.035920 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:11:48.036745 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:11:48.067180 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(2249): [remote 94.130.99.225:443] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) [Tue Jun 01 14:11:48.068469 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_util_ssl.c(476): AH02412: [localhost:80] Cert matches for name 'localhorst.org' [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:11:48.227809 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2340): AH00943: https: has released connection for (localhorst.org) Regards Horst [Impact] * Due to https://bz.apache.org/bugzilla/show_bug.cgi?id=63679 the Online Certificate Status Protocol (OCSP) fails in proxy mode. * The fix is simple (the wrong context was checked) and is upstream for a while without further changes. * Backporting that fix [1] resolves the use case [1]: https://github.com/apache/httpd/commit/c11b1cd3b11f [Test Plan] * Autopkgtest plus the steps that were outlined in comment 8 & 9. [Where problems could occur] * Apache does many things, but the change "only" affects the ssl engine. Therefore unexpected problems would be around any sort of ssl activity. But the way the change works is actually ont he SSLVerify path, so it comes down to "making ssl connections" not e.g. later SSL transmission behavior or throughtput. [Other Info] * If we manage to get a certbot system up on canonistack (as I did in the past) to hit this issue we will use that testbed instead of the local tests. ---- Description: Ubuntu 20.04.2 LTS Release: 20.04 Codename: focal After dist-upgrade bionic -> focal and Apache Update from: 2.4.29-1ubuntu4.14 to: 2.4.41-4ubuntu3.1 Overall I found a hint in https://downloads.apache.org/httpd/CHANGES_2.4 [...]   *) mod_ssl: OCSP does not apply to proxy mode. PR 63679.      [Lubos Uhliarik <luhliari redhat.com>, Yann Ylavic] [...] https://bz.apache.org/bugzilla/show_bug.cgi?id=63679 Backported to 2.4.x (r1872226), will be in the next release. https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/ssl/ssl_engine_kernel.c?view=markup&pathrev=1872226 -> This is part of 2.4.42 <- and a overall Question is can you please also backport that Version from ssl_engine_kernel.c in your 2.4.41-4ubuntu3.1 Apache? My Further on investigation. I Create a new VM with 20.04 an compile Apache :~$ apt-get source apache2 The Only thing i do is to replace :~$ apache2-2.4.41/modules/ssl/ssl_engine_kernel.c with the downloaded Version from upstream Apache https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/ssl/ssl_engine_kernel.c?revision=1872226&view=co&pathrev=1872226 The *.deb Packages i Saved away. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Reproduce the Error Create a New VM with 20.04 :~# apt-get install apache2 :~# mkdir /etc/apache2/ssl :~# vim /etc/apache2/ssl/letsencryt.crt in letsencryt.crt has only the intermediate ans rootCA from letsencryt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - :~# vim /etc/apache2/sites-enabled/000-default.conf <VirtualHost 127.0.0.1:80>     ServerAdmin web@localhorst.org     ServerName localhost     ProxyPreserveHost Off     ProxyRequests Off     SSLProxyEngine On     SSLProxyVerify require     SSLProxyCheckPeerName On     SSLProxyCheckPeerExpire On     SSLProxyVerifyDepth 2     SSLProxyCACertificateFile ssl/letsencryt.crt     SSLProxyCipherSuite ECDHE-RSA-AES256-GCM-SHA384,DHE-RSA-AES256-GCM-SHA384     SSLProxyProtocol -all +TLSv1.2     ProxyPass / https://localhorst.org/     LogLevel debug     CustomLog ${APACHE_LOG_DIR}/localhorst_access.log common </VirtualHost> :~# vim /etc/apache2/apache2.conf LogLevel debug :~# a2enmod proxy_http ssl :~# systemctl restart apache2 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - I Create a local Firewall for better overview Block outgoing Traffic - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - The Proxy crashed because -> connecting to OCSP responder. With the Apache Version within bionic this does not happend. There is no connection to the OCSP responder. :~# curl http://127.0.0.1:80/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Proxy Error</title> </head><body> <h1>Proxy Error</h1> The proxy server could not handle the request<p>Reason: <strong>Error during SSL Handshake with remote server</strong></p><p /> <hr> <address>Apache/2.4.41 (Ubuntu) Server at 127.0.0.1 Port 80</address> </body></html> :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:04:11.286448 2021] [authz_core:debug] [pid 6009:tid 140286852331264] mod_authz_core.c(845): [client 127.0.0.1:47958] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:04:11.286530 2021] [proxy:debug] [pid 6009:tid 140286852331264] mod_proxy.c(1253): [client 127.0.0.1:47958] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:04:11.286549 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:04:11.286588 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2379): [client 127.0.0.1:47958] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:04:11.288378 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2588): [client 127.0.0.1:47958] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:04:11.318587 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.318697 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.318726 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:04:11.368501 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:04:11.369207 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:04:11.369934 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_ocsp.c(76): [remote 94.130.99.225:443] AH01918: no OCSP responder specified in certificate and no default configured [Tue Jun 01 14:04:11.370521 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:04:11.517640 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_util_ocsp.c(96): [remote 94.130.99.225:443] AH01973: connecting to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:04:11.521410 2021] [ssl:error] [pid 6009:tid 140286852331264] (101)Network is unreachable: [remote 94.130.99.225:443] AH01974: could not connect to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:04:11.521875 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH02276: Certificate Verification: Error (50): application verification failure [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:04:11.529291 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH02003: SSL Proxy connect failed [Tue Jun 01 14:04:11.529591 2021] [ssl:info] [pid 6009:tid 140286852331264] SSL Library Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed [Tue Jun 01 14:04:11.529708 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01998: Connection closed to child 0 with abortive shutdown (server localhost:80) [Tue Jun 01 14:04:11.529999 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01997: SSL handshake failed: sending 502 [Tue Jun 01 14:04:11.530169 2021] [proxy:error] [pid 6009:tid 140286852331264] (20014)Internal error (specific information not available): [client 127.0.0.1:47958] AH01084: pass request body failed to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.530288 2021] [proxy:error] [pid 6009:tid 140286852331264] [client 127.0.0.1:47958] AH00898: Error during SSL Handshake with remote server returned by / [Tue Jun 01 14:04:11.530379 2021] [proxy_http:error] [pid 6009:tid 140286852331264] [client 127.0.0.1:47958] AH01097: pass request body failed to 94.130.99.225:443 (localhorst.org) from 127.0.0.1 () [Tue Jun 01 14:04:11.530482 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2340): AH00943: HTTPS: has released connection for (localhorst.org) :~# tail -f /var/log/ulog/syslogemu.log Jun 1 14:04:12 devubu2004 fw-net REJECT IN= OUT=enp0s3 MAC= SRC=10.0.2.15 DST=95.101.91.160 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=59096 DF PROTO=TCP SPT=52194 DPT=80 SEQ=2173056195 ACK=0 WINDOW=64240 SYN URGP=0 UID=33 GID=33 MARK=0 Jun 1 14:04:12 devubu2004 fw-net REJECT IN= OUT=enp0s3 MAC= SRC=10.0.2.15 DST=95.101.91.146 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=32240 DF PROTO=TCP SPT=40016 DPT=80 SEQ=508673920 ACK=0 WINDOW=64240 SYN URGP=0 UID=33 GID=33 MARK=0 :~$ host r3.o.lencr.org r3.o.lencr.org is an alias for o.lencr.edgesuite.net. o.lencr.edgesuite.net is an alias for a1887.dscq.akamai.net. a1887.dscq.akamai.net has address 95.101.91.160 a1887.dscq.akamai.net has address 95.101.91.146 a1887.dscq.akamai.net has IPv6 address 2a02:26f0:10c::5f65:5a12 a1887.dscq.akamai.net has IPv6 address 2a02:26f0:10c::5f65:5ac0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Try out open the local Firewall :~# vim /etc/shorewall/rules [...] ACCEPT $FW net:95.101.91.160 tcp http ACCEPT $FW net:95.101.91.146 tcp http :~# systemctl reload shorewall - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Does not help crashed with the Following Error :~$ curl http://127.0.0.1:80/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Proxy Error</title> </head><body> <h1>Proxy Error</h1> The proxy server could not handle the request<p>Reason: <strong>Error during SSL Handshake with remote server</strong></p><p /> <hr> <address>Apache/2.4.41 (Ubuntu) Server at 127.0.0.1 Port 80</address> </body></html> :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:08:02.137740 2021] [authz_core:debug] [pid 6009:tid 140286835545856] mod_authz_core.c(845): [client 127.0.0.1:47974] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:08:02.137793 2021] [proxy:debug] [pid 6009:tid 140286835545856] mod_proxy.c(1253): [client 127.0.0.1:47974] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:08:02.137803 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:08:02.137810 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2379): [client 127.0.0.1:47974] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:08:02.137817 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2588): [client 127.0.0.1:47974] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:08:02.167485 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.168160 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.168655 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:08:02.216198 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:08:02.217565 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:08:02.218976 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_ocsp.c(76): [remote 94.130.99.225:443] AH01918: no OCSP responder specified in certificate and no default configured [Tue Jun 01 14:08:02.219265 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:08:02.358471 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(96): [remote 94.130.99.225:443] AH01973: connecting to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:08:02.386985 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(124): [remote 94.130.99.225:443] AH01975: sending request to OCSP responder [Tue Jun 01 14:08:02.579215 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Server: nginx [Tue Jun 01 14:08:02.581036 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Content-Type: application/ocsp-response [Tue Jun 01 14:08:02.581749 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Content-Length: 503 [Tue Jun 01 14:08:02.581822 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: ETag: "17C919F5E6C36BB41BEAF2C8A1BD012BBFDC3157CAC59588FBFDAE973D089853" [Tue Jun 01 14:08:02.581843 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Last-Modified: Mon, 31 May 2021 09:00:00 UTC [Tue Jun 01 14:08:02.581859 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Cache-Control: public, no-transform, must-revalidate, max-age=43160 [Tue Jun 01 14:08:02.581875 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Expires: Wed, 02 Jun 2021 02:07:22 GMT [Tue Jun 01 14:08:02.581891 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Date: Tue, 01 Jun 2021 14:08:02 GMT [Tue Jun 01 14:08:02.581906 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Connection: close [Tue Jun 01 14:08:02.581922 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(282): [remote 94.130.99.225:443] AH01987: OCSP response: got 503 bytes, 503 total [Tue Jun 01 14:08:02.583980 2021] [ssl:error] [pid 6009:tid 140286835545856] AH01924: Bad OCSP responder answer (bad nonce) [Tue Jun 01 14:08:02.585222 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH02276: Certificate Verification: Error (50): application verification failure [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:08:02.586201 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH02003: SSL Proxy connect failed [Tue Jun 01 14:08:02.587160 2021] [ssl:info] [pid 6009:tid 140286835545856] SSL Library Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed [Tue Jun 01 14:08:02.587226 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01998: Connection closed to child 0 with abortive shutdown (server localhost:80) [Tue Jun 01 14:08:02.587272 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01997: SSL handshake failed: sending 502 [Tue Jun 01 14:08:02.587354 2021] [proxy:error] [pid 6009:tid 140286835545856] (20014)Internal error (specific information not available): [client 127.0.0.1:47974] AH01084: pass request body failed to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.587391 2021] [proxy:error] [pid 6009:tid 140286835545856] [client 127.0.0.1:47974] AH00898: Error during SSL Handshake with remote server returned by / [Tue Jun 01 14:08:02.587407 2021] [proxy_http:error] [pid 6009:tid 140286835545856] [client 127.0.0.1:47974] AH01097: pass request body failed to 94.130.99.225:443 (localhorst.org) from 127.0.0.1 () [Tue Jun 01 14:08:02.587424 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2340): AH00943: HTTPS: has released connection for (localhorst.org) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Close the Firewall Again :~# vim /etc/shorewall/rules [...] #ACCEPT $FW net:95.101.91.160 tcp http #ACCEPT $FW net:95.101.91.146 tcp http :~# systemctl reload shorewall - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Installed the self compiled apache Version withe the Pateched ssl_engine_kernel.c Version :~# cd /home/vagrant/deb/ :~# dpkg -i apache2_2.4.41-4ubuntu3.1_amd64.deb apache2-bin_2.4.41-4ubuntu3.1_amd64.deb apache2-data_2.4.41-4ubuntu3.1_all.deb apache2-utils_2.4.41-4ubuntu3.1_amd64.deb :~# systemctl stop apache2 :~# systemctl start apache2 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Apache Proxy is working again as expected :~# curl http://127.0.0.1:80/ -> webite is comming :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:11:47.953485 2021] [authz_core:debug] [pid 7437:tid 140452002883328] mod_authz_core.c(845): [client 127.0.0.1:47980] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:11:47.953554 2021] [proxy:debug] [pid 7437:tid 140452002883328] mod_proxy.c(1253): [client 127.0.0.1:47980] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:11:47.953570 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:11:47.953576 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2379): [client 127.0.0.1:47980] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:11:47.955415 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2588): [client 127.0.0.1:47980] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:11:47.985343 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:11:47.985479 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:11:47.985505 2021] [ssl:info] [pid 7437:tid 140452002883328] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:11:48.034945 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:11:48.035920 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:11:48.036745 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:11:48.067180 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(2249): [remote 94.130.99.225:443] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) [Tue Jun 01 14:11:48.068469 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_util_ssl.c(476): AH02412: [localhost:80] Cert matches for name 'localhorst.org' [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:11:48.227809 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2340): AH00943: https: has released connection for (localhorst.org) Regards Horst
2021-07-07 11:36:02 Robie Basak apache2 (Ubuntu Focal): status Triaged Fix Committed
2021-07-07 11:36:03 Robie Basak bug added subscriber Ubuntu Stable Release Updates Team
2021-07-07 11:36:05 Robie Basak bug added subscriber SRU Verification
2021-07-07 11:36:07 Robie Basak tags server-next server-next verification-needed verification-needed-focal
2021-07-07 11:40:36 Robie Basak description [Impact] * Due to https://bz.apache.org/bugzilla/show_bug.cgi?id=63679 the Online Certificate Status Protocol (OCSP) fails in proxy mode. * The fix is simple (the wrong context was checked) and is upstream for a while without further changes. * Backporting that fix [1] resolves the use case [1]: https://github.com/apache/httpd/commit/c11b1cd3b11f [Test Plan] * Autopkgtest plus the steps that were outlined in comment 8 & 9. [Where problems could occur] * Apache does many things, but the change "only" affects the ssl engine. Therefore unexpected problems would be around any sort of ssl activity. But the way the change works is actually ont he SSLVerify path, so it comes down to "making ssl connections" not e.g. later SSL transmission behavior or throughtput. [Other Info] * If we manage to get a certbot system up on canonistack (as I did in the past) to hit this issue we will use that testbed instead of the local tests. ---- Description: Ubuntu 20.04.2 LTS Release: 20.04 Codename: focal After dist-upgrade bionic -> focal and Apache Update from: 2.4.29-1ubuntu4.14 to: 2.4.41-4ubuntu3.1 Overall I found a hint in https://downloads.apache.org/httpd/CHANGES_2.4 [...]   *) mod_ssl: OCSP does not apply to proxy mode. PR 63679.      [Lubos Uhliarik <luhliari redhat.com>, Yann Ylavic] [...] https://bz.apache.org/bugzilla/show_bug.cgi?id=63679 Backported to 2.4.x (r1872226), will be in the next release. https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/ssl/ssl_engine_kernel.c?view=markup&pathrev=1872226 -> This is part of 2.4.42 <- and a overall Question is can you please also backport that Version from ssl_engine_kernel.c in your 2.4.41-4ubuntu3.1 Apache? My Further on investigation. I Create a new VM with 20.04 an compile Apache :~$ apt-get source apache2 The Only thing i do is to replace :~$ apache2-2.4.41/modules/ssl/ssl_engine_kernel.c with the downloaded Version from upstream Apache https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/ssl/ssl_engine_kernel.c?revision=1872226&view=co&pathrev=1872226 The *.deb Packages i Saved away. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Reproduce the Error Create a New VM with 20.04 :~# apt-get install apache2 :~# mkdir /etc/apache2/ssl :~# vim /etc/apache2/ssl/letsencryt.crt in letsencryt.crt has only the intermediate ans rootCA from letsencryt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - :~# vim /etc/apache2/sites-enabled/000-default.conf <VirtualHost 127.0.0.1:80>     ServerAdmin web@localhorst.org     ServerName localhost     ProxyPreserveHost Off     ProxyRequests Off     SSLProxyEngine On     SSLProxyVerify require     SSLProxyCheckPeerName On     SSLProxyCheckPeerExpire On     SSLProxyVerifyDepth 2     SSLProxyCACertificateFile ssl/letsencryt.crt     SSLProxyCipherSuite ECDHE-RSA-AES256-GCM-SHA384,DHE-RSA-AES256-GCM-SHA384     SSLProxyProtocol -all +TLSv1.2     ProxyPass / https://localhorst.org/     LogLevel debug     CustomLog ${APACHE_LOG_DIR}/localhorst_access.log common </VirtualHost> :~# vim /etc/apache2/apache2.conf LogLevel debug :~# a2enmod proxy_http ssl :~# systemctl restart apache2 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - I Create a local Firewall for better overview Block outgoing Traffic - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - The Proxy crashed because -> connecting to OCSP responder. With the Apache Version within bionic this does not happend. There is no connection to the OCSP responder. :~# curl http://127.0.0.1:80/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Proxy Error</title> </head><body> <h1>Proxy Error</h1> The proxy server could not handle the request<p>Reason: <strong>Error during SSL Handshake with remote server</strong></p><p /> <hr> <address>Apache/2.4.41 (Ubuntu) Server at 127.0.0.1 Port 80</address> </body></html> :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:04:11.286448 2021] [authz_core:debug] [pid 6009:tid 140286852331264] mod_authz_core.c(845): [client 127.0.0.1:47958] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:04:11.286530 2021] [proxy:debug] [pid 6009:tid 140286852331264] mod_proxy.c(1253): [client 127.0.0.1:47958] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:04:11.286549 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:04:11.286588 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2379): [client 127.0.0.1:47958] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:04:11.288378 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2588): [client 127.0.0.1:47958] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:04:11.318587 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.318697 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.318726 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:04:11.368501 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:04:11.369207 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:04:11.369934 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_ocsp.c(76): [remote 94.130.99.225:443] AH01918: no OCSP responder specified in certificate and no default configured [Tue Jun 01 14:04:11.370521 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:04:11.517640 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_util_ocsp.c(96): [remote 94.130.99.225:443] AH01973: connecting to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:04:11.521410 2021] [ssl:error] [pid 6009:tid 140286852331264] (101)Network is unreachable: [remote 94.130.99.225:443] AH01974: could not connect to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:04:11.521875 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH02276: Certificate Verification: Error (50): application verification failure [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:04:11.529291 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH02003: SSL Proxy connect failed [Tue Jun 01 14:04:11.529591 2021] [ssl:info] [pid 6009:tid 140286852331264] SSL Library Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed [Tue Jun 01 14:04:11.529708 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01998: Connection closed to child 0 with abortive shutdown (server localhost:80) [Tue Jun 01 14:04:11.529999 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01997: SSL handshake failed: sending 502 [Tue Jun 01 14:04:11.530169 2021] [proxy:error] [pid 6009:tid 140286852331264] (20014)Internal error (specific information not available): [client 127.0.0.1:47958] AH01084: pass request body failed to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.530288 2021] [proxy:error] [pid 6009:tid 140286852331264] [client 127.0.0.1:47958] AH00898: Error during SSL Handshake with remote server returned by / [Tue Jun 01 14:04:11.530379 2021] [proxy_http:error] [pid 6009:tid 140286852331264] [client 127.0.0.1:47958] AH01097: pass request body failed to 94.130.99.225:443 (localhorst.org) from 127.0.0.1 () [Tue Jun 01 14:04:11.530482 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2340): AH00943: HTTPS: has released connection for (localhorst.org) :~# tail -f /var/log/ulog/syslogemu.log Jun 1 14:04:12 devubu2004 fw-net REJECT IN= OUT=enp0s3 MAC= SRC=10.0.2.15 DST=95.101.91.160 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=59096 DF PROTO=TCP SPT=52194 DPT=80 SEQ=2173056195 ACK=0 WINDOW=64240 SYN URGP=0 UID=33 GID=33 MARK=0 Jun 1 14:04:12 devubu2004 fw-net REJECT IN= OUT=enp0s3 MAC= SRC=10.0.2.15 DST=95.101.91.146 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=32240 DF PROTO=TCP SPT=40016 DPT=80 SEQ=508673920 ACK=0 WINDOW=64240 SYN URGP=0 UID=33 GID=33 MARK=0 :~$ host r3.o.lencr.org r3.o.lencr.org is an alias for o.lencr.edgesuite.net. o.lencr.edgesuite.net is an alias for a1887.dscq.akamai.net. a1887.dscq.akamai.net has address 95.101.91.160 a1887.dscq.akamai.net has address 95.101.91.146 a1887.dscq.akamai.net has IPv6 address 2a02:26f0:10c::5f65:5a12 a1887.dscq.akamai.net has IPv6 address 2a02:26f0:10c::5f65:5ac0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Try out open the local Firewall :~# vim /etc/shorewall/rules [...] ACCEPT $FW net:95.101.91.160 tcp http ACCEPT $FW net:95.101.91.146 tcp http :~# systemctl reload shorewall - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Does not help crashed with the Following Error :~$ curl http://127.0.0.1:80/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Proxy Error</title> </head><body> <h1>Proxy Error</h1> The proxy server could not handle the request<p>Reason: <strong>Error during SSL Handshake with remote server</strong></p><p /> <hr> <address>Apache/2.4.41 (Ubuntu) Server at 127.0.0.1 Port 80</address> </body></html> :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:08:02.137740 2021] [authz_core:debug] [pid 6009:tid 140286835545856] mod_authz_core.c(845): [client 127.0.0.1:47974] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:08:02.137793 2021] [proxy:debug] [pid 6009:tid 140286835545856] mod_proxy.c(1253): [client 127.0.0.1:47974] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:08:02.137803 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:08:02.137810 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2379): [client 127.0.0.1:47974] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:08:02.137817 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2588): [client 127.0.0.1:47974] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:08:02.167485 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.168160 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.168655 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:08:02.216198 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:08:02.217565 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:08:02.218976 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_ocsp.c(76): [remote 94.130.99.225:443] AH01918: no OCSP responder specified in certificate and no default configured [Tue Jun 01 14:08:02.219265 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:08:02.358471 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(96): [remote 94.130.99.225:443] AH01973: connecting to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:08:02.386985 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(124): [remote 94.130.99.225:443] AH01975: sending request to OCSP responder [Tue Jun 01 14:08:02.579215 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Server: nginx [Tue Jun 01 14:08:02.581036 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Content-Type: application/ocsp-response [Tue Jun 01 14:08:02.581749 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Content-Length: 503 [Tue Jun 01 14:08:02.581822 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: ETag: "17C919F5E6C36BB41BEAF2C8A1BD012BBFDC3157CAC59588FBFDAE973D089853" [Tue Jun 01 14:08:02.581843 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Last-Modified: Mon, 31 May 2021 09:00:00 UTC [Tue Jun 01 14:08:02.581859 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Cache-Control: public, no-transform, must-revalidate, max-age=43160 [Tue Jun 01 14:08:02.581875 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Expires: Wed, 02 Jun 2021 02:07:22 GMT [Tue Jun 01 14:08:02.581891 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Date: Tue, 01 Jun 2021 14:08:02 GMT [Tue Jun 01 14:08:02.581906 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Connection: close [Tue Jun 01 14:08:02.581922 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(282): [remote 94.130.99.225:443] AH01987: OCSP response: got 503 bytes, 503 total [Tue Jun 01 14:08:02.583980 2021] [ssl:error] [pid 6009:tid 140286835545856] AH01924: Bad OCSP responder answer (bad nonce) [Tue Jun 01 14:08:02.585222 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH02276: Certificate Verification: Error (50): application verification failure [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:08:02.586201 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH02003: SSL Proxy connect failed [Tue Jun 01 14:08:02.587160 2021] [ssl:info] [pid 6009:tid 140286835545856] SSL Library Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed [Tue Jun 01 14:08:02.587226 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01998: Connection closed to child 0 with abortive shutdown (server localhost:80) [Tue Jun 01 14:08:02.587272 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01997: SSL handshake failed: sending 502 [Tue Jun 01 14:08:02.587354 2021] [proxy:error] [pid 6009:tid 140286835545856] (20014)Internal error (specific information not available): [client 127.0.0.1:47974] AH01084: pass request body failed to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.587391 2021] [proxy:error] [pid 6009:tid 140286835545856] [client 127.0.0.1:47974] AH00898: Error during SSL Handshake with remote server returned by / [Tue Jun 01 14:08:02.587407 2021] [proxy_http:error] [pid 6009:tid 140286835545856] [client 127.0.0.1:47974] AH01097: pass request body failed to 94.130.99.225:443 (localhorst.org) from 127.0.0.1 () [Tue Jun 01 14:08:02.587424 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2340): AH00943: HTTPS: has released connection for (localhorst.org) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Close the Firewall Again :~# vim /etc/shorewall/rules [...] #ACCEPT $FW net:95.101.91.160 tcp http #ACCEPT $FW net:95.101.91.146 tcp http :~# systemctl reload shorewall - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Installed the self compiled apache Version withe the Pateched ssl_engine_kernel.c Version :~# cd /home/vagrant/deb/ :~# dpkg -i apache2_2.4.41-4ubuntu3.1_amd64.deb apache2-bin_2.4.41-4ubuntu3.1_amd64.deb apache2-data_2.4.41-4ubuntu3.1_all.deb apache2-utils_2.4.41-4ubuntu3.1_amd64.deb :~# systemctl stop apache2 :~# systemctl start apache2 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Apache Proxy is working again as expected :~# curl http://127.0.0.1:80/ -> webite is comming :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:11:47.953485 2021] [authz_core:debug] [pid 7437:tid 140452002883328] mod_authz_core.c(845): [client 127.0.0.1:47980] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:11:47.953554 2021] [proxy:debug] [pid 7437:tid 140452002883328] mod_proxy.c(1253): [client 127.0.0.1:47980] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:11:47.953570 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:11:47.953576 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2379): [client 127.0.0.1:47980] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:11:47.955415 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2588): [client 127.0.0.1:47980] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:11:47.985343 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:11:47.985479 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:11:47.985505 2021] [ssl:info] [pid 7437:tid 140452002883328] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:11:48.034945 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:11:48.035920 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:11:48.036745 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:11:48.067180 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(2249): [remote 94.130.99.225:443] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) [Tue Jun 01 14:11:48.068469 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_util_ssl.c(476): AH02412: [localhost:80] Cert matches for name 'localhorst.org' [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:11:48.227809 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2340): AH00943: https: has released connection for (localhorst.org) Regards Horst [Impact]  * Due to https://bz.apache.org/bugzilla/show_bug.cgi?id=63679 the    Online Certificate Status Protocol (OCSP) fails in proxy mode.  * The fix is simple (the wrong context was checked) and is upstream for    a while without further changes.  * Backporting that fix [1] resolves the use case [1]: https://github.com/apache/httpd/commit/c11b1cd3b11f [Test Plan]  * Autopkgtest plus the steps that were outlined in comment 8 & 9. * [racb] Also see the request for further testing in comment 14. [Where problems could occur]  * Apache does many things, but the change "only" affects the ssl    engine. Therefore unexpected problems would be around any sort    of ssl activity.    But the way the change works is actually ont he SSLVerify path,    so it comes down to "making ssl connections" not e.g. later SSL    transmission behavior or throughtput. [Other Info]  * If we manage to get a certbot system up on canonistack (as I did in    the past) to hit this issue we will use that testbed instead of the    local tests. ---- Description: Ubuntu 20.04.2 LTS Release: 20.04 Codename: focal After dist-upgrade bionic -> focal and Apache Update from: 2.4.29-1ubuntu4.14 to: 2.4.41-4ubuntu3.1 Overall I found a hint in https://downloads.apache.org/httpd/CHANGES_2.4 [...]   *) mod_ssl: OCSP does not apply to proxy mode. PR 63679.      [Lubos Uhliarik <luhliari redhat.com>, Yann Ylavic] [...] https://bz.apache.org/bugzilla/show_bug.cgi?id=63679 Backported to 2.4.x (r1872226), will be in the next release. https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/ssl/ssl_engine_kernel.c?view=markup&pathrev=1872226 -> This is part of 2.4.42 <- and a overall Question is can you please also backport that Version from ssl_engine_kernel.c in your 2.4.41-4ubuntu3.1 Apache? My Further on investigation. I Create a new VM with 20.04 an compile Apache :~$ apt-get source apache2 The Only thing i do is to replace :~$ apache2-2.4.41/modules/ssl/ssl_engine_kernel.c with the downloaded Version from upstream Apache https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/ssl/ssl_engine_kernel.c?revision=1872226&view=co&pathrev=1872226 The *.deb Packages i Saved away. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Reproduce the Error Create a New VM with 20.04 :~# apt-get install apache2 :~# mkdir /etc/apache2/ssl :~# vim /etc/apache2/ssl/letsencryt.crt in letsencryt.crt has only the intermediate ans rootCA from letsencryt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - :~# vim /etc/apache2/sites-enabled/000-default.conf <VirtualHost 127.0.0.1:80>     ServerAdmin web@localhorst.org     ServerName localhost     ProxyPreserveHost Off     ProxyRequests Off     SSLProxyEngine On     SSLProxyVerify require     SSLProxyCheckPeerName On     SSLProxyCheckPeerExpire On     SSLProxyVerifyDepth 2     SSLProxyCACertificateFile ssl/letsencryt.crt     SSLProxyCipherSuite ECDHE-RSA-AES256-GCM-SHA384,DHE-RSA-AES256-GCM-SHA384     SSLProxyProtocol -all +TLSv1.2     ProxyPass / https://localhorst.org/     LogLevel debug     CustomLog ${APACHE_LOG_DIR}/localhorst_access.log common </VirtualHost> :~# vim /etc/apache2/apache2.conf LogLevel debug :~# a2enmod proxy_http ssl :~# systemctl restart apache2 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - I Create a local Firewall for better overview Block outgoing Traffic - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - The Proxy crashed because -> connecting to OCSP responder. With the Apache Version within bionic this does not happend. There is no connection to the OCSP responder. :~# curl http://127.0.0.1:80/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Proxy Error</title> </head><body> <h1>Proxy Error</h1> The proxy server could not handle the request<p>Reason: <strong>Error during SSL Handshake with remote server</strong></p><p /> <hr> <address>Apache/2.4.41 (Ubuntu) Server at 127.0.0.1 Port 80</address> </body></html> :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:04:11.286448 2021] [authz_core:debug] [pid 6009:tid 140286852331264] mod_authz_core.c(845): [client 127.0.0.1:47958] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:04:11.286530 2021] [proxy:debug] [pid 6009:tid 140286852331264] mod_proxy.c(1253): [client 127.0.0.1:47958] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:04:11.286549 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:04:11.286588 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2379): [client 127.0.0.1:47958] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:04:11.288378 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2588): [client 127.0.0.1:47958] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:04:11.318587 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.318697 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.318726 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:04:11.368501 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:04:11.369207 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:04:11.369934 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_ocsp.c(76): [remote 94.130.99.225:443] AH01918: no OCSP responder specified in certificate and no default configured [Tue Jun 01 14:04:11.370521 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:04:11.517640 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_util_ocsp.c(96): [remote 94.130.99.225:443] AH01973: connecting to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:04:11.521410 2021] [ssl:error] [pid 6009:tid 140286852331264] (101)Network is unreachable: [remote 94.130.99.225:443] AH01974: could not connect to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:04:11.521875 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH02276: Certificate Verification: Error (50): application verification failure [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:04:11.529291 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH02003: SSL Proxy connect failed [Tue Jun 01 14:04:11.529591 2021] [ssl:info] [pid 6009:tid 140286852331264] SSL Library Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed [Tue Jun 01 14:04:11.529708 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01998: Connection closed to child 0 with abortive shutdown (server localhost:80) [Tue Jun 01 14:04:11.529999 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01997: SSL handshake failed: sending 502 [Tue Jun 01 14:04:11.530169 2021] [proxy:error] [pid 6009:tid 140286852331264] (20014)Internal error (specific information not available): [client 127.0.0.1:47958] AH01084: pass request body failed to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:04:11.530288 2021] [proxy:error] [pid 6009:tid 140286852331264] [client 127.0.0.1:47958] AH00898: Error during SSL Handshake with remote server returned by / [Tue Jun 01 14:04:11.530379 2021] [proxy_http:error] [pid 6009:tid 140286852331264] [client 127.0.0.1:47958] AH01097: pass request body failed to 94.130.99.225:443 (localhorst.org) from 127.0.0.1 () [Tue Jun 01 14:04:11.530482 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2340): AH00943: HTTPS: has released connection for (localhorst.org) :~# tail -f /var/log/ulog/syslogemu.log Jun 1 14:04:12 devubu2004 fw-net REJECT IN= OUT=enp0s3 MAC= SRC=10.0.2.15 DST=95.101.91.160 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=59096 DF PROTO=TCP SPT=52194 DPT=80 SEQ=2173056195 ACK=0 WINDOW=64240 SYN URGP=0 UID=33 GID=33 MARK=0 Jun 1 14:04:12 devubu2004 fw-net REJECT IN= OUT=enp0s3 MAC= SRC=10.0.2.15 DST=95.101.91.146 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=32240 DF PROTO=TCP SPT=40016 DPT=80 SEQ=508673920 ACK=0 WINDOW=64240 SYN URGP=0 UID=33 GID=33 MARK=0 :~$ host r3.o.lencr.org r3.o.lencr.org is an alias for o.lencr.edgesuite.net. o.lencr.edgesuite.net is an alias for a1887.dscq.akamai.net. a1887.dscq.akamai.net has address 95.101.91.160 a1887.dscq.akamai.net has address 95.101.91.146 a1887.dscq.akamai.net has IPv6 address 2a02:26f0:10c::5f65:5a12 a1887.dscq.akamai.net has IPv6 address 2a02:26f0:10c::5f65:5ac0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Try out open the local Firewall :~# vim /etc/shorewall/rules [...] ACCEPT $FW net:95.101.91.160 tcp http ACCEPT $FW net:95.101.91.146 tcp http :~# systemctl reload shorewall - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Does not help crashed with the Following Error :~$ curl http://127.0.0.1:80/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Proxy Error</title> </head><body> <h1>Proxy Error</h1> The proxy server could not handle the request<p>Reason: <strong>Error during SSL Handshake with remote server</strong></p><p /> <hr> <address>Apache/2.4.41 (Ubuntu) Server at 127.0.0.1 Port 80</address> </body></html> :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:08:02.137740 2021] [authz_core:debug] [pid 6009:tid 140286835545856] mod_authz_core.c(845): [client 127.0.0.1:47974] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:08:02.137793 2021] [proxy:debug] [pid 6009:tid 140286835545856] mod_proxy.c(1253): [client 127.0.0.1:47974] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:08:02.137803 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:08:02.137810 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2379): [client 127.0.0.1:47974] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:08:02.137817 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2588): [client 127.0.0.1:47974] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:08:02.167485 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.168160 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.168655 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:08:02.216198 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:08:02.217565 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:08:02.218976 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_ocsp.c(76): [remote 94.130.99.225:443] AH01918: no OCSP responder specified in certificate and no default configured [Tue Jun 01 14:08:02.219265 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_kernel.c(1764): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:08:02.358471 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(96): [remote 94.130.99.225:443] AH01973: connecting to OCSP responder 'r3.o.lencr.org' [Tue Jun 01 14:08:02.386985 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(124): [remote 94.130.99.225:443] AH01975: sending request to OCSP responder [Tue Jun 01 14:08:02.579215 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Server: nginx [Tue Jun 01 14:08:02.581036 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Content-Type: application/ocsp-response [Tue Jun 01 14:08:02.581749 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Content-Length: 503 [Tue Jun 01 14:08:02.581822 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: ETag: "17C919F5E6C36BB41BEAF2C8A1BD012BBFDC3157CAC59588FBFDAE973D089853" [Tue Jun 01 14:08:02.581843 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Last-Modified: Mon, 31 May 2021 09:00:00 UTC [Tue Jun 01 14:08:02.581859 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Cache-Control: public, no-transform, must-revalidate, max-age=43160 [Tue Jun 01 14:08:02.581875 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Expires: Wed, 02 Jun 2021 02:07:22 GMT [Tue Jun 01 14:08:02.581891 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Date: Tue, 01 Jun 2021 14:08:02 GMT [Tue Jun 01 14:08:02.581906 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(234): [remote 94.130.99.225:443] AH01981: OCSP response header: Connection: close [Tue Jun 01 14:08:02.581922 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_ocsp.c(282): [remote 94.130.99.225:443] AH01987: OCSP response: got 503 bytes, 503 total [Tue Jun 01 14:08:02.583980 2021] [ssl:error] [pid 6009:tid 140286835545856] AH01924: Bad OCSP responder answer (bad nonce) [Tue Jun 01 14:08:02.585222 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH02276: Certificate Verification: Error (50): application verification failure [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:08:02.586201 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH02003: SSL Proxy connect failed [Tue Jun 01 14:08:02.587160 2021] [ssl:info] [pid 6009:tid 140286835545856] SSL Library Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed [Tue Jun 01 14:08:02.587226 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01998: Connection closed to child 0 with abortive shutdown (server localhost:80) [Tue Jun 01 14:08:02.587272 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01997: SSL handshake failed: sending 502 [Tue Jun 01 14:08:02.587354 2021] [proxy:error] [pid 6009:tid 140286835545856] (20014)Internal error (specific information not available): [client 127.0.0.1:47974] AH01084: pass request body failed to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:08:02.587391 2021] [proxy:error] [pid 6009:tid 140286835545856] [client 127.0.0.1:47974] AH00898: Error during SSL Handshake with remote server returned by / [Tue Jun 01 14:08:02.587407 2021] [proxy_http:error] [pid 6009:tid 140286835545856] [client 127.0.0.1:47974] AH01097: pass request body failed to 94.130.99.225:443 (localhorst.org) from 127.0.0.1 () [Tue Jun 01 14:08:02.587424 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2340): AH00943: HTTPS: has released connection for (localhorst.org) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Close the Firewall Again :~# vim /etc/shorewall/rules [...] #ACCEPT $FW net:95.101.91.160 tcp http #ACCEPT $FW net:95.101.91.146 tcp http :~# systemctl reload shorewall - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Installed the self compiled apache Version withe the Pateched ssl_engine_kernel.c Version :~# cd /home/vagrant/deb/ :~# dpkg -i apache2_2.4.41-4ubuntu3.1_amd64.deb apache2-bin_2.4.41-4ubuntu3.1_amd64.deb apache2-data_2.4.41-4ubuntu3.1_all.deb apache2-utils_2.4.41-4ubuntu3.1_amd64.deb :~# systemctl stop apache2 :~# systemctl start apache2 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Apache Proxy is working again as expected :~# curl http://127.0.0.1:80/ -> webite is comming :~# tail -f /var/log/apache2/error.log [Tue Jun 01 14:11:47.953485 2021] [authz_core:debug] [pid 7437:tid 140452002883328] mod_authz_core.c(845): [client 127.0.0.1:47980] AH01628: authorization result: granted (no directives) [Tue Jun 01 14:11:47.953554 2021] [proxy:debug] [pid 7437:tid 140452002883328] mod_proxy.c(1253): [client 127.0.0.1:47980] AH01143: Running scheme https handler (attempt 0) [Tue Jun 01 14:11:47.953570 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org) [Tue Jun 01 14:11:47.953576 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2379): [client 127.0.0.1:47980] AH00944: connecting https://localhorst.org/ to localhorst.org:443 [Tue Jun 01 14:11:47.955415 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2588): [client 127.0.0.1:47980] AH00947: connected / to localhorst.org:443 [Tue Jun 01 14:11:47.985343 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:11:47.985479 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org) [Tue Jun 01 14:11:47.985505 2021] [ssl:info] [pid 7437:tid 140452002883328] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80) [Tue Jun 01 14:11:48.034945 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 2, CRL checking mode: none (0) [subject: CN=DST Root CA X3,O=Digital Signature Trust Co. / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 44AFB080D6A327BA893039862EF8406B / notbefore: Sep 30 21:12:19 2000 GMT / notafter: Sep 30 14:01:15 2021 GMT] [Tue Jun 01 14:11:48.035920 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=R3,O=Let's Encrypt,C=US / issuer: CN=DST Root CA X3,O=Digital Signature Trust Co. / serial: 400175048314A4C8218C84A90C16CDDF / notbefore: Oct 7 19:21:40 2020 GMT / notafter: Sep 29 19:21:40 2021 GMT] [Tue Jun 01 14:11:48.036745 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(1759): [remote 94.130.99.225:443] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:11:48.067180 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_kernel.c(2249): [remote 94.130.99.225:443] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) [Tue Jun 01 14:11:48.068469 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_util_ssl.c(476): AH02412: [localhost:80] Cert matches for name 'localhorst.org' [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834352A845E6D1745969DCCE / notbefore: May 13 08:11:44 2021 GMT / notafter: Aug 11 08:11:44 2021 GMT] [Tue Jun 01 14:11:48.227809 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2340): AH00943: https: has released connection for (localhorst.org) Regards Horst
2021-07-08 07:17:30 Horst Platz tags server-next verification-needed verification-needed-focal server-next verification-done-focal verification-needed
2021-07-08 10:28:12 Christian Ehrhardt  tags server-next verification-done-focal verification-needed server-next verification-done verification-done-focal
2021-07-15 16:45:21 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2021-07-15 16:45:19 Launchpad Janitor apache2 (Ubuntu Focal): status Fix Committed Fix Released