Regression running ssllabs.com/ssltest causes 2 apache process to eat up 100% cpu, easy DoS

[Stefan Huehner]

[Impact]
With latest apache 2.4.29-1ubuntu4.7 published to 18.04 LTS bionic, when running ssllabs.com/ssltest against it to verify the configuration it leaves 2 apache processes using 100% indefinitely.

Downgrading to 2.4.29-1ubuntu4.6 make it not reproducible anymore.

[Test Case]
We didn't find a reproducer that didn't involve https://ssllabs.com/ssltest, so the test case needs a publicly facing server with a DNS record.

On a test system that has a public IP and is reachable via https on a hostname (not just IP):
sudo apt update
sudo apt install apache2
sudo a2enmod ssl
sudo a2ensite default-ssl.conf
sudo service apache2 restart

In a terminal, monitor the apache2 processes CPU usage with top.

Go to https://www.ssllabs.com/ssltest/ and input the url to your test server, using https. After a few seconds, the site will ask you if it should ignore the certificate error, confirm, and let it continue the test.

After a few minutes, the test will finish and you will get a report. Go back to the terminal where top is running, and the apache2 processes will be spinning and using CPU, even though there isn't anymore traffic.

With the fixed packages, the apache processes will remain idle.

[Regression Potential]
This upload is already fixing a regression which fixed a previous regression (#1833039), which shows that the situation is tricky. The fix here (clear-retry-flags-before-abort.patch) is at least not changing anything in the previous patch from bug #1833039, so that fix was correct.
The second patch, for http/2 errors with openssl 1.1.1, unfortunately has no test case, and deals with error status and is specific to openssl 1.1.1. It's been applied upstream (and backported to the 2.4.x branch) for many months now. The trunk commit at http://svn.apache.org/viewvc?view=revision&revision=1843954 has a more elaborate explanation about behavior changes this does, and doesn't, introduce.
We do have a DEP8 test that covers HTTP/2 SSL downloads, and it passes. But it also passed before this patch. I also manually tried such downloads of varying sizes (up to 10Mbytes) with no failures.

[Other Info]
While investigating this issue, another fix for an openssl 1.1.1 issue was found in the apache upstream git repo which involves http2 and how the code handles SSL_read() return values: https://github.com/apache/httpd/commit/644cff9977efa322fe6c0ae3357a5b8cb1eeec11
No upstream bug was found, nor could I come up with a reproducer case, but it seemed sensible to include that patch in this SRU, which was, after all, triggered by the openssl 1.1.1 upgrade in bionic.
The d/t/run-test-suite DEP8 test is falsely returning success, but it's not running due to being called as root, and it doesn't fail either. I filed bug #1836898 about this, and ran it manually for both cosmic and bionic. There is one test failure, but it's a silly one, introduced by a patch that added a comment. The test actually parses C comments in that particular header file. The bug has the details.

cosmic patched to actually run the testsuite, showing that failure:
http://people.ubuntu.com/~ahasenack/dep8-apache2-1836329-cosmic-actually-run-test-suite/log

Same for bionic:
http://people.ubuntu.com/~ahasenack/dep8-apache2-1836329-bionic-actually-run-test-suite/log

[Original Description]

With latest apache 2.4.29-1ubuntu4.7 published to 18.04 LTS bionic, when running ssllabs.com/ssltest against it to verify the configuration it leaves 2 apache processes using 100% indefinitely.

Downgrading to 2.4.29-1ubuntu4.6 make it not reproducible anymore.

So far i do not know if it is easy/likely to hit this case in normal https usage or only triggered by that testing site.

But given that this is backported to LTS and allows easy DoS maybe the 4.7 should be backed out?

So likely regression in the update to 4.7 having only single fix:
https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039

Extra info observed when that ssltest is over but processes are still there using up cpu:

/server-status shows both processes 25234,25235 here in 'Reading' state:
Srv PID Acc M CPU SS Req Conn Child Slot Client Protocol VHost Request
0-0 25234 0/0/0 W 0.00 0 0 0.0 0.00 0.00 127.0.0.1 http/1.1 ip-172-30-1-107.eu-west-1.compu GET /server-status HTTP/1.1
0-0 25234 0/0/0 R 0.00 641 0 0.0 0.00 0.00 64.41.200.107 http/1.1
1-0 25235 0/1/1 _ 0.00 505 2 0.0 0.00 0.00 64.41.200.107 http/1.1
1-0 25235 0/1/1 _ 0.00 501 0 0.0 0.00 0.00 64.41.200.107 http/1.1
1-0 25235 0/1/1 _ 0.00 500 0 0.0 0.00 0.00 64.41.200.107 http/1.1
1-0 25235 0/1/1 _ 0.00 494 0 0.0 0.00 0.00 64.41.200.107 http/1.1
1-0 25235 0/1/1 _ 0.00 604 0 0.0 0.00 0.00 64.41.200.106 http/1.1
1-0 25235 0/1/1 _ 0.00 604 0 0.0 0.00 0.00 64.41.200.107 http/1.1
1-0 25235 0/1/1 _ 16.93 596 0 0.0 0.00 0.00 64.41.200.107 http/1.1
1-0 25235 0/1/1 _ 0.01 595 1 0.0 0.00 0.00 64.41.200.106 http/1.1
1-0 25235 0/0/0 R 0.00 679 0 0.0 0.00 0.00 64.41.200.106 http/1.1

netstat on system:
tcp6 1 0 172.30.1.57:443 64.41.200.106:58658 CLOSE_WAIT
tcp6 1 0 172.30.1.57:443 64.41.200.107:60842 CLOSE_WAIT

with on other connections to 443 port.

[Christian Ehrhardt  (paelzer)]

Andreas said (on the other bug) he wants to look into this today - assigning him and set prio to high according to the discussion so far.

@Stefan - since https://www.ssllabs.com/ssltest/index.html is external have you found anything else, maybe something reproducible locally without needing to expose the test host to the internet that would trigger the same?

[Christian Ehrhardt  (paelzer)]

I followed [1] to do some checks against the version reported to be bad.

The easiest copy and paste setup would be:
$ apt install apache2
$ IP=$(hostname -i | cut -d' ' -f 2)
$ sed -i -e "/ServerAdmin/a ServerName $IP" -e 's/ssl-cert-snakeoil.pem/apache-selfsigned.crt/' -e 's/ssl-cert-snakeoil.key/apache-selfsigned.key/' /etc/apache2/sites-available/default-ssl.conf
$ sed -i -e "/ServerAdmin/a Redirect \"/\" \"https://$IP/\"" /etc/apache2/sites-available/000-default.conf
$ sudo cat << EOF > /etc/apache2/conf-available/ssl-params.conf
SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder On
Header always set X-Frame-Options DENY
Header always set X-Content-Type-Options nosniff
SSLCompression off
SSLUseStapling on
SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
SSLSessionTickets Off
EOF
$ (sleep 2s; printf "\n"; sleep 2s; printf "\n"; sleep 2s; printf "\n"; sleep 2s; printf "\n"; sleep 2s; printf "\n"; sleep 2s; printf "$IP\n"; sleep 2s; printf "\n";) | sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt
$ sudo a2enmod ssl
$ sudo a2enmod headers
$ sudo a2ensite default-ssl
$ sudo a2enconf ssl-params
$ sudo apache2ctl configtest
$ sudo systemctl restart apache2

The above works in a LXD container and would give a basic setup to test.
Basic usage of this server was fine for me, doing some ssl checks now ...

[1]: https://www.digitalocean.com/community/tutorials/how-to-create-a-self-signed-ssl-certificate-for-apache-in-ubuntu-18-04

[Christian Ehrhardt  (paelzer)]

Since ssllabs [1] is only for exposed hosts I checked their list of other tools [2].
From there I picked the offline tools that seemed usable locally [3][4][5]

[1]: https://www.ssllabs.com/ssltest/index.html
[2]: https://github.com/ssllabs/research/wiki/Assessment-Tools
[3]: https://github.com/mozilla/cipherscan
[4]: https://github.com/nabla-c0d3/sslyze
[5]: https://github.com/drwetter/testssl.sh/releases

[Christian Ehrhardt  (paelzer)]

#1 cipherscan [3]
./cipherscan 10.253.194.151:443 -servername 10.253.194.151
..
Target: 10.253.194.151:443
Certificate: untrusted, bits, signature
TLS ticket lifetime hint:
NPN protocols:
OCSP stapling: not supported
Cipher ordering: server
Curves ordering: none - fallback: no
Renegotiation test error
Supported compression methods test error

TLS Tolerance: no
Fallbacks required:
big-SSLv3 config not supported, connection failed
big-TLSv1.0 config not supported, connection failed
big-TLSv1.1 config not supported, connection failed
big-TLSv1.2 config not supported, connection failed
small-SSLv3 config not supported, connection failed
small-TLSv1.0 config not supported, connection failed
small-TLSv1.0-notlsext config not supported, connection failed
small-TLSv1.1 config not supported, connection failed
small-TLSv1.2 config not supported, connection failed
v2-big-TLSv1.2 no fallback req, connected: TLSv1.2 DHE-RSA-AES256-GCM-SHA384
v2-small-SSLv3 config not supported, connection failed
v2-small-TLSv1.0 config not supported, connection failed
v2-small-TLSv1.1 config not supported, connection failed
v2-small-TLSv1.2 no fallback req, connected: TLSv1.2 DHE-RSA-AES256-SHA

Intolerance to:
 SSL 3.254 : absent
 TLS 1.0 : PRESENT
 TLS 1.1 : PRESENT
 TLS 1.2 : absent
 TLS 1.3 : absent
 TLS 1.4 : absent

[Christian Ehrhardt  (paelzer)]

#2 sslyze [4]
$ apt install python-pip
$ pip install --upgrade setuptools
$ pip install --upgrade sslyze
$ python -m sslyze --regular 10.253.194.151:443

 AVAILABLE PLUGINS
 -----------------

  OpenSslCcsInjectionPlugin
  CompressionPlugin
  HeartbleedPlugin
  OpenSslCipherSuitesPlugin
  SessionRenegotiationPlugin
  FallbackScsvPlugin
  SessionResumptionPlugin
  HttpHeadersPlugin
  RobotPlugin
  CertificateInfoPlugin

 CHECKING HOST(S) AVAILABILITY
 -----------------------------

   10.253.194.151:443 => 10.253.194.151

 SCAN RESULTS FOR 10.253.194.151:443 - 10.253.194.151
 ----------------------------------------------------

 * OpenSSL CCS Injection:
                                          OK - Not vulnerable to OpenSSL CCS injection

 * Session Renegotiation:
       Client-initiated Renegotiation: OK - Rejected
       Secure Renegotiation: OK - Supported

 * OpenSSL Heartbleed:
                                          OK - Not vulnerable to Heartbleed

 * Resumption Support:
      With Session IDs: OK - Supported (5 successful, 0 failed, 0 errors, 5 total attempts).
      With TLS Tickets: NOT SUPPORTED - TLS ticket not assigned.

 * SSLV3 Cipher Suites:
      Server rejected all cipher suites.

 * TLSV1 Cipher Suites:
      Server rejected all cipher suites.

 * SSLV2 Cipher Suites:
      Server rejected all cipher suites.

 * TLSV1_3 Cipher Suites:
      Server rejected all cipher suites.

 * Downgrade Attacks:
       TLS_FALLBACK_SCSV: OK - Supported

 * TLSV1_2 Cipher Suites:
       Forward Secrecy OK - Supported
       RC4 OK - Not Supported

     Preferred:
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH-256 bits 256 bits HTTP 200 OK
     Accepted:
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 DH-2048 bits 256 bits HTTP 200 OK
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA ECDH-256 bits 256 bits HTTP 200 OK
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDH-256 bits 256 bits HTTP 200 OK
        DHE_RSA_WITH_AES_256_CCM_8 - 256 bits HTTP 200 OK
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH-256 bits 256 bits HTTP 200 OK
        TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 DH-2048 bits 256 bits HTTP 200 OK
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA DH-2048 bits 256 bits HTTP 200 OK
        TLS_DHE_RSA_WITH_AES_256_CCM - 256 bits HTTP 200 OK
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH-256 bits 128 bits HTTP 200 OK ...

[Christian Ehrhardt  (paelzer)]

#3 testssl [5]

$ wget https://github.com/drwetter/testssl.sh/archive/3.0rc5.tar.gz
$ tar xf 3.0rc5.tar.gz
$ cd testssl.sh-3.0rc5/
$ ./testssl.sh 10.253.194.151:443

###########################################################
    testssl.sh 3.0rc5 from https://testssl.sh/dev/

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.0.2-chacha (1.0.2k-dev)" [~183 ciphers]
 on e:./bin/openssl.Linux.x86_64
 (built: "Jan 18 17:12:17 2019", platform: "linux-x86_64")

 Start 2019-07-15 07:03:37 -->> 10.253.194.151:443 (10.253.194.151) <<--

 rDNS (10.253.194.151): b.lxd.
 Service detected: HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2 not offered (OK)
 SSLv3 not offered (OK)
 TLS 1 not offered
 TLS 1.1 not offered
 TLS 1.2 offered (OK)
 TLS 1.3 not offered
 NPN/SPDY not offered
 ALPN/HTTP2 http/1.1 (offered)

 Testing cipher categories

 NULL ciphers (no encryption) not offered (OK)
 Anonymous NULL Ciphers (no authentication) not offered (OK)
 Export ciphers (w/o ADH+NULL) not offered (OK)
 LOW: 64 Bit + DES, RC[2,4] (w/o export) not offered (OK)
 Triple DES Ciphers / IDEA not offered (OK)
 Average: SEED + 128+256 Bit CBC ciphers offered
 Strong encryption (AEAD ciphers) offered (OK)

 Testing robust (perfect) forward secrecy, (P)FS -- omitting Null Authentication/Encryption, 3DES, RC4

 PFS is offered (OK) ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-CCM8 DHE-RSA-AES256-CCM DHE-RSA-AES256-SHA256
                              DHE-RSA-AES256-SHA ECDHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-GCM-SHA256
 Elliptic curves offered: prime256v1 secp384r1 secp521r1 X25519 X448
 DH group offered: RFC3526/Oakley Group 14 (2048 bits)

 Testing server preferences

 Has server cipher order? yes (OK)
 Negotiated protocol TLSv1.2
 Negotiated cipher ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Cipher order
    TLSv1.2: ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA DHE-RSA-AES256-CCM8
               DHE-RSA-AES256-CCM DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA

 Testing server defaults (Server Hello)

 TLS extensions (standard) "renegotiation info/#65281" "EC point formats/#11" "max fragment length/#1" "application layer protocol negotiation/#16" "encrypt-then-mac/#22"
                              "extended master secret/#23"
 Session Ticket RFC 5077 hint no -- no lifetime advertised
 SSL Session ID support yes
 Session Resumption Tickets no, ID: yes
 TLS clock skew Random values, no fingerprinting possible
 Signature Algorithm SHA256 with RSA
 Server key size RSA 2048 bits
 Server key usage --...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Giraffe (dodger-forum)]

Can confirm this bugs, affects our server as well.

[Christian Ehrhardt  (paelzer)]

#1, #2 and #3 gave me nothing in terms of "load" on the target apache server after the actual run.
I had the expected worker threads around but those were idling.

I had no connections in CLOSE_WAIT state nor showing up as reading in the server status module.

@Stefan - we have two obvious paths from here.
a) only the ssllabs test causes this, which will make testing of usual non exposed debug systems much harder
b) some other part of your setup combined with the new version triggers this behavior.

You could help tremendously if you'd have the options to set up either:
- verify (A): setup a default config like I outlined but exposed to the internet and run the ssllabs test against it (if you have the mans to do so)
- verify (b): run the tests mentioned above (and others if you know/have them) against your existing setup. Does and of them trigger the bad behavior for your setup?
If it does try to drop custom config one by one until we have identified which part is the critical one.

Marking incomplete to get this extra info, I hope our both pre-work will help Andreas (who will show up in a bit) to get a fast start on this.

[Christian Ehrhardt  (paelzer)]

@Giraffe - glad for this extra confirmation, it would be great if you could help as well to trim down the test to something more easily reproducible.

Since we have more affected people I'll raise severity and set the regression tag.
If only the initial bug fixed wouldn't be severe as well I'd already upload a revert for now.
But I'll leave that decision to Andreas who will be here in a bit.

I might try to sort out the upstream patch and create a test PPA if something feasible comes up, but for now the biggest step forward would be a simplification of the repro-steps.

[Giraffe (dodger-forum)]

@Christian,
I'll see what i can do, happy to help out.

Would it help if i e-mail you our "/etc/apache2/mods-enabled/ssl.conf" config and a copy of "ssl.conf"?

[Christian Ehrhardt  (paelzer)]

If non confidential attaching it here would be best, so I can try with exactly your settings.
If confidential still the best would be to replace the few critical parts with like <INSERTYOURVALUE> and then attach it here.
Only if all of the above is not feasible mailing it would be the fallback (which would still be better than not having tried with your config at all, so thanks for the offer).

[Christian Ehrhardt  (paelzer)]

I dropped my former ssl-params.conf and added from you for the mod config:
+ SSLHonorCipherOrder on
+ SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA$
+ SSLOpenSSLConfCmd Curves brainpoolP512r1:brainpoolP384r1:brainpoolP256r1:P-521:P-384:X448:X25519:P-256
+ SSLOpenSSLConfCmd DHParameters "/etc/ssl/dhparam.pem"
+ SSLSessionTickets off
+ SSLOpenSSLConfCmd Options +PrioritizeChaCha

Only the /etc/ssl/dhparam.pem needs adaption for me generatign a custom pem:
$ openssl dhparam -out /etc/ssl/dhparam.pem 2048

And in the default-ssl.conf I added from you (the rest was just mail and our different certificates).

section #1 is the same as added in mod-ssl.conf
+ HSTS header
+ http2

And outside the /VirtualHost context you added the OCSP Stapling

With all the tests thrown against it I still get no hanging apache workers in my case.

That seems all the difference we had in our config (unless we missed other files being important).
That leaves:
- the ssllabs test does something special we haven't captured yet in our try to recreate
- it depends on the certificate itself (my self signed vs whatever real cert you use)
- other config is important that we haven't found yet

TODO that remains:
- if one could run ssllabs test against such an easy default self signed setup as I have outlined here (to verify that it triggers the issue and is not depending on further config).
- experiment and find a local test tool that triggers the issue

[Giraffe (dodger-forum)]

If you need more info about the system the server runs on I'll be happy to share them privately, if so, please let me know.

[Christian Ehrhardt  (paelzer)]

I have set up what I described in comment #2 in a openstack instance that has a public IP.
That already was a crap of setup work, but now I realized that I also need dns to be able to use ssllabs check :-/

I have had some free DNS entries to configure and for now have set one of my domains to point to that test host.

@ahasenack - your ssh key is imported on the system which has this ssl setup and hopefully later today is DNS propagated enough to be able to run the ssllabs test against it.

@ahasenack - I'll share the domain privately so hat you can continue testing.

[Christian Ehrhardt  (paelzer)]

OK, DNS propagation has happened.

Scanning with ssllabs test:
- 2.4.29-1ubuntu4.6 (all ok)
- 2.4.29-1ubuntu4.7 (ok as well)

I see worker threads start while the test is running, like 32 or so.
But never does the cpu consumption peak to anything high.

Nor do processes hang around in CLOSED_WAIT state.

There must be something else to it.

Next I ran:
- 2.4.29-1ubuntu4.7 + giraffe config which seems to trigger the bug

And I got:
~100% CPU top:
4375 www-data 20 0 758280 7072 4944 S 88.2 1.5 4:32.91 apache2
netstat
tcp6 1 0 paelzer-apache-bu:https www.ssllabs.com:43511 CLOSE_WAIT

\o/

Now this might be flaky, I'll reset and try again.

[Christian Ehrhardt  (paelzer)]

Tried 2 more times with this config we are now at 3/3 hits.
Seems reproducible enough?

Difference in ssllabs output:
 HTTP Strict Transport Security (HSTS) with long duration deployed on this server.
 Which is green but downgrades the protocol result by 5%
 Anyway, this is one of the changes that we will disable when hunting for the critical config.

The section the 100% starts to show up is labelled "Testing renegotiation" which would match expectations as the fix was about SSL renegotiation.

I'm now dropping config differences one by one ...

[Christian Ehrhardt  (paelzer)]

I tried my local tools and testssh also tests renegotiation but doesn't trigger this :-/
sslyze also has:
 * Session Renegotiation:
       Client-initiated Renegotiation: OK - Rejected
       Secure Renegotiation: OK - Supported

$ openssl s_client -connect dusma.de:443
...
Secure Renegotiation IS supported
...
R
RENEGOTIATING
write:errno=0

[Christian Ehrhardt  (paelzer)]

Disable:
#HSTS Header
Header always set Strict-Transport-Security: "max-age=63072000; includeSubDomains; preload"
=> Still triggering ...

Disable:
#Enable http2
Protocols h2 http/1.1
# AND
SSLUseStapling on
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors off
SSLStaplingCache shmcb:/var/run/ocsp(128000)
=> Still triggering ...

Disable (in mod and site config):
SSLSessionTickets off
SSLOpenSSLConfCmd Options +PrioritizeChaCha
=> Still triggering ...

Disable (in mod and site config):
SSLHonorCipherOrder on
SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:!DSS
SSLOpenSSLConfCmd Curves X448:X25519:P-256:P-384
SSLOpenSSLConfCmd DHParameters "/etc/ssl/dhparam.pem"
=> Still triggering ...

This matches my comment #2 config now.
So the flaky part was the one time it worked find on the initial run?

Retrying two more times ...
Yeah base config still triggers the issue ...
So other than first assumed it was either
a) not the config
OR
b) we needed to run multiple tests to enter some bad state (but apache restarts in between)

In any of the abvoe cases, @andreas you can use the system to test and builds that you have.
Just take a potential pass with a grain of salt and rerun it a few times.

[Christian Ehrhardt  (paelzer)]

I downgraded to 2.4.29-1ubuntu4.6 and confirm the issue not showing up there.
I upgraded to 2.4.29-1ubuntu4.7 and can trigger it again.

You can up/downgrade with:
$ sudo dpkg -i ~/apachedebs/2.4.29-1ubuntu4.7/*
$ sudo dpkg -i ~/apachedebs/2.4.29-1ubuntu4.6/*

The test is at:
 https://www.ssllabs.com/ssltest/analyze.html?d=dusma.de&hideResults=on
and you can restart it with "clear cache"

Leaving the test system in this state for Andreas to continue.

[Andreas Hasenack (ahasenack)]

eoan is fine

[Andreas Hasenack (ahasenack)]

Cosmic affected (2.4.34-1ubuntu2.2)

[Andreas Hasenack (ahasenack)]

Disco is also clean, as expected.

[Andreas Hasenack (ahasenack)]

Actually, disco and eoan never had this bug, so the correct status for those tasks is "invalid".

[Andreas Hasenack (ahasenack)]

I applied https://github.com/apache/httpd/commit/524608b65ec410e797a7283e6e142f8e5a74be26 and https://github.com/apache/httpd/commit/7fa21ea6602b30cc43d4f485777545dd73bb25a6 and that seems to work. Will clean the packaging up with those patches, check if perhaps only one is needed.

PPA with bionic test packages at https://launchpad.net/~ahasenack/+archive/ubuntu/apache-regression-test-2/

[TJ (tj)]

@ahasenack: 2.4.29-1ubuntu4.8~sslreadrc~ppa3 confirmed working without processes spinning

[Stefan Huehner (stefan-huehner)]

@ahasenack: Installing your 2.4.29-1ubuntu4.8~sslreadrc~ppa3 in copy of the server where i initially discovered the issue make it not reproducible anymore.

If you need more testing later as you indirectly said in #27 just let me know i keep that testing instance around.

@paelzer: Sorry for not replying to your questions yesterday only got back to check on the issue today.

[Andreas Hasenack (ahasenack)]

Thanks TJ, Stefan and Christian

I'm cleaning up the packaging, properly formatting the patches, and doing a quick check if I really need both patches or just the last one I added. I'll post updates here, and then prepare an SRU.

[Andreas Hasenack (ahasenack)]

Only https://github.com/apache/httpd/commit/7fa21ea6602b30cc43d4f485777545dd73bb25a6 is needed to fix this issue reported here, but I'll include https://github.com/apache/httpd/commit/524608b65ec410e797a7283e6e142f8e5a74be26 as well since it's a change regarding openssl 1.1.1 that might bite us in the future. Unless testing reveals another surprise.

[Andreas Hasenack (ahasenack)]

I suddenly realized cosmic is EOL. I'll let the sru team weigh in if this fix can still be published for cosmic or not.

[Steve Langasek (vorlon)]

Because it is an SRU regression (apache2 2.4.34-1ubuntu2.2 introduced the same change to cosmic that apache2 2.4.29-1ubuntu4.7 did for bionic), I would accept a fix for cosmic for expedited SRU verification despite the EOL being in 2 days time.

[Andreas Hasenack (ahasenack)]

Uploaded to cosmic and bionic proposed queues, unapproved.

[Steve Langasek (vorlon)]

Hello Stefan, or anyone else affected,

Accepted apache2 into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.34-1ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Stefan, or anyone else affected,

Accepted apache2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.29-1ubuntu4.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Giraffe (dodger-forum)]

Giraffe here,

I've just tested the following packages on our server:
ii apache2 2.4.29-1ubuntu4.8 amd64 Apache HTTP Server
ii apache2-bin 2.4.29-1ubuntu4.8 amd64 Apache HTTP Server
ii apache2-data 2.4.29-1ubuntu4.8 all Apache HTTP Server
ii apache2-utils 2.4.29-1ubuntu4.8 amd64 Apache HTTP Server

The issue with ssllabs and the CPU usage is fixed/gone everything else is working fine no new regressions noted.

Thanks alot!

[Andreas Hasenack (ahasenack)]

Cosmic verification

First, reproducing the bug with:
 *** 2.4.34-1ubuntu2.2 500
        500 http://us.archive.ubuntu.com/ubuntu cosmic-updates/main amd64 Packages

Then I enabled the required modules, restarted apache, kept monitoring cpu usage per process using top, and asked ssllabs for the test run

After the test finished, I still had apache processes using as much cpu as they could:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 4709 www-data 20 0 759644 7616 5292 S 48.5 0.8 1:46.23 apache2
 4710 www-data 20 0 759448 5068 2988 S 48.5 0.5 1:55.00 apache2

Installing the packages from proposed now:
 *** 2.4.34-1ubuntu2.3 500
        500 http://us.archive.ubuntu.com/ubuntu cosmic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.34-1ubuntu2.2 500
        500 http://us.archive.ubuntu.com/ubuntu cosmic-updates/main amd64 Packages

Hitting "clear cache" in the ssl labs report page triggers a re-run.

Test finished, and no sign of apache processes near the top:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 4818 root 20 0 13808 4016 2848 S 1.0 0.4 0:05.14 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc new-session -n - /usr/bin/byobu-shell
    1 root 20 0 123356 9388 6752 S 0.0 0.9 0:06.27 /sbin/init

Cosmic verification succeeded.

[Stefan Huehner (stefan-huehner)]

Bionic-verification: done and fine

Using *** 2.4.29-1ubuntu4.8 100
        100 /var/lib/dpkg/status

from bionic-proposed i can no longer reproduce any of the bad effects (apache2 processes left with 100% cpu or left-over tcp connections).

Also we updated a single production machine to let it soak with real traffic and did not find any other bad behavior so far.

Thanks everyone for their work to get this fixed so quickly!

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.34-1ubuntu2.3

---------------
apache2 (2.4.34-1ubuntu2.3) cosmic; urgency=medium

  * d/p/ssl-read-rc-value-openssl-1.1.1.patch: Handle SSL_read() return code 0
    similarly to <0 with openssl 1.1.1
  * d/p/clear-retry-flags-before-abort.patch: clear retry flags before
    aborting on client-initiated reneg (LP: #1836329)

 -- Andreas Hasenack <email address hidden> Tue, 16 Jul 2019 17:27:06 -0300

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for apache2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.29-1ubuntu4.8

---------------
apache2 (2.4.29-1ubuntu4.8) bionic; urgency=medium

  * d/p/ssl-read-rc-value-openssl-1.1.1.patch: Handle SSL_read() return code 0
    similarly to <0 with openssl 1.1.1
  * d/p/clear-retry-flags-before-abort.patch: clear retry flags before
    aborting on client-initiated reneg (LP: #1836329)

 -- Andreas Hasenack <email address hidden> Tue, 16 Jul 2019 15:14:45 -0300