backport mod_reqtimeout with handshake support

[Jesse Williamson]

[Impact]

When running TCP Defensics suite which sends corrupt packages towards vip__public port 443, the suite is hanging after the half suite because there are no free connections. The connections will be in state "established" ~ 2 hours.

1.2. Detailed trouble description
# ip netns exec haproxy netstat -npea | grep XXX.XXX.XXX.XXX | grep -i establish | grep 443
tcp 0 0 XXX.XXX.XXX.XXX:443 YYY.YY.YYY.YY:2940 ESTABLISHED 115 81148003 29817/haproxy
tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:24979 ESTABLISHED 115 81802005 29817/haproxy
tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:19394 ESTABLISHED 115 81782263 29817/haproxy
tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:13931 ESTABLISHED 115 81752052 29817/haproxy
tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:12668 ESTABLISHED 115 81743719 29817/haproxy
tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:2961 ESTABLISHED 115 81139548 29817/haproxy
tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:8918 ESTABLISHED 115 81738132 29817/haproxy
tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:2957 ESTABLISHED 115 81148041 29817/haproxy
tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:10552 ESTABLISHED 115 81744903 29817/haproxy

This issue can be resolved by enabling the parameter(mod_reqtimeout). This parameter is available in apache 2.4.39 (released on 2019-04-01).

[Test Case]

This test case has been brought to my attention by an impacted user:
"
You must have an apache2 server, with an haproxy in front of it, and you initiate SSL connections with "nc" between 50 and 8000 connections and because the SSL connection process is never finished all those connections get stucked and never timeout.
"

Reproducer (Thanks to Szilard): https://pastebin.ubuntu.com/p/6Hk64CDc7H/

[Regression Potential]

* The backport already exist in Bionic/Disco (done by security team via the security channel)

* It is also backported upstream into 2.4 (branch : 2.4.x)

* It was tested pre-release by an impacted user, and the outcome was positive:

"I have tested the below packages for enabling handshake parameter(mod_reqtimeout) in apache. Looks the package is working fine. "

* Local autopkgtest inside qemu, revealed no issues:
autopkgtest [12:09:48]: @@@@@@@@@@@@@@@@@@@@ summary
duplicate-module-load PASS
htcacheclean PASS
ssl-passphrase PASS
chroot PASS

[Other Info]

[Original description]
Backport the handshake feature in mod_reqtimeout (in Apache 2.4.39) to Apache 2.4.18.

Lack of this feature was exhausting free connections when sent corrupted packets.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "apache2_modreqtimeout.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Łukasz Zemczak (sil2100)]

Originally I was a bit confused since I didn't see the changes as being part of any -security updates for disco and bionic. But when looking at the source for both series, the changes seemed to be there. I guess it might be included in their upstream releases then.

[Łukasz Zemczak (sil2100)]

Hello Jesse, or anyone else affected,

Accepted apache2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.18-2ubuntu3.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Eric Desrochers (slashd)]

[VERIFICATION XENIAL]

* Feedback #1:

From an impacted user:
"
They confirmed that from their perspective the test is OK, and the apache2 packages are delivering expected result
"

* Feedback #2:
From SustEng Mauricio (mfo):
"
The backport in xenial-proposed worked exactly as eoan
(with the AcceptFilter bits mentioned in previous comment)
...
"

[Eric Desrochers (slashd)]

[VERIFICATION XENIAL - Part 2]
* Feedback #3:
"
I also tested and now it works perfectly, I can count the seconds which I configure for the handshake timeout and the connection is terminated exactly when the handshake timeout expires

Great job!
"

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.18-2ubuntu3.14

---------------
apache2 (2.4.18-2ubuntu3.14) xenial; urgency=medium

  * Backport mod_reqtimeout with handshake support (LP: #1846138)
    - d/p/0001-mod-reqtimeout-revent-long-response-times.patch
    - d/p/0002-mod_reqtimeout-fix-body-timeout-disabling-for-CONNECT-request.patch
    - d/p/0003-mod_reqtimeout-Merge-r1853901-r1853906-r1853908-r1853929-r1853935-r.patch

 -- Jesse Williamson <email address hidden> Tue, 08 Oct 2019 13:31:25 +0000

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for apache2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.