400-Bad-request responses (possibly due to failing authority validation)

[phyphor]

We recently upgraded all our proxy servers (running squid and haproxy) to Ubuntu 22.04.2 LTS after a squid bug was resolved (ubuntu/+source/squid/+bug/1989380) and we appear to have found a bug with haproxy where some, but not all proxied requests fail.

In particular we have found that setting the proxy in Windows to the haproxy port (8080) on the server it seems to work fine for all modern browsers but older browsers (including IE) fail, as too does SQL Server 2017, Visual Studio 2017, and Maltego 4.3.0. When they fail they report they are unable to access the servers on the internet, which is necessary for them to do things like register/authenticate credentials for use. When we use the squid port (3128) itself all applications work fine, so we believe the problem is with haproxy.

(As an aside, we are not able to upgrade the individual pieces of software for two reasons, the first is that they are specific versions referenced & required for use, and secondly there are too many different instances of software, across multiple OSes, to catch them all.)

We have a workaround in place but I believe it is appropriate to report here so that it can be updated & patched if deemed suitable.
The workaround is adding the following to our haproxy.cfg:
    option accept-invalid-http-request

The version of haproxy in the latest LTS release of Ubuntu (22.04.2 LTS) is 2.4.18-0ubuntu1.2 and our investigations lead us to believe that this is due to a known bug in this version (which is where we discovered the wrkaround), as seen here: https://github.com/haproxy/haproxy/issues/1761

This bug has been patched in later versions of haproxy with this commit: https://github.com/haproxy/haproxy/commit/3f5fbe940733bba84b5ee875af5b13aa3144aa41

We are able to spin up additional servers to test with PPA releases if this would help, as I did for the aforementioned squid bug.

Please let me know if more information would be hlepful.

Segments of apport included below, but I can provide the full file (with some redactions for security/privacy) if necessary.

== ApportVersion =================================
2.20.11-0ubuntu82.3

== Architecture =================================
amd64

== CasperMD5CheckResult =================================
pass

== Date =================================
Thu Mar 30 11:03:12 2023

== DistroRelease =================================
Ubuntu 22.04

== InstallationDate =================================
Installed on 2021-11-22 (492 days ago)

== InstallationMedia =================================
Ubuntu-Server 20.04.3 LTS "Focal Fossa" - Release amd64 (20210824)

== Package =================================
haproxy 2.4.18-0ubuntu1.2

== PackageArchitecture =================================
amd64

== ProblemType =================================
Bug

== ProcVersionSignature =================================
Ubuntu 5.15.0-67.74-generic 5.15.85

== SourcePackage =================================
haproxy

== Tags =================================
 jammy uec-images

== Uname =================================
Linux 5.15.0-67-generic x86_64

== UpgradeStatus =================================
Upgraded to jammy on 2023-02-24 (33 days ago)

[Lena Voytek (lvoytek)]

Thank you for the bug report. I created a PPA based on the upstream information you provided, located here: https://launchpad.net/~lvoytek/+archive/ubuntu/haproxy-fix-bad-request-responses

It is based on the commits:

https://github.com/haproxy/haproxy/commit/3f5fbe940733bba84b5ee875af5b13aa3144aa41
https://github.com/haproxy/haproxy/commit/658f971621839f3b928da099dfe3092b47cbc958
https://github.com/haproxy/haproxy/commit/ca7218aaf073627b665459bd881b2b35a481602a

If you would like to test it with 22.04 you can run:

sudo add-apt-repository ppa:lvoytek/haproxy-fix-bad-request-responses
sudo apt update
sudo apt upgrade

This should affect kinetic too, but the fix has been added to lunar. I'll mark the bug info as such

[phyphor (phyphor)]

I've deployed this to one of our servers and it looks to have successfully resolved the issue. We're leaving it for a while to make sure nothing else breaks, but I should be able to provide an update soon.

[phyphor (phyphor)]

Things seem fine but 2.4.18-0ubuntu1.3, which doesn't appear to have the fix in place, has been released on the normal repos at around the same time so we are now getting that instead of the fixed 2.4.18-0ubuntu1.3~ppa3 which has interrupted our testing.

[Lena Voytek (lvoytek)]

Sorry for the bad timing. The PPA has been updated to the latest release now

[phyphor (phyphor)]

It's been happily deployed now for over a week, without needing the workaround, and we've had no issues with it.

Do you need anything else from me to get this into SRU?

[Lena Voytek (lvoytek)]

That's great to hear! Currently this bug fix is waiting on an upcoming version update - (LP: #2012557)
Looking through the changelog, this issue should actually be fixed in version 2.4.19. Once the update to 2.4.22 is released, if you're able to test it and see the issue no longer exists, please let me know!
Thanks!

[phyphor (phyphor)]

That works. I'll hold on until then.

[phyphor (phyphor)]

Just performed some quick testing with 2.4.22-0ubuntu0.22.04.1 and this also seems to have resolved the issue. Thanks for your prompt attention on this.

[Lena Voytek (lvoytek)]

Forgot to mark this as fix released with the new update, doing that now. Thanks for the verification!