Rebuild haproxy with openssl 1.1.1 will change features (bionic)

Bug #1841936 reported by David Hedberg on 2019-08-29
20
This bug affects 1 person
Affects Status Importance Assigned to Milestone
HAProxy
New
Unknown
haproxy (Ubuntu)
Medium
Unassigned
Bionic
Medium
Ubuntu Security Team

Bug Description

[Impact]

 * openssl 1.1.1 has been backported to Bionic for its longer
   support upstream period

 * That would allow the extra feature of TLSv1.3 in some consuming
   packages what seems "for free". Just with a no change rebuild it would
   pick that up.

[Test Case]

 * run "haproxy -vv" and check the reported TLS versions to include 1.3

[Regression Potential]

 * This should be low, the code already runs against the .so of the newer
   openssl library. This would only make it recognize the newer TLS
   support.
   i'd expect more trouble as-is with the somewhat big delta between what
   it was built against vs what it runs with than afterwards.
 * [1] and [2] indicate that any config that would have been made for
   TLSv1.2 [1] would not apply to the v1.3 as it would be configured in
   [2].
   It is good to have no entry for [2] yet as following the defaults of
   openssl is the safest as that would be updated if new insights/CVEs are
   known.
   But this could IMHO be the "regression that I'd expect", one explcitly
   configured the v1.2 things and once both ends support v1.3 that might
   be auto-negotiated. One can then set "force-tlsv12" but that is an
   administrative action [3]
 * Yet AFAIK this fine grained control [2] for TLSv1.3 only exists in
   >=1.8.15 [4] and Bionic is on haproxy 1.8.8. So any user of TLSv1.3 in
   Bionic haproxy would have to stay without that. There are further
   changes to TLS v1.3 handling enhancements [5] but also fixes [6] which
   aren't in 1.8.8 in Bionic.
   So one could say enabling this will enable an inferior TLSv1.3 and one
   might better not enable it, for an SRU the bar to not break old
   behavior is intentionally high - I tried to provide as much as possible
   background, the decision is up to the SRU team.

[1]: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.1-ssl-default-bind-ciphers
[2]: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.1-ssl-default-bind-ciphersuites
[3]: https://www.haproxy.com/documentation/hapee/1-8r2/traffic-management/tls/#define-bind-directives-on-the-frontend
[4]: https://github.com/haproxy/haproxy/blob/master/CHANGELOG#L2131
[5]: https://github.com/haproxy/haproxy/commit/526894ff3925d272c13e57926aa6b5d9d8ed5ee3
[6]: https://github.com/haproxy/haproxy/commit/bc34cd1de2ee80de63b5c4d319a501fc0d4ea2f5

[Other Info]

 * If this is nack'ed we will need an upload that prevents to enable
   TLSv1.3 to avoid enabling it by accident on e.g. a security update.

---

haproxy needs to be rebuilt after #1797386 to take advantage of TLSv1.3.

(If that's not desirable for some reason, then maybe TLSv1.3 should be actively disabled to avoid any surprises in case of a future bug fix release.)

---

Output of haproxy -vv with stock package:

Built with OpenSSL version : OpenSSL 1.1.0g 2 Nov 2017
Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018 (VERSIONS DIFFER!)
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2

---

Output after rebuilding the package from source:

Built with OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3

CVE References

Hi David,
this would be like [1] doen by xnox who also was leading the overall 1.1.1 SRU activity IIRC:
I subscribed him to share his two cents.

@xnox - was there a specific reason not to do this rebuild for Bionic after the openssl SRU?

[1]: https://launchpad.net/ubuntu/+source/haproxy/1.8.13-2build1

Changed in haproxy (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Dimitri John Ledkov (xnox) wrote :

rebuilding packages to enable TLSv1.3 is out of scope for the openssl SRU and is introudction of new features.

The point of the OpenSSL 1.1.1 SRU was to bring bionic onto using an LTS version of openssl to relief the maintainance burden (security, fips, SRU) for the next 8.5 years, given that 1.1.0 is not a widely adopted version of OpenSSL and is going to go end of life upstream in two weeks time.

And just to be clear, this is not a precedent, and no further major upgrades to OpenSSL are planned in Ubuntu.

Dimitri John Ledkov (xnox) wrote :

@paelzer if you wish to rebuild haproxy, feel free to start SRU to achieve that, but it is out of scope for me / foundations / security to do so.

On Fri, Aug 30, 2019 at 1:00 PM Dimitri John Ledkov
<email address hidden> wrote:
>
> @paelzer if you wish to rebuild haproxy, feel free to start SRU to
> achieve that, but it is out of scope for me / foundations / security to
> do so.

Perfect, thanks Dimitri for the explanation!
I mostly wanted to avoid that there are explicit, yet unknown to me,
reasons for not doing so which was answered.

tags: added: server-triage-discuss
tags: removed: server-triage-discuss
summary: - Rebuild haproxy with openssl 1.1.1 (bionic)
+ Rebuild haproxy with openssl 1.1.1 will change features (bionic)

We talked about it, and strictly speaking this could be anywhere from.

a) yay a new feature for free for just a no change rebuild
to
b) this new feature is violating the SRU policy and not allowed to happen, we need to upload code that avoids that any coming e.g. security fix will switch it on unplanned.

And there is a lot of middle ground between (a) and (b) to bikeshed forever.

We discussed this in the Server Team and came to the point that we are ok either way (enabling it now or uploading a "avoid this to be activated" change). What we'd want to avoid is that this just hangs around and will be enabled "by accident" on any other upload.

For the decision if we want to push for

@Security Team:
- (we heard Dimitri in comment #2) from securities POV, was there a general plan around the new openssl and TLS to enable it in Bionic throughout packages?
- if there is a bigger plan does it come with a reference we can use as argument for the SRU to convince the SRU team that the change is safe and required?
- what would be your security POV guidance for this case here, should we rebuild and enable or upload a change to avoid enabling it in Bionic at all?
- are there other packages known which might need the same treatment?

I talked with Alex of the security Team.
Here the TL;DR summary:
- security would prefer and be +1 on enabling TLSv1.3 in haproxy in Bionic
  - Server team is ok as well, while it is a feature addition it seems not to take away any
  - thereby it would fall under the third section of [1] "add features without affecting existing
    features"
- In case the SRU Team "nacks" this upload then instead we should prepare and upload a change to
  "avoid to enable TLSv1.3 by accident"

I checked later releases, >=Disco are already built with the new version so no other than Bionic would need to be changed.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates#Other_safe_cases

Changed in haproxy (Ubuntu):
status: Confirmed → Triaged
Changed in haproxy (Ubuntu Bionic):
status: New → Triaged
Changed in haproxy (Ubuntu):
status: Triaged → Fix Released
Changed in haproxy (Ubuntu Bionic):
importance: Undecided → Medium
assignee: nobody → Christian Ehrhardt  (paelzer)

Prepped the SRU Template and found a few potential tricky bits while summarizing, I'll now bring this to the attention of the SRU Team to make a call.

description: updated
description: updated
Changed in haproxy (Ubuntu Bionic):
assignee: Christian Ehrhardt  (paelzer) → nobody
Robie Basak (racb) wrote :

The SRU team discussed this last week and we agreed that enabling TLS 1.3 is appropriate.

> [Test Case]

> * run "haproxy -vv" and check the reported TLS versions to include 1.3

I think we should additionally check during SRU verification that TLS functionality is working correctly, both with an existing protocol and also with 1.3, as we know we're perturbing it.

Changed in haproxy (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic

Hello David, or anyone else affected,

Accepted haproxy into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.8.8-1ubuntu0.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The basic check works as expected.

Pre:
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2

Upgrade to proposed worked without any issue.

Post:
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3

@David and maybe others could you try the version from proposed in a real setup to be more confident that this didn't by accident break something? Please report here so that we then can mark this verified.

I have done some preliminary testing with 1.8.8-1ubuntu0.5, and most things look good.

However, in our case we have an old (external) client using java6 that we sadly still need to support for a while longer. Using the connection simulation in testssl.sh (and also ssllabs) I can see that connections from java6 now fails with our configuration where it previously succeeded.

I suspect that this is not (directly) related to TLSv1.3. The problem with java6 is usually that it only supports dh parameters with 1024 bits (and TLSv1.0).

According to testssl.sh the dh-parameters offered now is:
 DH group offered: RFC3526/Oakley Group 15 (3072 bits)

Before the upgrade it was:
 DH group offered: HAProxy (1024 bits)

I have tried generating custom dh parameters with 1024 bits and specifying them both with the default ssl-dh-param-file setting and directly in the certificate file. I have also tried disabling TLSv1.3 (using no-tlsv13). Neither seem to help.

Thanks a lot David for testing.
It is important that we track this down and make sure it is not introducing a regression.

The related haproxy config would be:
"tune.ssl.default-dh-param <number>"
Sets the maximum size of the Diffie-Hellman parameters used to generate the ephemeral/temporary Diffie-Hellman key when using DHE key exchange. Default <number> value is 1024. Higher values increase CPU load and may not be supported by some clients (IE:Java 7).

That exactly is our case, and /etc/haproxy/haproxy.cfg hasn't changed.
In the default config this argument isn't set so it should still be 1024.

I set up an apache2 with (self signed) certs and enabled TLS1.0. That is old, but what you reported for your compat needs.
First I tested the apache itself with testssl.sh.
Your output in regard to the DH length seems to be from the PFS section.
I get here:
 DH group offered: RFC3526/Oakley Group 14 (2048 bits)

Then I was configuring haproxy+ssl (pre-SRU) in front of apache2 and it gave me

Without the setting above haproxy gives me a nice warning of:
[WARNING] 269/070510 (32043) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.

In regard to DH length in PFS in testssl both showed me:
 DH group offered: HAProxy (1024 bits)

That is just as expected for now.
Upgrading to haproxy from proposed.

The PFS section now really did get bumped, in my case to:
 DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)

The full diff of old/new haproxy build is
+ TLSv1.3 (good)
+ PFS (critical for the SRU):
  - changes cipher list
  - extened Elliptic curves
  - increased DH size 1024->2048
+ Some noise due to TLSv1.3 (ok)

One diff element also might shows the a potential this increased.
old:
 LOGJAM (CVE-2015-4000), experimental VULNERABLE (NOT ok): common prime: HAProxy (1024 bits),
new:
 LOGJAM (CVE-2015-4000), experimental common prime with 2048 bits detected: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits),
The USN [1] indicated this was solved by disabling "DH EXPORT ciphers", so it shouldn't be a real issue.
But chances are that the newer openssl has a new "minimum DH size" for security reasons in regard to CVE-2015-4000.

Checking workaround by setting an explicit max key in the config ...
=> didn't work still size 2048.
So there is a new minimum in place enforced by the new openssl since/due-to the rebuild.

Attaching the testssl logs ...

@Ubuntu-security: is the bump of offered DH keys something known/expected with rebuilds against the newer OpenSSL for the CVE found or others?
If it is, is this a wanted change considering the breakage of some old clients as reported above?

[1]: https://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-4000.html

Changed in haproxy (Ubuntu Bionic):
assignee: nobody → Ubuntu Security Team (ubuntu-security)

Assigned to ubuntu-security for their guidance (as outlined and asked in comment #12) on the increased minimum DH key size for:

a) this particular case in haproxy

but also

b) potential openssl 1.1.1 rebuilds in general

tags: added: server-triage-discuss
Robie Basak (racb) on 2019-10-08
tags: added: bionic-openssl-1.1

In [1] a workaround for those (hopefully a few) installations that need the lower (not recommended) key size was suggested.

Again this isn't what "should be done", but what users could do if affected.
The reason not to do it is:
a) LOGJAM (CVE-2015-4000), common prime: HAProxy (1024 bits)
b) due to that haproxy always warned you: "[WARNING] 286/090504 (13834) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear."

So we actually want and think it is good that the default size got bumped.
But people that need the old smaller key size (like David in comment #11) need to have a way to drop back to the old key size.

With haproxy and testssl I checked (again) Bionic as-is:
updates (1.8.8-1ubuntu0.4)
 DH group offered: HAProxy (1024 bits)
proposed (1.8.8-1ubuntu0.5):
 DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)

proposed (1.8.8-1ubuntu0.5) + seclevel 0/1 config:
 :-/ I Failed to convince haproxy from using any config in that regard.
 Tried several combinations of "CipherString = DEFAULT@SECLEVEL=0" in /etc/ssl/openssl.cnf
 Tried to get it into /etc/haproxy/haproxy.cfg
 But none worked to get the key size down

This might be my lack of haproxy experience.
I tried some combinations that came to my mind and asked in the IRC channel or haproxy, but got no response. If someone could take a look that knows haproxy better. Maybe we can release this fix for TLSv1.3 IF there is a configuration workaround for the few people that need the small (vulnreable) key size.

[1]: https://lists.ubuntu.com/archives/ubuntu-devel/2019-October/040814.html

I found this example for apache2:
SSLCipherSuite @SECLEVEL=2:kEECDH:kRSA:kEDH:kPSK:kDHEPSK:kECDHEPSK:-aDSS:-3DES:!DES:!RC4:!RC2:!IDEA:-SEED:!eNULL:!aNULL:!MD5:-SHA384:-CAMELLIA:-ARIA:-AESCCM8

Which reads similar to the default haproxy config:
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

now modified to
ssl-default-bind-ciphers @SECLEVEL=0:ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

But even that (along all the other combinations that felt even less appropriate) worked.
I always get the 2048 bit key now :-/

haproxy IRC replied (thanks) in the meantime and suggested [1] so I'm giving that a try now ...

[1]: https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.1-ssl-dh-param-file

At first it rejected my config :
Oct 14 10:02:38 b haproxy[27966]: [ALERT] 286/100238 (27966) : parsing [/etc/haproxy/haproxy.cfg:22] : 'ssl-dh-param-file': unable to load DH parameters from file </etc/haproxy/ssl.cfg>.
Oct 14 10:02:38 b haproxy[27966]: [ALERT] 286/100238 (27966) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg

Following the addition of the feature [1] I found [2]. From there I experimented with pem files.

# Create small DH parameter:
openssl dhparam -out /etc/haproxy/ssl.pem 1024
# Config haproxy to use that one in /etc/haproxy/haproxy.cfg:
ssl-dh-param-file /etc/haproxy/ssl.pem

But all of that seems like defining static DH params for haproxy.
And the key used in PFS is about an "ephemeral DH key" right?
I still struggle to convince haproxy to do what I want :-/

Everyone with more haproxy knowledge is welcome to give this a try ...

[1]: https://<email address hidden>/msg18086.html
[2]: https://www.openssl.org/docs/man1.0.2/man3/PEM_read_bio_DHparams.html

Hmm ... "the default DH parameters that are used during the SSL/TLS handshake when
ephemeral Diffie-Hellman (DHE) key exchange is used" so the documentation agrees that this is for the DHE.

But as mentioned so far I fail to put something in there that makes testssl to report 1024 bit.

./testssl.sh --pfs 10.253.194.137:443
...
 DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)

From upstream:
[15:47] <jethro> cpaelzer: we have investigated a bit and think there is an problem with how haproxy handles dhparam < 2048 with openssl 1.1.1. This will be further looked at and hopefuly we find a solution to allow 1024 dhparam. thanks for the report.

I'd say we call this verification failed for now, then we need to sort out this new extra bug.
Fix that extra bug from Ubuntu 20.04 downwards to Bionic and then on Bionic can along that fix this request to rebuild it for TLSv1.3

tags: added: verification-failed verification-failed-bionic
removed: server-triage-discuss verification-needed verification-needed-bionic

After further IRC discussion reported upstream in https://github.com/haproxy/haproxy/issues/324 as we'd want to either know how to config and/or a fix to be able to get the key size down when built against openssl 1.1.1

Changed in haproxy:
status: Unknown → New

Test with that patch worked, we now have to wait until upstream really accepts and until they backport it to 1.8.x for us to be able to pick it up.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.