Rebuild openssl 1.1.1 to pickup TLSv1.3 (bionic) and unbreak existing builds against 1.1.1 (dh key size)

Bug #1841936 reported by David Hedberg
28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
HAProxy
Fix Released
Unknown
haproxy (Ubuntu)
Fix Released
Medium
Christian Ehrhardt 
Bionic
Fix Released
Medium
Christian Ehrhardt 
Disco
Fix Released
Undecided
Christian Ehrhardt 
Eoan
Fix Released
Undecided
Christian Ehrhardt 
Focal
Fix Released
Medium
Christian Ehrhardt 

Bug Description

[Impact-Bionic]

 * openssl 1.1.1 has been backported to Bionic for its longer
   support upstream period

 * That would allow the extra feature of TLSv1.3 in some consuming
   packages what seems "for free". Just with a no change rebuild it would
   pick that up.

[Impact Disco-Focal]

 * openssl >=1.1.1 is in Disco-Focal already and thereby it was built
   against that already. That made it pick up TLSv1.3, but also a related
   bug that broke the ability to control the DHE key, it was always in
   "ECDH auto" mode. Therefore the daemon didn't follow the config
   anymore.
   Upgraders would regress having their DH key behavior changed
   unexpectedly.

[Test Case]

 A)
 * run "haproxy -vv" and check the reported TLS versions to include 1.3
 B)
 * download https://github.com/drwetter/testssl.sh
 * Install haproxy
   * ./testssl.sh --pfs <localip>:443
   * Check the reported DH key/group (shoudl stay 1024)
   * Check if settings work to bump it like
       tune.ssl.default-dh-param 2048
     into
       /etc/haproxy/haproxy.cfg

[Regression Potential-Bionic]

 * This should be low, the code already runs against the .so of the newer
   openssl library. This would only make it recognize the newer TLS
   support.
   i'd expect more trouble as-is with the somewhat big delta between what
   it was built against vs what it runs with than afterwards.
 * [1] and [2] indicate that any config that would have been made for
   TLSv1.2 [1] would not apply to the v1.3 as it would be configured in
   [2].
   It is good to have no entry for [2] yet as following the defaults of
   openssl is the safest as that would be updated if new insights/CVEs are
   known.
   But this could IMHO be the "regression that I'd expect", one explcitly
   configured the v1.2 things and once both ends support v1.3 that might
   be auto-negotiated. One can then set "force-tlsv12" but that is an
   administrative action [3]
 * Yet AFAIK this fine grained control [2] for TLSv1.3 only exists in
   >=1.8.15 [4] and Bionic is on haproxy 1.8.8. So any user of TLSv1.3 in
   Bionic haproxy would have to stay without that. There are further
   changes to TLS v1.3 handling enhancements [5] but also fixes [6] which
   aren't in 1.8.8 in Bionic.
   So one could say enabling this will enable an inferior TLSv1.3 and one
   might better not enable it, for an SRU the bar to not break old
   behavior is intentionally high - I tried to provide as much as possible
   background, the decision is up to the SRU team.

[Regression Potential-Disco-Focal]

 * The fixes let the admin regain control of the DH key configuration
   which is the fix. But remember that the default config didn't specify
   any. Therefore we have two scenarios:
   a) an admin had set custom DH parameters which were ignored. He had no
      chance to control them and needs the fix. He might have been under
      the impression that his keys are safe (there is a CVE against small
      ones) and only now is he really safe -> gain high, regression low
   b) an admin had not set anything, the default config is meant to use
      (compatibility) and the program reported "I'm using 1024, but you
      should set it higher". But what really happened was ECDH auto mode
      which has longer keys and different settings. Those systems will
      be "fixed" by finally following the config, but that means the key
      will "now" after the fix be vulnerable.
      -> for their POV a formerly secure setup will become vulnerable
   I'd expect that any professional setup would use explicit config as it
   has seen the warning since day #1 and also any kind of deployment
   recipes should use big keys. So the majority of users should be in (a).
   c) And OTOH there are people like the reporter who strictly NEED the
      key to be small for their devices they have to support. They will
      finally be able to use the small keys again which formerly was
      impossible.

   Summary: (a) good (c) good, (b) good, but a regression in some POV

[1]: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.1-ssl-default-bind-ciphers
[2]: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.1-ssl-default-bind-ciphersuites
[3]: https://www.haproxy.com/documentation/hapee/1-8r2/traffic-management/tls/#define-bind-directives-on-the-frontend
[4]: https://github.com/haproxy/haproxy/blob/master/CHANGELOG#L2131
[5]: https://github.com/haproxy/haproxy/commit/526894ff3925d272c13e57926aa6b5d9d8ed5ee3
[6]: https://github.com/haproxy/haproxy/commit/bc34cd1de2ee80de63b5c4d319a501fc0d4ea2f5

[Other Info]

 * If this is nack'ed we will need an upload that prevents to enable
   TLSv1.3 to avoid enabling it by accident on e.g. a security update.

---

haproxy needs to be rebuilt after #1797386 to take advantage of TLSv1.3.

(If that's not desirable for some reason, then maybe TLSv1.3 should be actively disabled to avoid any surprises in case of a future bug fix release.)

---

Output of haproxy -vv with stock package:

Built with OpenSSL version : OpenSSL 1.1.0g 2 Nov 2017
Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018 (VERSIONS DIFFER!)
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2

---

Output after rebuilding the package from source:

Built with OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3

Related branches

CVE References

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi David,
this would be like [1] doen by xnox who also was leading the overall 1.1.1 SRU activity IIRC:
I subscribed him to share his two cents.

@xnox - was there a specific reason not to do this rebuild for Bionic after the openssl SRU?

[1]: https://launchpad.net/ubuntu/+source/haproxy/1.8.13-2build1

Changed in haproxy (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

rebuilding packages to enable TLSv1.3 is out of scope for the openssl SRU and is introudction of new features.

The point of the OpenSSL 1.1.1 SRU was to bring bionic onto using an LTS version of openssl to relief the maintainance burden (security, fips, SRU) for the next 8.5 years, given that 1.1.0 is not a widely adopted version of OpenSSL and is going to go end of life upstream in two weeks time.

And just to be clear, this is not a precedent, and no further major upgrades to OpenSSL are planned in Ubuntu.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

@paelzer if you wish to rebuild haproxy, feel free to start SRU to achieve that, but it is out of scope for me / foundations / security to do so.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote : Re: [Bug 1841936] Re: Rebuild haproxy with openssl 1.1.1 (bionic)

On Fri, Aug 30, 2019 at 1:00 PM Dimitri John Ledkov
<email address hidden> wrote:
>
> @paelzer if you wish to rebuild haproxy, feel free to start SRU to
> achieve that, but it is out of scope for me / foundations / security to
> do so.

Perfect, thanks Dimitri for the explanation!
I mostly wanted to avoid that there are explicit, yet unknown to me,
reasons for not doing so which was answered.

tags: added: server-triage-discuss
tags: removed: server-triage-discuss
summary: - Rebuild haproxy with openssl 1.1.1 (bionic)
+ Rebuild haproxy with openssl 1.1.1 will change features (bionic)
Revision history for this message
Christian Ehrhardt  (paelzer) wrote : Re: Rebuild haproxy with openssl 1.1.1 will change features (bionic)

We talked about it, and strictly speaking this could be anywhere from.

a) yay a new feature for free for just a no change rebuild
to
b) this new feature is violating the SRU policy and not allowed to happen, we need to upload code that avoids that any coming e.g. security fix will switch it on unplanned.

And there is a lot of middle ground between (a) and (b) to bikeshed forever.

We discussed this in the Server Team and came to the point that we are ok either way (enabling it now or uploading a "avoid this to be activated" change). What we'd want to avoid is that this just hangs around and will be enabled "by accident" on any other upload.

For the decision if we want to push for

@Security Team:
- (we heard Dimitri in comment #2) from securities POV, was there a general plan around the new openssl and TLS to enable it in Bionic throughout packages?
- if there is a bigger plan does it come with a reference we can use as argument for the SRU to convince the SRU team that the change is safe and required?
- what would be your security POV guidance for this case here, should we rebuild and enable or upload a change to avoid enabling it in Bionic at all?
- are there other packages known which might need the same treatment?

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I talked with Alex of the security Team.
Here the TL;DR summary:
- security would prefer and be +1 on enabling TLSv1.3 in haproxy in Bionic
  - Server team is ok as well, while it is a feature addition it seems not to take away any
  - thereby it would fall under the third section of [1] "add features without affecting existing
    features"
- In case the SRU Team "nacks" this upload then instead we should prepare and upload a change to
  "avoid to enable TLSv1.3 by accident"

I checked later releases, >=Disco are already built with the new version so no other than Bionic would need to be changed.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates#Other_safe_cases

Changed in haproxy (Ubuntu):
status: Confirmed → Triaged
Changed in haproxy (Ubuntu Bionic):
status: New → Triaged
Changed in haproxy (Ubuntu):
status: Triaged → Fix Released
Changed in haproxy (Ubuntu Bionic):
importance: Undecided → Medium
assignee: nobody → Christian Ehrhardt  (paelzer)
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Prepped the SRU Template and found a few potential tricky bits while summarizing, I'll now bring this to the attention of the SRU Team to make a call.

description: updated
description: updated
Changed in haproxy (Ubuntu Bionic):
assignee: Christian Ehrhardt  (paelzer) → nobody
Revision history for this message
Robie Basak (racb) wrote :

The SRU team discussed this last week and we agreed that enabling TLS 1.3 is appropriate.

> [Test Case]

> * run "haproxy -vv" and check the reported TLS versions to include 1.3

I think we should additionally check during SRU verification that TLS functionality is working correctly, both with an existing protocol and also with 1.3, as we know we're perturbing it.

Changed in haproxy (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello David, or anyone else affected,

Accepted haproxy into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.8.8-1ubuntu0.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Christian Ehrhardt  (paelzer) wrote : Re: Rebuild haproxy with openssl 1.1.1 will change features (bionic)

The basic check works as expected.

Pre:
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2

Upgrade to proposed worked without any issue.

Post:
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3

@David and maybe others could you try the version from proposed in a real setup to be more confident that this didn't by accident break something? Please report here so that we then can mark this verified.

Revision history for this message
David Hedberg (david-hedberg-t) wrote :

I have done some preliminary testing with 1.8.8-1ubuntu0.5, and most things look good.

However, in our case we have an old (external) client using java6 that we sadly still need to support for a while longer. Using the connection simulation in testssl.sh (and also ssllabs) I can see that connections from java6 now fails with our configuration where it previously succeeded.

I suspect that this is not (directly) related to TLSv1.3. The problem with java6 is usually that it only supports dh parameters with 1024 bits (and TLSv1.0).

According to testssl.sh the dh-parameters offered now is:
 DH group offered: RFC3526/Oakley Group 15 (3072 bits)

Before the upgrade it was:
 DH group offered: HAProxy (1024 bits)

I have tried generating custom dh parameters with 1024 bits and specifying them both with the default ssl-dh-param-file setting and directly in the certificate file. I have also tried disabling TLSv1.3 (using no-tlsv13). Neither seem to help.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks a lot David for testing.
It is important that we track this down and make sure it is not introducing a regression.

The related haproxy config would be:
"tune.ssl.default-dh-param <number>"
Sets the maximum size of the Diffie-Hellman parameters used to generate the ephemeral/temporary Diffie-Hellman key when using DHE key exchange. Default <number> value is 1024. Higher values increase CPU load and may not be supported by some clients (IE:Java 7).

That exactly is our case, and /etc/haproxy/haproxy.cfg hasn't changed.
In the default config this argument isn't set so it should still be 1024.

I set up an apache2 with (self signed) certs and enabled TLS1.0. That is old, but what you reported for your compat needs.
First I tested the apache itself with testssl.sh.
Your output in regard to the DH length seems to be from the PFS section.
I get here:
 DH group offered: RFC3526/Oakley Group 14 (2048 bits)

Then I was configuring haproxy+ssl (pre-SRU) in front of apache2 and it gave me

Without the setting above haproxy gives me a nice warning of:
[WARNING] 269/070510 (32043) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.

In regard to DH length in PFS in testssl both showed me:
 DH group offered: HAProxy (1024 bits)

That is just as expected for now.
Upgrading to haproxy from proposed.

The PFS section now really did get bumped, in my case to:
 DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)

The full diff of old/new haproxy build is
+ TLSv1.3 (good)
+ PFS (critical for the SRU):
  - changes cipher list
  - extened Elliptic curves
  - increased DH size 1024->2048
+ Some noise due to TLSv1.3 (ok)

One diff element also might shows the a potential this increased.
old:
 LOGJAM (CVE-2015-4000), experimental VULNERABLE (NOT ok): common prime: HAProxy (1024 bits),
new:
 LOGJAM (CVE-2015-4000), experimental common prime with 2048 bits detected: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits),
The USN [1] indicated this was solved by disabling "DH EXPORT ciphers", so it shouldn't be a real issue.
But chances are that the newer openssl has a new "minimum DH size" for security reasons in regard to CVE-2015-4000.

Checking workaround by setting an explicit max key in the config ...
=> didn't work still size 2048.
So there is a new minimum in place enforced by the new openssl since/due-to the rebuild.

Attaching the testssl logs ...

@Ubuntu-security: is the bump of offered DH keys something known/expected with rebuilds against the newer OpenSSL for the CVE found or others?
If it is, is this a wanted change considering the breakage of some old clients as reported above?

[1]: https://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-4000.html

Changed in haproxy (Ubuntu Bionic):
assignee: nobody → Ubuntu Security Team (ubuntu-security)
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Assigned to ubuntu-security for their guidance (as outlined and asked in comment #12) on the increased minimum DH key size for:

a) this particular case in haproxy

but also

b) potential openssl 1.1.1 rebuilds in general

tags: added: server-triage-discuss
Robie Basak (racb)
tags: added: bionic-openssl-1.1
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

In [1] a workaround for those (hopefully a few) installations that need the lower (not recommended) key size was suggested.

Again this isn't what "should be done", but what users could do if affected.
The reason not to do it is:
a) LOGJAM (CVE-2015-4000), common prime: HAProxy (1024 bits)
b) due to that haproxy always warned you: "[WARNING] 286/090504 (13834) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear."

So we actually want and think it is good that the default size got bumped.
But people that need the old smaller key size (like David in comment #11) need to have a way to drop back to the old key size.

With haproxy and testssl I checked (again) Bionic as-is:
updates (1.8.8-1ubuntu0.4)
 DH group offered: HAProxy (1024 bits)
proposed (1.8.8-1ubuntu0.5):
 DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)

proposed (1.8.8-1ubuntu0.5) + seclevel 0/1 config:
 :-/ I Failed to convince haproxy from using any config in that regard.
 Tried several combinations of "CipherString = DEFAULT@SECLEVEL=0" in /etc/ssl/openssl.cnf
 Tried to get it into /etc/haproxy/haproxy.cfg
 But none worked to get the key size down

This might be my lack of haproxy experience.
I tried some combinations that came to my mind and asked in the IRC channel or haproxy, but got no response. If someone could take a look that knows haproxy better. Maybe we can release this fix for TLSv1.3 IF there is a configuration workaround for the few people that need the small (vulnreable) key size.

[1]: https://lists.ubuntu.com/archives/ubuntu-devel/2019-October/040814.html

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I found this example for apache2:
SSLCipherSuite @SECLEVEL=2:kEECDH:kRSA:kEDH:kPSK:kDHEPSK:kECDHEPSK:-aDSS:-3DES:!DES:!RC4:!RC2:!IDEA:-SEED:!eNULL:!aNULL:!MD5:-SHA384:-CAMELLIA:-ARIA:-AESCCM8

Which reads similar to the default haproxy config:
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

now modified to
ssl-default-bind-ciphers @SECLEVEL=0:ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

But even that (along all the other combinations that felt even less appropriate) worked.
I always get the 2048 bit key now :-/

haproxy IRC replied (thanks) in the meantime and suggested [1] so I'm giving that a try now ...

[1]: https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.1-ssl-dh-param-file

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

At first it rejected my config :
Oct 14 10:02:38 b haproxy[27966]: [ALERT] 286/100238 (27966) : parsing [/etc/haproxy/haproxy.cfg:22] : 'ssl-dh-param-file': unable to load DH parameters from file </etc/haproxy/ssl.cfg>.
Oct 14 10:02:38 b haproxy[27966]: [ALERT] 286/100238 (27966) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg

Following the addition of the feature [1] I found [2]. From there I experimented with pem files.

# Create small DH parameter:
openssl dhparam -out /etc/haproxy/ssl.pem 1024
# Config haproxy to use that one in /etc/haproxy/haproxy.cfg:
ssl-dh-param-file /etc/haproxy/ssl.pem

But all of that seems like defining static DH params for haproxy.
And the key used in PFS is about an "ephemeral DH key" right?
I still struggle to convince haproxy to do what I want :-/

Everyone with more haproxy knowledge is welcome to give this a try ...

[1]: https://<email address hidden>/msg18086.html
[2]: https://www.openssl.org/docs/man1.0.2/man3/PEM_read_bio_DHparams.html

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hmm ... "the default DH parameters that are used during the SSL/TLS handshake when
ephemeral Diffie-Hellman (DHE) key exchange is used" so the documentation agrees that this is for the DHE.

But as mentioned so far I fail to put something in there that makes testssl to report 1024 bit.

./testssl.sh --pfs 10.253.194.137:443
...
 DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

From upstream:
[15:47] <jethro> cpaelzer: we have investigated a bit and think there is an problem with how haproxy handles dhparam < 2048 with openssl 1.1.1. This will be further looked at and hopefuly we find a solution to allow 1024 dhparam. thanks for the report.

I'd say we call this verification failed for now, then we need to sort out this new extra bug.
Fix that extra bug from Ubuntu 20.04 downwards to Bionic and then on Bionic can along that fix this request to rebuild it for TLSv1.3

tags: added: verification-failed verification-failed-bionic
removed: server-triage-discuss verification-needed verification-needed-bionic
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

After further IRC discussion reported upstream in https://github.com/haproxy/haproxy/issues/324 as we'd want to either know how to config and/or a fix to be able to get the key size down when built against openssl 1.1.1

Changed in haproxy:
status: Unknown → New
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Test with that patch worked, we now have to wait until upstream really accepts and until they backport it to 1.8.x for us to be able to pick it up.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The fix was now backported to 1.8 (Thanks Upstream!) and we can start re-preparing an SRU of this.

http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h=19dd0431b06019d5cbd253662822b15412f67144

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Might need some other cleanups or fixes from [2]. That needs evaluation for the SRU policy.

[2]: http://git.haproxy.org/?p=haproxy-1.8.git;a=shortlog

tags: added: server-next
Changed in haproxy:
status: New → Fix Released
Changed in haproxy (Ubuntu Bionic):
assignee: Ubuntu Security Team (ubuntu-security) → Christian Ehrhardt  (paelzer)
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

We'd need several cleanups:
Cleanup:
5b673a658fb1a0a42dbe948b413fceeff1af0642
82b00a11b298a497b4ca93a3f3bf3c7f1399ebc2
b1e3ee6f214d82ebe98140f577777b4c47d88084
And more from there for context.
They are all meant to be no-ops changing the retval handling.
It seems less of an impact to backport the change for that context difference between the 1.8.8 that we have and the 1.8.21 that this was coded for (much better than the 2.0 fix it is for sure).

So only focus on 19dd0431b06019d5cbd253662822b15412f67144 being the actual fix.

But when checking that it becomes clear that the fix makes use of the err message pass-back mechanism that was introduced by the cleanups. Only that way it will be able to report any errors. OTOH it before didn't go into any details and that isn't the point of the fix here.

At least b1e3ee6f214d82ebe98140f577777b4c47d88084 is needed IMHO as that changes the global_dh flow of the code by dropping the "ret = 0; /* DH params not found */". That is actually part of the fix depending on configuration, even thou declared a cleanup.

Ok, that makes up the changes that I'm gonna try and test it now.

Changed in haproxy (Ubuntu Bionic):
status: Fix Committed → In Progress
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

PPA: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1841936-haproxy-openssl

Tested:
a) load dh params from file
b) Test default (without config) size
c) Test config with higher size

Remember, even the broken default config said in the log:
WARNING] 295/095512 (19391) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear
But that was wrong (as you see above) and even setting so didn't change anything at all.

Pre-Fix:
 1024 Warning is shown
 DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)

With Fix:
a) load dh params from file
 NO 1024 Warning is shown
 DH group offered: Unknown DH group (1024 bits)
 That matches my custom key that I have set
b) Test default (without config) size
 1024 Warning is shown
 DH group offered: HAProxy (1024 bits)
c) Test config with higher size
 NO 1024 Warning is shown
 DH group offered: HAProxy (2048 bits)

That finally is as one would expect.
This allows us to fix Disco-Focal and to rebuild the one in Bionic without regressing.

I need to modify the SRU Template as it has different regression statements for those than for Bionic.

And per policy Focal has to be done before the SRUs.
Marking bug tasks accordingly.

Changed in haproxy (Ubuntu Eoan):
status: New → Triaged
Changed in haproxy (Ubuntu Focal):
status: Fix Released → Triaged
Changed in haproxy (Ubuntu Disco):
status: New → Triaged
Changed in haproxy (Ubuntu Bionic):
status: In Progress → Triaged
Changed in haproxy (Ubuntu Disco):
assignee: nobody → Christian Ehrhardt  (paelzer)
Changed in haproxy (Ubuntu Eoan):
assignee: nobody → Christian Ehrhardt  (paelzer)
Changed in haproxy (Ubuntu Focal):
assignee: nobody → Christian Ehrhardt  (paelzer)
summary: - Rebuild haproxy with openssl 1.1.1 will change features (bionic)
+ Rebuild openssl 1.1.1 to pickup TLSv1.3 (bionic) and unbreak existing
+ builds against 1.1.1 (dh key size)
description: updated
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Code and Builds for Versions D-F ready as well.
4 MPs are up and linked in here the bug (see below description)

1.8.8 and 1.8.19 work the same way.
I also picked the change from the 2.0 branch, for Eoan/Focal, but there testing shows errors to load the dh file that the frontend defines like:
    bind 10.253.194.115:443 ssl crt /etc/ssl/certs/haproxy.pem

That now fails to load, the same works in the 1.8 branches and it works (well more likely it wasn't used) prior to the fix.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I think I found the issue, retesting ...

Yes that was it, my mistake but fixed.
Now the Eoan/Focal branches behave as expected as well.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI - syncing 2.0.8 into Focal, which will cover this bug.
The SRU MPs are still up for review.

=> https://launchpad.net/ubuntu/+source/haproxy/2.0.8-1

Changed in haproxy (Ubuntu Focal):
status: Triaged → Fix Committed
Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello David, or anyone else affected,

Accepted haproxy into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/2.0.5-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in haproxy (Ubuntu Eoan):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-eoan
removed: verification-failed
Changed in haproxy (Ubuntu Disco):
status: Triaged → Fix Committed
tags: added: verification-needed-disco
Revision history for this message
Robie Basak (racb) wrote :

Hello David, or anyone else affected,

Accepted haproxy into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.8.19-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in haproxy (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed-bionic
removed: verification-failed-bionic
Revision history for this message
Robie Basak (racb) wrote :

Hello David, or anyone else affected,

Accepted haproxy into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.8.8-1ubuntu0.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
David Hedberg (david-hedberg-t) wrote :

I have tested 1.8.8-1ubuntu0.6 (bionic) with our existing configuration, and I cannot find anything wrong it.

We use the default 1024 bit DH-parameters, and override it with better parameters for some of our frontends using parameters specified directly in the certificate file (concatenated).

testssl.sh and ssllabs report only expected changes (TLSv1.3). No configuration change was needed.

Thank you!

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Prior to Update:
E: DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)
D: DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)
B: DH group offered: HAProxy (1024 bits)
=> D+E on wrong defaults!

With tuning to specific key (2048):
tune.ssl.default-dh-param 2048
E: DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)
D: DH group offered: RFC5114/2048-bit DSA group with 224-bit prime order subgroup (2048 bits)
B: DH group offered: HAProxy (2048 bits)
=> E+D ignore the config!

## Post Update ##

E: DH group offered: HAProxy (1024 bits)
D: DH group offered: HAProxy (1024 bits)
B: DH group offered: HAProxy (1024 bits)
=> E+D back on the expected default
=> B not broken by rebuild

With tuning to specific key (2048):
tune.ssl.default-dh-param 2048
E: DH group offered: HAProxy (2048 bits)
D: DH group offered: HAProxy (2048 bits)
B: DH group offered: HAProxy (2048 bits)
=> E+D: Config now works
=> B not broken by rebuild

Also on Bionic now (for the initial TLSv1.3 request):
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
 Testing protocols via sockets except NPN+ALPN
 SSLv2 not offered (OK)
 SSLv3 not offered (OK)
 TLS 1 offered (deprecated)
 TLS 1.1 offered (deprecated)
 TLS 1.2 offered (OK)
 TLS 1.3 offered (OK): final
...

Thanks to David for the extended test with a real configuration!
Marking this verified

tags: added: verification-done verification-done-bionic verification-done-disco verification-done-eoan
removed: verification-needed verification-needed-bionic verification-needed-disco verification-needed-eoan
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package haproxy - 2.0.5-1ubuntu0.1

---------------
haproxy (2.0.5-1ubuntu0.1) eoan; urgency=medium

  * Fix configurability of dh_params that regressed since building
    against openssl 1.1.1 (LP: #1841936)
    - d/p/lp-1841936-BUG-MEDIUM-ssl-tune.ssl.default-dh-param-value-ignor.patch
    - d/p/lp-1841936-CLEANUP-ssl-make-ssl_sock_load_dh_params-handle-errc.patch

 -- Christian Ehrhardt <email address hidden> Wed, 23 Oct 2019 12:58:09 +0200

Changed in haproxy (Ubuntu Eoan):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for haproxy has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package haproxy - 1.8.19-1ubuntu1.1

---------------
haproxy (1.8.19-1ubuntu1.1) disco; urgency=medium

  * Fix configurability of dh_params that regressed since building
    against openssl 1.1.1 (LP: #1841936)
    - d/p/lp-1841936-BUG-MEDIUM-ssl-tune.ssl.default-dh-param-value-ignor.patch
    - d/p/lp-1841936-CLEANUP-ssl-make-ssl_sock_load_dh_params-handle-errc.patch

 -- Christian Ehrhardt <email address hidden> Wed, 23 Oct 2019 12:34:38 +0200

Changed in haproxy (Ubuntu Disco):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package haproxy - 1.8.8-1ubuntu0.6

---------------
haproxy (1.8.8-1ubuntu0.6) bionic; urgency=medium

  * Fix issues around dh_params when building against openssl 1.1.1
    to avoid regressing the minimal key size (LP: 1841936)
    - d/p/lp-1841936-BUG-MEDIUM-ssl-tune.ssl.default-dh-param-value-ignor.patch
    - d/p/lp-1841936-CLEANUP-ssl-make-ssl_sock_load_dh_params-handle-errc.patch

haproxy (1.8.8-1ubuntu0.5) bionic; urgency=medium

  * no change rebuild to pick up openssl 1.1.1 and via that
    TLSv1.3 (LP: #1841936)

 -- Christian Ehrhardt <email address hidden> Wed, 23 Oct 2019 11:37:53 +0200

Changed in haproxy (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

2.0.8-1 is synced in Focal, closing that bug as well,

Changed in haproxy (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.