openssl 1.0.1-4ubuntu2 breaks a bunch of ciphers

[Thomas Bushnell, BSG]

in version 1.0.1-4ubuntu2, we see:

openssl (1.0.1-4ubuntu2) precise-proposed; urgency=low

  * Backport more upstream patches to work around TLS 1.2 failures
    (LP #965371):
...
    - Truncate the number of ciphers sent in the client hello to 50. Most
      broken servers should now work.
 ...

 -- Colin Watson <email address hidden> Wed, 18 Apr 2012 15:03:56 +0100

We have a server which offers a very small number of ciphers. When this change hit, suddenly our hosts could no longer contact this server, getting the error:

$ openssl s_client -connect HOSTNAME:9140
CONNECTED(00000003)
139736292189856:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:724:

The problem here was tracked down to a failure to find a matching cipher. If we specify -cipher RC4-SSH (the only one essentially which the server permits) or -ssl3, the connection succeeds.

The problem is this truncation of the number of ciphers sent. RC4-SSH shows up at something like #74 on our list, so it is getting truncated. When we specify exactly the cipher to use, of course it works, and if we say -ssl3, then that also reduces the number which would be sent, and now RC4-SSH is in the top fifty again.

This is a pretty disastrous change, in fact; it means that openssl basically now supports only fifty ciphers at a time, and then an essentially random and unpredictable set. Not only does this mean a loss of functionality, it could be a loss in security if clients get pushed to less secure ciphers because the more secure ones happened to be after number fifty in the list.

[Jordon Bedwell (envygeeks)]

openssl s_client -showcerts -connect d3vwyrdyja2n00.cloudfront.net:443 - Fails
openssl s_client -showcerts -tls1 -connect d3vwyrdyja2n00.cloudfront.net:443 - Works

[Colin Watson (cjwatson)]

Well, I'm sorry for the problems, but if anyone can suggest something
which fixes this and also does an equivalent job at fixing problems seen
against other servers, I'm all ears. I was sucked into this whole thing
as a result of dealing with a request for performance improvements which
required upgrading to OpenSSL 1.0.1 (although there were plenty of
people clamouring for TLS 1.2 support as well), and am now heartily
wishing I'd never touched it as I'm stuck backporting an increasingly
wobbly stack of upstream patches! This really needs upstream work to
sort out ...

[Thomas Bushnell, BSG (tbushnell)]

Colin, I hope you'll reconsider this change and revert it.

I understand that there are buggy servers which fail when they get offered too many ciphers by clients, but they *always* failed; that's nothing new. So in order to expand the use cases for the library, this change has caused a regression. It's much worse to take correctly-working server/client pairs and deliberately break them than to fail to support incorrectly-working server/client pairs.

It's not just us; Jordon Bedwell above had the same problem. It's going to break a *lot* of people.

Moreover, it is really an important security issue as well as an interoperability one. I have a right to expect that I will get the most secure cipher from the set formed by the intersection of the client's and the server's supported sets; with this change, I do not, because the client has artificially eliminated some of its supported set.

This is a serious, serious regression, both in security and in interoperability.

[Thomas Bushnell, BSG (tbushnell)]

One of our engineers says this:

FWIW, looking at the code, the problematic chunk, added to ssl/s23_clnt.c by tls12_workarounds.patch, was

@@ -467,6 +469,15 @@
                               SSLerr(SSL_F_SSL23_CLIENT_HELLO,SSL_R_NO_CIPHERS_AVAILABLE);
                               return -1;
                               }
+#ifdef OPENSSL_MAX_TLS1_2_CIPHER_LENGTH
+ /* Some servers hang if client hello > 256 bytes
+ * as hack workaround chop number of supported ciphers
+ * to keep it well below this if we use TLS v1.2
+ */
+ if (TLS1_get_version(s) >= TLS1_2_VERSION
+ && i > OPENSSL_MAX_TLS1_2_CIPHER_LENGTH)
+ i = OPENSSL_MAX_TLS1_2_CIPHER_LENGTH & ~1;
+#endif
                       s2n(i,p);
                       p+=i;

OPENSSL_MAX_TLS1_2_CIPHER_LENGTH is defined to 50, and is actually the number of bytes to use for the cipher list in the handshake, not the number of ciphers. Each cipher uses 2 bytes, so we actually get only 25 ciphers.

And somebody that knows openssl might want to double-check that call to TLS1_get_version(s) - right before this chunk, there's a call to the function that actually adds the ciphers to the handshake buffer (ssl_cipher_list_to_bytes). That function compares the return value of TLS1_get_client_version(s) with TLS1_2_VERSION and then decides to skip the TLS1.2-only ciphers, which puts RC4-SHA among the first 50.

Either changing OPENSSL_MAX_TLS1_2_CIPHER_LENGTH to 100 (which actually means 50 ciphers) or changing the TLS1_get_version(s) to TLS1_get_client_version(s) fixes things, though I have no idea what this last change means.

[Thomas Bushnell, BSG (tbushnell)]

Other things that would save us:

1) Any way to disable this regression from configuration files or the like. (Merely adding an option in the library interface wouldn't help unless puppet also can be told to pass that option.)
2) Any way from configuration files to tell SSL to use the equivalent of the -tls1, or -cipher, switches to openssl s_client.
3) Any way from configuration files or command line options to tell puppet to tell SSL the equivalent of (2).

Sadly, this one change has forced us to probably miss our target release of our precise distro by a month. :(

[Colin Watson (cjwatson)]

I do understand the severity, and I don't mean to minimise it; the
difficulty is that OpenSSL 1.0.1 has been problematic from the start,
and every change fixes some cases while breaking others. Rolling back
all the way to 1.0.0, while perhaps the safest option in some cases, is
by now too invasive a change to attempt; rolling back libraries to
earlier ABIs is in general problematic once they've been widely
deployed.

It's not true that the buggy servers in question always failed. These
were regressions and they were reported to me as such. You can find the
details in bug 965371, its duplicates, and the linked Debian bug.

Every change that I have made in an attempt to fix it has been directly
backported from upstream CVS and/or recommended by upstream developers.
Here's the commit where they recommend 50:

  http://cvs.openssl.org/chngview?cn=22408

I haven't done the packet arithmetic in detail, but a quick capture here
against cs3-api.salesforce.com (one of the servers previously reported
as failing, though I haven't checked if it was for this reason) shows
that the client hello is currently 240 bytes. If that's true across the
board, then we can only fit in eight more ciphers before exceeding 255
bytes, which isn't enough for you.

As such, I'm happier with the suggested workaround to use
TLS1_get_client_version than with adjusting the workaround than with
tweaking the number. The client version is supposed to be what was sent
by the client in the hello, so in general I think I'd expect s->version
and s->client_version to be the same while *sending* the client hello,
but that's evidently not the case and I have a suspicion that the
version downgrades applied in the current set of backported workarounds
are only applied to s->client_version. I'm going to try this in
-proposed and see how it goes; but this has been so delicate that I
really want to get as widespread testing as possible before promoting it
to general use.

[Martin Pitt (pitti)]

Hello Thomas, or anyone else affected,

Accepted openssl into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Thomas Bushnell, BSG (tbushnell)]

Colin: I think I understand now much better; thanks. The upgrade to 1.0.1 increased the number of ciphers, which in turn broke communication with bad servers (of which there may be an important number). What a thorny little knot of a problem. Short of fixing these bad servers (which might be a hard and many-year problem), perhaps the only real solution is to have our clients sort the ciphers by security preference, and then attempt multiple connections. Obviously that's not a quick fix!

I see 1.0.1-4ubuntu5 has hit proposed, so I'll give that a try right now.

[Thomas Bushnell, BSG (tbushnell)]

I'm pleased to report that 1.0.1-4ubuntu5 seems to resolve our symptoms. Many thanks Martin and Colin for that!

It only kicks the can down the road a bit, but it does give some breathing room.

[jrun (jeremy-burks)]

Accidentally changed the status. Looking how to revert the change. Sorry!

[Samuel Carlier (9-sacuel-6)]

i have installed 1.0.1-4ubuntu5 but it still doesnt work

openssl s_client -connect d18kq98amm3n6k.cloudfront.net:443
CONNECTED(00000003)
140107426719392:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:724:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 174 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

[Samuel Carlier (9-sacuel-6)]

i am sorry was to quick to post..

i only installed openssl (1.0.1-4ubuntu5) but libssl1.0.0 (1.0.1-4ubuntu5) was needed aswell
it works!

[chrone (chrone81)]

on ubuntu 12.04 server edition x64, i have to enable higher than RC4 chiper such as camelia or aes on apache to be enable to test curl https://mydomain.com here.

if i set it to RC4 only from apache mod ssl.conf, then the curl https://mydomain.com produced an error message regarding handshake failure.

i don't know which package cause this bug, whether is it curl, php5-curl, apache, or openssl.

hope there will be better solution on next update. we used curl to send email through google mail smtp. it breaks after upgrade from ubuntu 11.10 to 12.04. :(

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 1.0.1-4ubuntu5

---------------
openssl (1.0.1-4ubuntu5) precise-proposed; urgency=low

  * debian/patches/CVE-2012-2110b.patch: Use correct error code in
    BUF_MEM_grow_clean()

openssl (1.0.1-4ubuntu4) precise-proposed; urgency=low

  * Check TLS1_get_client_version rather than TLS1_get_version for client
    hello cipher list truncation, in a further attempt to get things working
    again for everyone (LP: #986147).
 -- Jamie Strandboge <email address hidden> Tue, 24 Apr 2012 08:29:32 -0500

[Evgeny Brazgin (xapienz)]

Please make fix for quantal.
Currently quantal has openssl_1.0.1c-3ubuntu1, it has this bug. After installing openssl_1.0.1-4ubuntu5 from precise everything works fine, but dpkg thinks it is downgrade.

[Marc Deslauriers (mdeslaur)]

@xapienz: please file a new bug for your issue, this bug is closed. Thanks.

[Tyler Hicks (tyhicks)]

For xapienz and others affected by this bug again in Quantal, this issue is being tracked in bug #1051892

[Peter (peter-md)]

Is there any date for this fix to be released? Ubuntu 12.04.1 still shipped with 1.0.1-4ubuntu3.

[Robie Basak (racb)]

The fix for this is already released in precise-updates (which is on 1.0.1-4ubuntu5.5). See https://help.ubuntu.com/community/Repositories/Ubuntu#Updates_Tab for enabling -updates if you need help with this.

I'm not sure how to verify what version shipped in 12.04.1. But if you are waiting for an official CD image update and it isn't in 12.04.1, then the fix be in 12.04.2 (see https://wiki.ubuntu.com/LTS and https://wiki.ubuntu.com/PrecisePangolin/ReleaseSchedule) since it's already available in precise-updates.

[kylea (kylea)]

3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Description: Ubuntu 12.04.1 LTS
Release: 12.04

Are we sure this is fixed - I still have to specify -ssl3

--- this works:

openssl s_client -connect WWW.ITVSN.COM.AU:443 -state -ssl3

--- this fails:

openssl s_client -connect WWW.ITVSN.COM.AU:443 -state

CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:unknown state
SSL3 alert read:fatal:protocol version
SSL_connect:error in unknown state
139717957809824:error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version:s23_clnt.c:724:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 226 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

[kylea (kylea)]

BTW - have 1.0.1-4ubuntu5.5 of libssl1.0.0 installed

[Klavs Klavsen (kl-vsen)]

I have the same problem with precise.

I have just dist-upgraded - and have:
libssl1.0.0 1.0.1-4ubuntu5.7
openssl 1.0.1-4ubuntu5.7

yet openssl s_client can ONLY connect if I use -tls1 or -ssl3

[Klavs Klavsen (kl-vsen)]

it hits my php application - which can't connect to the server- and I don't know how to make it use -ssl3 or equivalent.

[Klavs Klavsen (kl-vsen)]

Ohh - and here's a site I just found I can't visit (but it works from CentOS) : https://www.soljerome.com/blog/2011/12/17/mirroring-rhn-with-mrepo-on-rhel6/

pretty annoying :(

[jsnydr (jsnydr)]

I am currently experiencing this problem with my 12.04 installation.

'dpkg' says that I have 1.0.1-4ubuntu5 installed.

I am using precise-updates and synaptic seems to think I am completely up to date.

I can work around by forcing a specific version of SSL manually.

[Seth Arnold (seth-arnold)]

jsnydr, klavs, kylea, note that you may also be experiencing what is documented at https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/861137 -- some servers and services cannot handle TLS 1.1 or newer handshakes.

It would be worth going to some effort to determine if your peers can handle TLS 1.1 or TLS 1.2 connections, and if not, file bug reports or trouble tickets with the vendors of those services or servers.

Thanks