Invalid GnuTLS cipher suite strings causes libldap to crash

[Jouko Orava]

If the cipher suite string is unacceptable to GnuTLS, libldap_r-2.4 crashes due to a double free. GnuTLS is extremely picky about the cipher suite strings it accepts; as a first measure, try LDAP cipher suite string "SECURE256" or "NORMAL". If that stops the crash, then you have encountered this bug.

Typically, the crash report begins with something like

*** glibc detected *** APPLICATION: double free or corruption (!prev)
/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fc68cff0b96]
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2(+0x38769)[0x7fc68bb13769]
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2(+0x3570e)[0x7fc68bb1070e]
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2(ldap_pvt_tls_init_def_ctx+0x1d)[0x7fc68bb108ed]
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2(+0x35965)[0x7fc68bb10965]
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2(+0x35a6d)[0x7fc68bb10a6d]
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2(ldap_int_tls_start+0x5d)[0x7fc68bb1149d]

The actual double free happens in openldap/libraries/libldap/tls2.c:ldap_int_tls_init_ctx(), in the ldap_pvt_tls_ctx_free(lo->ldo_tls_ctx); call in the error_exit: path.

The root cause of the double free is lack of GnuTLS return value checks when calling gnutls_priority*() functions. The code simply assumes they succeed, and when GnuTLS fails to provide a valid context due to those failures, ldap_int_tls_init_ctx() tries to free the never-fully-initialized context.

A simple fix is to create GnuTLS security contexts using the configured cipher suite string, instead of "NORMAL" as openldap/libraries/libldap/tls_g.c now does. If the cipher suite string is invalid, then do not create the context at all. This is caught earlier in ldap_int_tls_init_ctx(), and avoids the crash.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Suggested patch to fix libldap crash with invalid GnuTLS cipher suite strings" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Please could you clarify exactly which Ubuntu package versions of openldap are affected by this bug? Has this been reported and/or fixed upstream, and if so could you please provide appropriate links? And could you please post exact steps needed to reproduce this crash?

Once done, please change the bug status back to New. Thanks!

[Jouko Orava (joorava)]

This bugs affects libldap-2.4.-2, at least versions versions 2.4.28 (2.4.28-1.1ubuntu4) and 2.4.31 (2.4.31-1ubuntu2), when compiled against GnuTLS. The bug exists in latest openldap.org upstream versions from 2.4.28 to 2.4.33 at least; probably since they switched from custom parsing the cipher suite for GnuTLS to GnuTLS's own. Based on this upstream report,
    http://www.openldap.org/its/index.cgi/Incoming?id=6939#themesg
I suspect the bug has existed upstream at least since 2.4.25.

Based on the symptoms and crash reports, I strongly believe
    https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1026057
    https://bugs.launchpad.net/ubuntu/+source/libnss-ldap/+bug/1090554
    http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=640384
are all the same bug.

To reproduce, I used a simplified ldapsearch-like program, which connects to an ldap server (ldap:// URIs), then sets the desired priority (cipher suite) based on a command-line parameter, issues StartTLS to secure the connection, then does a simple anonymous search. If the cipher suite is not valid for GnuTLS, the program crashes in libldap_r-2.4-2. The test program is attached for easier bug verification. (Note: You need an LDAP server supporting StartTLS to connect to.)

It seems that in any application that relies on libldap parsing the cipher suite will crash the same way, when the cipher suite is not valid for GnuTLS, and the connection involves StartTLS. This includes at least sssd: put e.g. "ldap_tls_cipher_suite = FOOBAR" and "ldap_id_use_start_tls = True" in sssd.conf, and use an ldap:// URI, and sssd will crash.

Correct cipher suites for GnuTLS are very complex to construct; for example, to allow only AES and CAMELLIA 256-bit ciphers with 160 or 256-bit SHA MACs and RSA key exchange and any compression and any TLS version (excluding SSH3.0), one needs to use "'NONE:+VERS-TLS1.2:+VERS-TLS1.1:+VERS-TLS1.0:+COMP-ALL:+AES-256-CBC:+CAMELLIA-256-CBC:+RSA:+SHA1:+SHA256".

If the Ubuntu OpenLDAP developers and users can confirm the bug, and that this patch fixes the issue, it would be easier to push the patch upstream. I do not currently use LDAP on my machines, as I only investigated the issue to help a friend. So, it does not matter to me whether the patch is accepted or not, or if the bug is further investigated or not. I'm very willing to help, though.

[Robie Basak (racb)]

Thank you for your detailed investigation into this. I appreciate the time you've spent on this.

Marking as medium importance, since a workaround is available (which I believe is to fix the cipher suite string to something valid, right?)

> If the Ubuntu OpenLDAP developers and users can confirm the bug, and that this patch fixes the issue, it would be easier to push the patch upstream.

This is reasonable, although I'm not sure there are enough Ubuntu OpenLDAP developers to make this likely. Until then this bug may sit unattended, but your writeup will at least help others so I appreciate it being here.

For anyone else who comes across this: please mark it as "affects me too". If you can spare time to work on it, please confirm that it affects Ubuntu, check to see if the upstream non-packaged release is affected, look towards getting this reported upstream and note any new information or progress here. We can certainly patch the Ubuntu package if the bug and patch can be verified. But if upstream commit it and/or acknowledge the bug, that really makes things easier. Thanks!

[Jouko Orava (joorava)]

Correct. The workaround to avoid the crash is to use a strictly valid GnuTLS cipher suite string, for example "NORMAL" or "SECURE128" or "SECURE192" or "SECURE256".

In those rare cases where those existing defaults are not acceptable (due to security concerns, for example), the minimal "search.c" program I attached to #4 can be used to try to find a valid cipher suite string, connecting to an LDAP server (using ldap:// URI, and StartTLS). It also reports the cipher, mac, and kx achieved when the StartTLS is successful.

I'll see if I can report this upstream to openldap.org, too.

[Jouko Orava (joorava)]

Reported upstream at openldap.org, as Incoming/7500,
    https://www.openldap.org/its/index.cgi/Incoming?id=7500

[Ryan Tandy (rtandy)]

Fix committed upstream, will be released in 2.4.40.

http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commit;h=7350a52adacb5f258925b14d2bc5136c8f4ddd9b

[Harry Coin (hcoin)]

Kindly notice that the fix mentioned above for .40, was dated not quite a year ago.

I'm not a ubuntu expert, but I think this page: https://launchpad.net/ubuntu/+source/openldap
explains the fix mentioned above is not available as a backport for trusty, nor native in utopic, nor even being tested in vivid.

Is it available on Ubuntu but I just missed it somehow?

[Ryan Tandy (rtandy)]

The fixed version is not in Ubuntu yet. This crash only happens on invalid configurations, though; slapd will still refuse to start on such a configuration. Fix your configuration to be correct, and you won't see the crash any more.

[Jouko Orava (joorava)]

rtandy, this is not specific to slapd, but affects all applications that use libldap2 and gnutls. Instead of returning a failure at START_TLS, the library just crashes at a double-free. This makes it difficult to find the actual problem in services like sssd that crash due to this bug, although the root cause is a simple configuration mistake. (gnutls cipherspecs are notoriously complicated, and very easy to get wrong. Crashing in such a case is, and should be considered, a serious bug. There is nothing an application can do to mitigate this.)

Attached is a backported patch from 2.4.40 to current Debian/Ubuntu source package. I applied this to 2.4.31-1+nmu2ubuntu8, added a dummy changelog entry, and recompiled the package. The changes are localized and safe, should apply cleanly to other versions too. The patched library no longer crashes: this fixes the bug.

In other words, this is a trivial bug for the Debian/Ubuntu openldap maintainers to fix, if they saw the bug serious enough to fix.

[Ryan Tandy (rtandy)]

On Wed, Mar 18, 2015 at 06:40:06PM -0000, Jouko Orava wrote:
>rtandy, this is not specific to slapd, but affects all applications that
>use libldap2 and gnutls.

Apologies for the lack of context. You're completely correct, but the
message I was replying to was about slapd specifically: he had just
reported bug 1433666 about slapd failing to start when configured with a
wrong cipher suite settings.

Thanks for providing a patch. I can't upload packages myself, but maybe
ubuntu-sponsors will consider it. This is already fixed in Debian jessie
and wheezy-backports, FWIW (but not wheezy itself).

[Jouko Orava (joorava)]

Well, considering that Ubuntu openldap maintainers consider e.g. CVE-2013-4449
(denial-of-service, 2.4.31 to 2.4.36 are vulnerable) not important enough to patch
or update to a later openldap version, I expect there to be zero chance of this bug
to be patched either. It seems that if it does not hurt the maintainers' systems,
it's not worth fixing.

The current Ubuntu version I am using right now, 14.04 LTS, is certainly the last
Ubuntu version I will be using. I am still evaluating the alternatives, but
definitely all Debian jessie derivatives are straight out.

I won't be monitoring this bug anymore, either.

[Harry Coin (hcoin)]

If this were a library used in a game or a bug in a screensaver I could see letting a formatting error in a string crash abort any program using the library sit for a year. I'm staggered really to experience this for a package as widely touted as gnutls, contending to be a replacement for openssl, and especially in a business supporting group like ubuntu that aims for site installs.

I think this 11-month 'maintainers have higher priorities' event is a strong sign gnutls just is so not ready for mission critical deployment that whatever priority it may have on launchpad--- in the maintainers minds this is a 'might fix, won't deploy'.

I've compiled it against openssl, and it's solid. Though I've stuck with ubuntu for many years now I have to agree with the sentiment upstream: this is a confidence buster.

[Harry Coin (hcoin)]

I just now noted the remark above suggesting the remedy to programs which crash abort when having a string parsing error is to not feed it strings it doesn't like. I suppose, mutatis mutandis, were the string one 99 of 100 leave defaulted it could be overlooked. However does anyone really think the string configuring the allowed ciphers isn't tweaked every few months in any serious deployment?

[Oleg Strikov (strikov-deactivatedaccount)]

Shell script which reproduces the issue: http://pastebin.ubuntu.com/10712595/
Please run this script only on a disposable instance in the cloud because it creates and adds ultimately trusted certificate to the target machine.

I was able to reproduce the issue on precise (12.04) and trusty (14.04).
I *was not* able to reproduce the issue on utopic (14.10) and vivid (15.04).
This happens because libldap is linked against later version of libgnutls in 14.10 and 15.04.
12.04 and 14.04 use 2.x generation of libgnutls while 14.10 and 15.04 use 3.x generation.
I assume that libgnutls 3.x does proper cleanup and doesn't return semi-initialized context on errors (which was the root cause of the bug).

Please note that this issue doesn't crash ldap server itself but only clients who passes incorrect SSL/TLS-related settings into libldap using ldap_set_option(NULL, LDAP_OPT_X_TLS_CIPHER_SUITE, <NAME>).

[Robie Basak (racb)]

13:40 <strikov> rbasak: i did a research on CVE attached to the bug and came to conclusion that it was attached incorrectly
13:41 <strikov> rbasak: this CVE is about a different thing and I have no idea why it was attached

Looking at the CVE details I agree, so unlinking.

[Oleg Strikov (strikov-deactivatedaccount)]

I plan to change the status of this bug for 12.04 (precise) and 14.04 (trusty) to Won't Fix.
In this comment I want to explain why I came to this decision.

This bug had CVE-2013-4449 linked to it. I don't think that this CVE is relevant because the patch proposed in this bug doesn't resolve the issue mentioned in the description of this CVE. I proved that by using the following repro script:
http://pastebin.ubuntu.com/10764620/
This script is derived from the repro case provided in the debian bug for CVE-2013-4449:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=729367#22
[!] Please note that this CVE can be reliably reproduced only on multicore machine (e.g. you can't use m1.small cloud instance). Some form of race condition takes place and your chances are much higher on multicore.

When CVE-2013-4449 is resolved this script should print 'Finished' at the end of execution.
When CVE is still here it prints 'No server found on localhost:389 <attempt>'.
'No server found' means that slapd crashed and can't be accessed via network and '<attempt>' is a number of iteration when slapd crashed (it usually takes from 3 to 15 iterations because some form of race condition needs to take place).
WITH and WITHOUT the proposed patch I get 'No server found' message on 12.04 (precise) and 14.04 (trusty).
It means that patch doesn't fix CVE-2013-4449.

Patch doesn't fix CVE-2013-4449 but it still can fix the issue mentioned in the bug description (incorrect cipher suite string leads to a crash). That's true but I don't think that we want to update 12.04 (precise) and 14.04 (trusty). ANY update may lead to unpredictable regressions (see https://wiki.ubuntu.com/StableReleaseUpdates) and the profit of patching should exceed the amount of potential issues it may create. OpenLDAP is an important infrastructural component and we need to have a very good reason to update it. I don't see such a reason. Client may crash itself by passing incorrect cipher suite to the API. While that's sad, it doesn't crash slapd itself and doesn't create any inconveniences to other users. This looks like a good fix for a development release but not stable release.

Please let me know if you have any objections or additional information about this bug.
We're open to discussion and can re-open this bug if needed.
Thanks to Jouko Orava and others for opening this bug and taking part in the discussion.

[Harry Coin (hcoin)]

On 04/07/2015 01:32 PM, Oleg Strikov wrote:
> Client may crash itself
> by passing incorrect cipher suite to the API. While that's sad, it
> doesn't crash slapd itself
To the contrary, it certainly does crash slapd itself. Anyone
upgrading will at some point silently switch from a slapd that used
openssl to gnutls --- without the package warning about nor updating the
apropos config string. As a result, "apt-get update;apt-get upgrade"
results in slapd crashing with a double free as it loads the previous
conf file. Most package maintainers would refer to this as a regression
inasmuch as the typical upgrade process fails to start and without any
obvious warning. The answer may be found by spending many hours
googling for 'what the heck'. For over a year this has gone unfixed.

At least improve the upgrade script to warn the installer and prevent
slapd from starting until some flag is set noting the user has corrected
the string and is aware the developers won't fix the issue.

In the alternative, I think a much better approach is to put a versions
of all these and related packages compiled against openSSL in the
appropriate repository. It is not material to me whether this is fixed
or not as I've removed all packages using gnutls until it's more mature,
and won't revisit this again for at least four years.

In fact I'm looking for other distros like Mint that actually check
whether upgrades generate regressions and classify risk assessments that
allow only proven upgrades to succeed.

It's really quite an eye-opener to me that a distro aiming to be
deployed outside the sole-user world doesn't see this as a problem.

--

Harry G Coin
Quiet Fountain LLC
2118 Lundy Ln
Bettendorf, Iowa 52722

[Oleg Strikov (strikov-deactivatedaccount)]

Hi Harry,

Thanks for the input.
Could you add more information on this please:
> silently switch from a slapd that used openssl to gnutls
I just looked through the launchpad package archive and it looks like we never had openldap linked against openssl in 12.04 and 14.04. First version of openldap which showed up in 12.04 was 2.4.25 and it has libgnutls-dev in build dependencies. Do you mean upgrading from any previous release to precise/trusty?

[Robie Basak (racb)]

Marking Won't Fix for SRUs as per Oleg's request. I don't see any real user impact to this bug here that would justify an SRU. Harry's case might be valid, but as Oleg was unable to reproduce and we don't have reproduction steps we wouldn't be able to pass SRU verification anyway. If somebody would like to post detailed steps to reproduce this bug that also demonstrates a use case which does create a real user impact, then I'd be happy to reconsider.

[Harry Coin (hcoin)]

On 04/10/2015 10:43 AM, Robie Basak wrote:
> Marking Won't Fix for SRUs as per Oleg's request. I don't see any real
> user impact to this bug here that would justify an SRU. Harry's case
> might be valid, but as Oleg was unable to reproduce and we don't have
> reproduction steps we wouldn't be able to pass SRU verification anyway.
> If somebody would like to post detailed steps to reproduce this bug that
> also demonstrates a use case which does create a real user impact, then
> I'd be happy to reconsider.
>
> ** Changed in: openldap (Ubuntu Precise)
> Status: In Progress => Won't Fix
>
> ** Changed in: openldap (Ubuntu Trusty)
> Status: In Progress => Won't Fix
>
Steps to reproduce:
1) Install older version that used openssl.
2) Set up a cipher suite of any sort.
3) Validate ldaps operation.
4) "upgrade" using current version built against gnutls.
5) Notice slapd won't start, complaining of double free, upgrade fails.

--

Harry G Coin
Quiet Fountain LLC
2118 Lundy Ln
Bettendorf, Iowa 52722

[Oleg Strikov (strikov-deactivatedaccount)]

> 1) Install older version that used openssl.
Do you mean manual installation of self-baked or foreign .deb?
To my knowledge (correct me if I'm wrong), you have no way to get openssl-based slapd from precise or trusty repo.

[Robie Basak (racb)]

> 1) Install older version that used openssl.

Which version, from what release, from what repo?

> 2) Set up a cipher suite of any sort.

How? What file did you edit? What did you change? What commands did you type?

> 3) Validate ldaps operation.

How?

I asked for detailed steps. The summary I already understood from your previous comments. I have already explained why I think this is insufficient, so I am not able to reconsider anything based on your reply.

[Ryan Tandy (rtandy)]

On Fri, Apr 10, 2015 at 04:30:32PM -0000, Harry Coin wrote:
>Steps to reproduce:
>1) Install older version that used openssl.
>2) Set up a cipher suite of any sort.
>3) Validate ldaps operation.
>4) "upgrade" using current version built against gnutls.
>5) Notice slapd won't start, complaining of double free, upgrade fails.

The nit-picker in me feels compelled to point out that the
openssl→gnutls change invalidating existing TLSCipherSuite settings
actually was dealt with, sort of:

http://anonscm.debian.org/cgit/pkg-openldap/openldap.git/commit/?id=327fcec47c59ccb7de65747327730eabc5656969

(This would have been applied when upgrading to hardy.)

However, in 2.4.14 the cipher suite parser used for gnutls was changed,
but this time there was no such upgrade handling:

http://www.openldap.org/its/?findid=6251
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=541256

AFAIK the latter change, not the former, would have introduced this when
upgrading to jaunty (or for LTS users, from hardy to lucid).

FWIW, upstream explicitly documents in ldap.conf(5) that TLSCipherSuite
settings are implementation dependent, and that openssl and gnutls
ciphersuite strings are not compatible. Even after fixing the
double-free, a manual "reconfigure ciphersuites for gnutls" step is
required in the upgrade steps listed above...

[Oleg Strikov (strikov-deactivatedaccount)]

Ryan: Many thanks for providing us with the whole picture!

Just to clarify things.
I *do* think that handling this issue is important.
That's bad when one of the services doesn't start w/o putting some hints to the log.
I just don't think that this is SRU type of issue.
Regular precise/trusty users won't meet this upgrading issue because they use *only* libgnutls-based slapd and their config contains correct cipher-suite information.

I'd be happy to fix this issue in development release but we have nothing to fix there.
Repro case provided in the bug works nicely on vivid and doesn't lead to a crash.