OpenLDAP: Backport a fix for use-after-free in GnuTLS-related code

[Maciej Puzio]

May I ask that you backport an upstream patch that resolves the issue of use-after-free in libldap that interferes with syncrepl, causing failures and segfaults.

OpenLDAP commit: 283f3ae1713df449cc170965b311b19157f7b7ea
Link: http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commitdiff;h=283f3ae1713df449cc170965b311b19157f7b7ea
Modifications to file: libraries/libldap/tls_g.c

This problem affects openldap 2.4.41 (in Ubuntu wily), 2.4.42 (in Ubuntu xenial), as well as in 2.4.44 (current upstream stable version). More details are availble on OpenLDAP project bug tracker at:
http://www.openldap.org/its/index.cgi/Software%20Bugs?id=8385

Thank you

[Maciej Puzio (maciej-puzio)]

Patch created by OpenLDAP team applies cleanly to openldap 2.4.41+dfsg-1ubuntu2 (wily).

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "tls_g.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[dog (thedogofpavlov)]

This patch may also resolve https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1547927

I'll confirm once available and I have an opportunity to test.

[Maciej Puzio (maciej-puzio)]

I created a PPA with patched deb packages, available at: https://launchpad.net/~maciej-puzio/+archive/ubuntu/openldap
Currently it contains openldap-2.4.41 source package with the above patch applied, as well as binary debs built from it, for amd64 and i386. These packages are for Ubuntu 15.10 (wily), but I can make them for other Ubuntu releases, if you would like that. I briefly tested the amd64 libldap, ldap-utils and slapd packages, they installed fine and appear to work. I did not test any of the i386 debs.

If you would like to install and test these packages, please run the following commands:
sudo apt-add-repository ppa:maciej-puzio/openldap
sudo apt-get update
sudo apt-get upgrade
Of course, please install them on a test machine, and not on the production server.

[Maciej Puzio (maciej-puzio)]

I have just found that Howard Chu of OpenLDAP team had already uploaded this patch to Launchpad VCS:
http://bazaar.launchpad.net/~vcs-imports/openldap/master/revision/20757
Hopefully we will have the package released soon.

[Maciej Puzio (maciej-puzio)]

I created patched openldap packages for xenial, available on the same PPA as above. I tested amd64 packages on xenial beta 2.

[Maciej Puzio (maciej-puzio)]

I reported the bug to Debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820244

[Marc Deslauriers (mdeslaur)]

Thanks for the patched packages!

I've uploaded your changes to yakkety with a slight change in the changelog to better describe the issue. I've also uploaded updates to wily and xenial for processing by the SRU team. Thanks!

[Marc Deslauriers (mdeslaur)]

As per the SRU requirements, could you please update the bug description with a testing procedure? See here for more information:

https://wiki.ubuntu.com/StableReleaseUpdates

Thanks!

[Maciej Puzio (maciej-puzio)]

Due to the nature of this bug (referencing previously freed memory leading to an undefined behavior), a reliable testing procedure is difficult to create. This bug was originally found by looking for a cause of syncrepl failures. The reproducibility of these failures was about 50%, enough to make syncrepl unusable, but syncrepl would persistently fail or persistently work correctly, sometimes for long stretches of testing iterations. While trying to set a test environment using virtual machines, I was unable to reproduce the syncrepl failures at all.

Because of that, in my original bug report to OpenLDAP project, I did not describe steps to reproduce the problem, but instead provided a debugging patch that reliably demonstrated the use-after-free issue. This patch replaced the offending free with an assignment of a special value to the variable that was to be freed. The value of that variable was then examined in places where it was accessed. However, while this approach demonstrates the bug well, it requires a rebuild of the code, and cannot be used to test the fixed package.

I would like to add that I went the "debug-it-yourself" route precisely because the symptoms were too unpredictable and too "mysterious" to hope for the usual bug report to succeed (by "usual bug report" I mean complaining about symptoms, listing steps to reproduce, etc).

To sum up, I can list steps I took during my testing, but these will be of limited use when reproducibility is concerned. I can also provide the debug patch with explanations. Please advise on what would be the best course of action.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.42+dfsg-2ubuntu4

---------------
openldap (2.4.42+dfsg-2ubuntu4) yakkety; urgency=medium

  * Fix use after free with GnuTLS. (LP: #1557248)

 -- Maciej Puzio <email address hidden> Fri, 25 Mar 2016 15:24:25 -0500

[Chris J Arges (arges)]

Hello Maciej, or anyone else affected,

Accepted openldap into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.4.41+dfsg-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Maciej, or anyone else affected,

Accepted openldap into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.4.42+dfsg-2ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Maciej Puzio (maciej-puzio)]

Chris, thank you very much for preparing the packages for -proposed repos. I started testing of xenial-proposed version, but tests are not progressing quickly, due to issues that I described above. In addition I have run into another problem, likely unrelated to this bug, which is further obscuring the results. Because of that I need more time to come up with reliable results; I hope to be ready next week. Thanks for your patience.

[Maciej Puzio (maciej-puzio)]

I can confirm that the following packages from xenial-proposed fix the bug:
slapd 2.4.42+dfsg-2ubuntu3.1
libldap-2.4-2 2.4.42+dfsg-2ubuntu3.1
ldap-utils 2.4.42+dfsg-2ubuntu3.1

I did not test the packages in wily-proposed. Setting the test environment is not trivial, and I don't think it is worthwhile to make this effort for the release that goes out of support in two months, and has been already superseded by a LTS release.

I apologize for a delay in replying to the verification request. This was caused by an unpleasant surprise encountered while testing the new packages. I attempted to recreate the test environment to mimic the setup in which I originally encountered this bug, but I did so slightly differently - and discovered another OpenLDAP bug that had basically the same symptoms. It was not immediately clear whether this situation was some unfixed edge case of the bug reported here, or if it was an entirely separate bug. Further analysis showed that it was the latter, the root cause is entirely different and similarities are coincidental. For reference, report for the new bug can be found at http://www.openldap.org/its/index.cgi?findid=8427

Testing methodology and environment:

Tests were done with both fixed and unfixed versions of affected packages, i.e. 2.4.42+dfsg-2ubuntu3 and 2.4.42+dfsg-2ubuntu3.1. Note that symptoms of this bug are intermittent, and several iterations may be needed for them to surface.

1. Configure two LDAP servers in dual master replication setup using slapd.conf config file as shown below.
2. Provide the servers with TLS certificates that are correct but use 1024-bit public key. (Note: SECURE256 requires 4096-bit RSA key)
3. Set tls_reqcert to allow in slapd.conf.
4. Start slapd on both servers.
5. Stop and restart slapd on server A.
6. Server B will write errors to syslog:
   slapd: do_syncrep2: rid=001 (-1) Can't contact LDAP server
   slapd: do_syncrepl: rid=001 rc -1 retrying (9 retries left)

Result when using fixed packages:
After predefined time server B will retry replication, and we won't see any further error messages.

Result when using unfixed packages:
Server B produces the following messages in a loop:
   slapd: do_syncrepl: rid=001 rc -1 retrying (8 retries left)
   slapd: slap_client_connect: URI=ldaps://10.0.0.1 DN="cn=root,dc=test" ldap_sasl_bind_s failed (-1)

The relevant parts of slapd.conf: (for server A at 10.0.0.1)

loglevel 1
serverID 001
moduleload syncprov
TLSCipherSuite SECURE256:-VERS-SSL3.0
TLSCACertificateFile /etc/ldap/ssl/ca.pem
TLSCertificateFile /etc/ldap/ssl/srvA.pem
TLSCertificateKeyFile /etc/ldap/ssl/srvA.key
syncrepl rid=001
        provider=ldaps://10.0.0.2
        type=refreshAndPersist
        retry="30 10 300 +"
        searchbase="dc=test"
        attrs="*,+"
        bindmethod=simple
        binddn="cn=root,dc=test"
        credentials="plaintext-password"
        tls_reqcert=allow
        keepalive="240:5:10"
mirrormode TRUE
overlay syncprov
syncprov-checkpoint 10 1440

[Martin Pitt (pitti)]

The verification of the Stable Release Update for openldap has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.42+dfsg-2ubuntu3.1

---------------
openldap (2.4.42+dfsg-2ubuntu3.1) xenial; urgency=medium

  * Fix use after free with GnuTLS. (LP: #1557248)

 -- Maciej Puzio <email address hidden> Fri, 25 Mar 2016 15:24:25 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.4.41+dfsg-1ubuntu2.1

---------------
openldap (2.4.41+dfsg-1ubuntu2.1) wily; urgency=medium

  * Fix use after free with GnuTLS. (LP: #1557248)

 -- Maciej Puzio <email address hidden> Wed, 23 Mar 2016 13:42:50 -0500