Segfault on OpenSSL engine initialisation when AES-NI is enabled

[solik]

Binary package hint: apache2

Installed software:
apache2-mpm-prefork 2.2.14-5ubuntu8
php5 5.3.2-1ubuntu4.2
libapache2-mod-php5 5.3.2-1ubuntu4.2
php5-curl 5.3.2-1ubuntu4.2

Ubuntu version: Ubuntu 10.04 LTS

If php5-curl module enabled in the apache2 and access any page on this server with https:// regardless of URL and handler of this URL (php5 or simple static file), apache reset connection logging following in the error.log:

[Mon Jun 07 08:48:58 2010] [notice] Apache/2.2.14 (Ubuntu) DAV/2 SVN/1.6.6 PHP/5.3.2-1ubuntu4.2 with Suhosin-Patch mod_python/3.3.1 Python/2.6.5 mod_ssl/2.2.14 OpenSSL/0.9.8k configured -- resuming normal operations
[Mon Jun 07 08:49:10 2010] [notice] child pid 24120 exit signal Segmentation fault (11)
[Mon Jun 07 08:49:10 2010] [notice] child pid 24121 exit signal Segmentation fault (11)

This bug was appeared after upgrade from 9.10 to 10.04, nothing was changed in the configs, previous version of ubuntu working ok.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: apache2.2-bin 2.2.14-5ubuntu8
ProcVersionSignature: Ubuntu 2.6.32-22.35-generic 2.6.32.11+drm33.2
Uname: Linux 2.6.32-22-generic x86_64
Architecture: amd64
Date: Mon Jun 7 08:49:30 2010
ExecutablePath: /usr/lib/apache2/mpm-prefork/apache2
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
ProcEnviron:
 PATH=(custom, no user)
 LANG=C
SourcePackage: apache2

[Chuck Short (zulcss)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please try to obtain a backtrace following the instructions at http://wiki.ubuntu.com/DebuggingProgramCrash and upload the backtrace (as an attachment) to the bug report. This will greatly help us in tracking down your problem.

[solik (jankkhvej)]

Here is backtrace, thank you for instructions!

[Miroslav Zacek (miroslav-zacek-skype)]

I have the same configuration and also get the segmentation fault. I'd like just to add that the apache crashes even if any SSL page is accessed on that server, not only via php5-curl.

[solik (jankkhvej)]

Just checked it today after update and the bug is gone. All SSL pages works.
So, what should we do now? ;-)

[solik (jankkhvej)]

Miroslav Zacek: that's exactly what I've reported.

[Miroslav Zacek (miroslav-zacek-skype)]

I checked it today and it is still there for me :-(

I checked the code and found that php5-curl in ext/curl/interface.c line 819 calls curl_global_init(CURL_GLOBAL_SSL). If I changed it to CURL_GLOBAL_NOTHING the apache didn't crash. So I followed this code through the libcurl to the openssl library and found out that if remove the initialisation of ENGINE_load_aesni (see debian/patches/aesni.patch line 2429) the php5 curl won't load https pages but the Apache won't segfault.

I have openssl, libssl0.9.8 and libssl-dev version 0.9.8k-7ubuntu8.

[Miroslav Zacek (miroslav-zacek-skype)]

See openssl bug #2305

[Greg Hanley (greg-s-hanley)]

After reading through the comments on the openssl bug report ( http://rt.openssl.org/index.html?q=2305 ) the part about having a Core i7 caught my attention. I am having the same problem but only on one of my servers and not the other; both running 10.04_64 and updated to the same kernel. I have two VMware ESXi servers, one is running quad-core Xeon X5550s and the other is running hex-core Xeon X5650s (same generation as the i7s). I moved the VM that was working from the older server to the new, and when using ssl, Apache started seg faulting, having made no changes to the configuration.

Miroslav Zacek: It sounds like you have found the issue in openssl. Is there any other troubleshooting I can do to help?

[Miroslav Zacek (miroslav-zacek-skype)]

Greg Hanley: it seems the bug was found and will be fixed in the next release of openssl. The reason why it is segfaulting on one machine and not the other is probably the fact that one CPU supports AES-NI and the other not. The problem is in multiple openssl initializations.

If you want to fix it before the official version comes there is a patch: http://rt.openssl.org/Ticket/Attachment/27712/13787/. Either use the patch or do the 2 small changes manually and recompile.

If you do not want to recompile, try to avoid multiple ssl initializations (e.g. in my case these were the apache mod-ssl and php5-curl), but this is not sure that it helps.

[William McGrath (auckland-railgeek)]

Miroslav: silly question; if the bug will be fixed in the next release of openssl, when should that release be available?

[Zhang Huangbin (michaelbibby)]

Any update on this issue?

[Zhang Huangbin (michaelbibby)]

Oops, i tested it in Ubuntu 10.04 (amd64) moment ago, seems it was fixed. at least it works well in iRedMail mail server solution, Thanks very much. :)

[William McGrath (auckland-railgeek)]

Fraid it's still an issue for me. Seems to resolve itself when removing php5-curl from the install.

[Miroslav Zacek (miroslav-zacek-skype)]

It hasn't been fixed yet, unfortunately :-(

So here is a small howto for (k)ubuntu if you need to fix it (root privileges assumed):

1) create a directory somewhere, e.g. /usr/src/openssl, chdir there and run:
apt-get source openssl
apt-get build-dep openssl

2) edit ./openssl-0.9.8k/debian/patches/aesni.patch: on line 1956 change
+ ENGINE_add (toadd);
to
+ if (ENGINE_add (toadd))

3) edit /openssl-0.9.8k/crypto/engine/eng_all.c: on line 64 add this new line (as first row in the ENGINE_load_builtin_engines(void) function):
OPENSSL_cpuid_setup();

4) now build the packages (from /usr/src/openssl/openssl-0.9.8k)
dpkg-buildpackage

5) Then you'll get new packages in /usr/src/openssl. Install libssl0.9.8_0.9.8k-7ubuntu8_*.deb and reboot (to test apache you do not need reboot, just restart apache). Be carefull on updates not to install the original version from the repositories.

[Miroslav Zacek (miroslav-zacek-skype)]

In the previous howto in step 2 please note there is no semicolon at the end of the line after the change. This is correct!. On the following line 1957 there is a function call that should be executed only if ENGINE_add (toadd) returns true. If you keep the semicolon there the code will compile but segmentation errors persist.

[Greg Hanley (greg-s-hanley)]

Thanks for the step-by-step of patching the source Miroslav Zacek! It seems to have fixed the seg faulting I was getting.

[Christie Koehler (christi3k)]

Confirming that the patch posted by Miroslav Zacek works for me as well. Many thanks!

[William McGrath (auckland-railgeek)]

Ditto here. Now we wait for openssl to be updated :-)

[solik (jankkhvej)]

Thank you, Miroslav! Can't test it now, 'cause I'm switched to Mac OS X, but anyway, well done!

[Jan Willem (jwknopper)]

Thanks! This fixes the problem for me as well.

[Wells Oliver (wells-submute)]

Why is this marked as 'low' priority? Apache + SSL + curl is probably a very common combination. This is incredibly tedious..

[Wells Oliver (wells-submute)]

Also confirming that the patch from Miroslav worked for me. AWESOME!

[Rhomboid (rhomboid)]

Thanks Miroslav. That fixed it for me as well. I can't believe this hasn't been fixed.

My test environment was 32-bit so I missed this initially and then spent way too long trying to figure it out.

[Colin Watson (cjwatson)]

I've uploaded a fix to lucid-proposed, waiting for approval. SRU team: apologies for the spurious diffs to Makefile and crypto/opensslconf.h - these just depend on what architecture you built the source package on, and it's difficult to make the package not do this.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 0.9.8o-1ubuntu4

---------------
openssl (0.9.8o-1ubuntu4) maverick; urgency=low

  * Update AES-NI patch to openssl-0.9.8-aesni-modes-perlasm-win32-v4.patch
    from http://rt.openssl.org/Ticket/Display.html?id=2067, fixing segfault
    on engine initialisation (LP: #590639).
 -- Colin Watson <email address hidden> Fri, 24 Sep 2010 12:20:49 +0100

[Martin Pitt (pitti)]

Accepted openssl into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Greg Hanley (greg-s-hanley)]

I installed the new proposed OpenSSL packages and it fixed my issues with apache2/mod_ssl/php5. The segmentation faults that I have been experiencing while using PHP extensions that also use openssl (in my case 'curl' and 'pgsql') are no longer an issue.

[Rhomboid (rhomboid)]

Verified it's fixed for me too using the proposed update (apache2 SSL, calling PHP curl in turn to an SSL site).

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 0.9.8k-7ubuntu8.2

---------------
openssl (0.9.8k-7ubuntu8.2) lucid-proposed; urgency=low

  * Update AES-NI patch to openssl-0.9.8-aesni-modes-perlasm-win32-v4.patch
    from http://rt.openssl.org/Ticket/Display.html?id=2067, fixing segfault
    on engine initialisation (LP: #590639).
 -- Colin Watson <email address hidden> Fri, 24 Sep 2010 12:25:28 +0100