Intermittent SSL connection faults when using TLSv1

[Darren Spiteri]

Binary package hint: openssl

Reported intermittent SSL connection issue on some apache mod_ssl vhosts.

Platform: Ubuntu 10.04.2 LTS
Tested: Apache2-2.2.14-5ubuntu8.4 and backported 2.2.17-1ubuntu1 from Natty

Firefox client will intermittently report:
Secure Connection Failed
An error occurred during a connection to oem-ibs.canonical.com.
Peer's certificate has an invalid signature.
(Error code: sec_error_bad_signature)

Condition will clear on reload.

Occassionally the server will alternately serve a good page followed by an SSL error until Apache is restarted. I am unable to reproduce the condition on demand, but have output from when the fault occurs. When the fault condition occurs it can be reproduced with any SSL client.

The fault presents on multiple distinct servers.

Initially suspected to be a bug with mod_ssl https://issues.apache.org/bugzilla/show_bug.cgi?id=46952, backport has eliminated this as has anecdotal reports of this same error presented from Dovecot.

Tested with SSL certs from different CAs.

Example:

$ openssl s_client -connect oem-ibs.canonical.com:443
CONNECTED(00000003)
depth=2 /C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
verify error:num=20:unable to get local issuer certificate
verify return:0
14563:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:
14563:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:697:
14563:error:1408D07B:SSL routines:SSL3_GET_KEY_EXCHANGE:bad signature:s3_clnt.c:1449:

[Darren Spiteri (dspiteri)]

And Lucid openssl 0.9.8k-7ubuntu8.6

[Andreas Hasenack (ahasenack)]

I also see this with other servers, like admin.landscape.canonical.com. We are using a wildcard cert, I suspect the OP is too.

[Marc Deslauriers (mdeslaur)]

Interesting...I think the next thing to try would be a backport of openssl from natty to lucid. I can prepare one if you are willing to give it a try...

[Andreas Hasenack (ahasenack)]

Do you know of some good debugging options we could use on the server side? I'm reading through the docs right now. We could enable debugging on our staging server I think, even if it takes days or weeks for the problem to happen again.

About trying the backport, I'll leave that question for our sysadmins who are also subscribed to this bug.

[Marc Deslauriers (mdeslaur)]

Not really, besides "LogLevel debug", but that will fill up your log file.

[Darren Spiteri (dspiteri)]

Hi I'll try backporting the Natty openssl package and see how it goes.

Not using a wildcard cert, although I have tested with one, as well as two seperate certs.

I have plenty of Apache debug logs, I'll distill some and upload when I have a moment Here's an ssldump that accompanied the s_client output above:

7 1 0.3464 (0.3464) C>S SSLv2 compatible client hello
  Version 3.1
  cipher suites
  Unknown value 0x39
  Unknown value 0x38
  Unknown value 0x35
  TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
  TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
  TLS_RSA_WITH_3DES_EDE_CBC_SHA
  SSL2_CK_3DES
  Unknown value 0x33
  Unknown value 0x32
  Unknown value 0x2f
  SSL2_CK_RC2
  TLS_RSA_WITH_RC4_128_SHA
  TLS_RSA_WITH_RC4_128_MD5
  SSL2_CK_RC4
  TLS_DHE_RSA_WITH_DES_CBC_SHA
  TLS_DHE_DSS_WITH_DES_CBC_SHA
  TLS_RSA_WITH_DES_CBC_SHA
  SSL2_CK_DES
  TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
  TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
  TLS_RSA_EXPORT_WITH_DES40_CBC_SHA
  TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
  SSL2_CK_RC2_EXPORT40
  TLS_RSA_EXPORT_WITH_RC4_40_MD5
  SSL2_CK_RC4_EXPORT40
  Unknown value 0xff
7 2 0.3557 (0.0093) S>CV3.1(81) Handshake
      ServerHello
        Version 3.1
        random[32]=
          4d f1 5f 69 e8 65 f9 9e 0e 21 fd f8 6e 05 11 bb
          45 6b b8 97 49 62 04 68 60 a2 4a 94 11 4a 81 84
        session_id[32]=
          c0 ca 5b 73 a3 9a 33 0a 65 30 8f 28 c2 db d1 d6
          47 ff b6 0c bf 48 0f dd 1e 95 33 9b 56 8b 04 3e
        cipherSuite Unknown value 0x39
        compressionMethod NULL
7 3 0.3557 (0.0000) S>CV3.1(3382) Handshake
      Certificate
7 4 0.3557 (0.0000) S>CV3.1(525) Handshake
      ServerKeyExchange
7 5 0.3557 (0.0000) S>CV3.1(4) Handshake
      ServerHelloDone
7 6 0.7052 (0.3494) C>SV3.1(2) Alert
    level fatal
    value decrypt_error
7 0.7054 (0.0002) S>C TCP FIN
7 0.7066 (0.0012) C>S TCP RST

For comparison, here's the ssldump of the prior, successful connection:

6 1 0.3416 (0.3416) C>S SSLv2 compatible client hello
  Version 3.1
  cipher suites
  Unknown value 0x39
  Unknown value 0x38
  Unknown value 0x35
  TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
  TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
  TLS_RSA_WITH_3DES_EDE_CBC_SHA
  SSL2_CK_3DES
  Unknown value 0x33
  Unknown value 0x32
  Unknown value 0x2f
  SSL2_CK_RC2
  TLS_RSA_WITH_RC4_128_SHA
  TLS_RSA_WITH_RC4_128_MD5
  SSL2_CK_RC4
  TLS_DHE_RSA_WITH_DES_CBC_SHA
  TLS_DHE_DSS_WITH_DES_CBC_SHA
  TLS_RSA_WITH_DES_CBC_SHA
  SSL2_CK_DES
  TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
  TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
  TLS_RSA_EXPORT_WITH_DES40_CBC_SHA
  TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
  SSL2_CK_RC2_EXPORT40
  TLS_RSA_EXPORT_WITH_RC4_40_MD5
  SSL2_CK_RC4_EXPORT40
  Unknown value 0xff
6 2 0.3512 (0.0095) S>CV3.1(81) Handshake
      ServerHello
        Version 3.1
        random[32]=
          4d f1 5f 5c 41 3e 94 a9 68 9d 48 73 90 29 b2 08
          62 b4 b6 6a 6b 98 ac 81 70 7d 44 a7 0c 6d fe ef
        session_id[32]=
          dd 42 bf a7 3b 46 a0 eb 38 19 a0 bf 56 c1 22 17
          1c aa b4 0c 97 79 ea b7...

[Andreas Hasenack (ahasenack)]

Do any of you have compression (deflate) enabled server-side on these machines where the error occurs?

[Darren Spiteri (dspiteri)]

On 06/28/2011 04:02 AM, Andreas Hasenack wrote:
> Do any of you have compression (deflate) enabled server-side on these
> machines where the error occurs?
>

Deflate was disabled during inital testing on the oem-ibs site.

[Andreas Hasenack (ahasenack)]

Right now, admin.landscape.canonical.com is failing again:

andreas@nsn2:~$ gnutls-cli admin.landscape.canonical.com
Resolving 'admin.landscape.canonical.com'...
Connecting to '91.189.90.188:443'...
*** Fatal error: Decryption has failed.
*** Handshake has failed
GNUTLS ERROR: Decryption has failed.
andreas@nsn2:~$

I tried gnutls-cli-debug against it, and the result is interesting, basically nothing is shown as supported:

andreas@nsn2:~$ gnutls-cli-debug admin.landscape.canonical.com
Resolving 'admin.landscape.canonical.com'...
Connecting to '91.189.90.188:443'...
Checking for TLS 1.1 support... no
Checking fallback from TLS 1.1 to... failed
Checking for TLS 1.0 support... no
Checking for SSL 3.0 support... no

Server does not support any of SSL 3.0, TLS 1.0 and TLS 1.1

Then I hit staging.landscape.canonical.com, which is a vhost (using SNI) on the same machine, but that is not exhibiting this behavior:

andreas@nsn2:~$ gnutls-cli staging.landscape.canonical.com
Resolving 'staging.landscape.canonical.com'...
Connecting to '91.189.90.188:443'...
- Ephemeral Diffie-Hellman parameters
 - Using prime: 1024 bits
 - Secret key: 1021 bits
 - Peer's public key: 1023 bits
- Certificate type: X.509
 - Got a certificate list of 3 certificates.
 - Certificate[0] info:
(...)

But the debug output is the same:

andreas@nsn2:~$ gnutls-cli-debug staging.landscape.canonical.com
Resolving 'staging.landscape.canonical.com'...
Connecting to '91.189.90.188:443'...
Checking for TLS 1.1 support... no
Checking fallback from TLS 1.1 to... failed
Checking for TLS 1.0 support... no
Checking for SSL 3.0 support... no

Server does not support any of SSL 3.0, TLS 1.0 and TLS 1.1

For comparison, here is what -debug says about Launchpad:

andreas@nsn2:~$ gnutls-cli-debug launchpad.net
Resolving 'launchpad.net'...
Connecting to '91.189.89.222:443'...
Checking for TLS 1.1 support... no
Checking fallback from TLS 1.1 to... TLS 1.0
Checking for TLS 1.0 support... yes
Checking for SSL 3.0 support... yes
Checking for HTTPS server name... not checked
Checking for version rollback bug in RSA PMS... no
Checking for version rollback bug in Client Hello... no
Checking whether we need to disable TLS 1.0... N/A
Checking whether the server ignores the RSA PMS version... no
Checking whether the server can accept Hello Extensions... yes
Checking whether the server can accept cipher suites not in SSL 3.0 spec... yes
Checking whether the server can accept a bogus TLS record version in the client hello... no
Checking for certificate information... N/A
Checking for trusted CAs... N/A
Checking whether the server understands TLS closure alerts... yes
Checking whether the server supports session resumption... yes
Checking for export-grade ciphersuite support... no
Checking RSA-export ciphersuite info... N/A
Checking for anonymous authentication support... no
Checking anonymous Diffie-Hellman group info... N/A
Checking for ephemeral Diffie-Hellman support... yes
Checking ephemeral Diffie-Hellman group info... N/A
Checking for AES cipher support (TLS extension)... yes
Checking for CAMELLIA cipher support (TLS extension)... no
Checking for 3DES cipher support... yes
Checking for ARCFOUR 128 ...

[Andreas Hasenack (ahasenack)]

I guess the output is not so interesting after all. It only shows that gnutls-cli-debug didn't manage to establish a secure session. After the graceful, it's working again as expected.

[Steve Magoun (smagoun)]

Set OEM-Priority importance, this is driving us nuts :)

[Steve Langasek (vorlon)]

Colin, do you know what's going on here?

[Steve Magoun (smagoun)]

@Darren - did you try the natty-->Lucid backport you mentioned in comment #6?

[Charlie Schluting ☃ (cschluti)]

The natty->lucid backport was completed (and installed on oem-ibs), and the same issue was experienced again yesterday.

What other information/testing can IS provide?

[Marc Deslauriers (mdeslaur)]

Since it wasn't an issue with openssl, I think the next step to try would be a backport of apache2 from Oneiric to lucid...

[Geoff Talvola (gtalvola)]

I was seeing this exact error (on Lucid), and yesterday I switched from apache2-mpm-worker to apache2-mpm-prefork and so far after 24 hours the problem hasn't happened again. It's too early to tell if this is a permanent fix but you might consider trying this and see if it helps.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Geoff Talvola (gtalvola)]

Since I switched from apache2-mpm-worker to apache2-mpm-prefork over a month ago, the problem has not happened even once.

[James M. Leddy (jm-leddy)]

Interesting, comment #18 seems to confirm this is an apache bug, or at least that apache2-mpm-prefork is required to reproduce this problem. However, the bug description points to this being a problem in the library because of anecdotal dovecot errors. Does anyone know if those dovecot bugs were reported/fixed?

[gtalvola (geoff-talvola)]

On the contrary, apache2-mpm-worker is required to reproduce the problem. The problem does not happen with apache2-mpm-prefork.

[James M. Leddy (jm-leddy)]

Would one of those affected be able to point me to either a test system or an easy way to reproduce the problem? I think more debugging data is needed.

[Andreas Hasenack (ahasenack)]

I don't have an easy, nor hard way, to reproduce this. At one point we suspected SNI was the culprit, because while the problem was happening, disabling it on the client made it work, but later that was not confirmed.

I also left a machine up with about 5 SSL vhosts and a script connecting to each one of them for days, and the problem didn't happen.

To be honest, I haven't seen it happening for a while on our hosts (staging.landscape.canonical.com and landscape.canonical.com). So unless someone is gracefulling apache every now and then and I don't know about it, the problem seems to have gone away.

[Nicola Heald (notnownikki)]

This happened a lot yesterday on the hexr.canonical.com server. It's not in a pattern I can predict, nor does it only happen on certain pages. When it's happening, it affects both serving of the web app and the static assets.

[James M. Leddy (jm-leddy)]

Okay, I just got a failure from hexr. I'm going to debug this as much as possible from the client side and then we'll take a look in to debugging hexr to diagnose the problem.

[James M. Leddy (jm-leddy)]

I tried to reproduce this, and unfortunately I haven't been able to get any failures since I did the first time in comment #24. When it does happen it is because the certificate is invalid.

[James M. Leddy (jm-leddy)]

could be a memory allocation issue.

https://code.google.com/p/cherokee/issues/detail?id=594

[James M. Leddy (jm-leddy)]

Okay, hexr is acting up again. Here is some verbose gnutls-client output :

$ gnutls-cli -d 5 hexr.canonical.com
Resolving 'hexr.canonical.com'...
Connecting to '91.189.89.67:443'...
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_RSA_AES_128_CBC_SHA256
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_RSA_AES_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_RSA_CAMELLIA_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_RSA_AES_256_CBC_SHA256
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_RSA_AES_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_RSA_CAMELLIA_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_RSA_3DES_EDE_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_DSS_AES_128_CBC_SHA256
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_DSS_AES_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_DSS_CAMELLIA_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_DSS_AES_256_CBC_SHA256
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_DSS_AES_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_DSS_CAMELLIA_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_DSS_3DES_EDE_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_DSS_ARCFOUR_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_PSK_SHA_AES_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_PSK_SHA_AES_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_PSK_SHA_3DES_EDE_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: DHE_PSK_SHA_ARCFOUR_SHA1
|<3>| HSK[0x10e1b30]: Removing ciphersuite: SRP_SHA_RSA_AES_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Removing ciphersuite: SRP_SHA_RSA_AES_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Removing ciphersuite: SRP_SHA_RSA_3DES_EDE_CBC_SHA1
|<3>| HSK[0x10e1b30]: Removing ciphersuite: SRP_SHA_DSS_AES_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Removing ciphersuite: SRP_SHA_DSS_AES_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Removing ciphersuite: SRP_SHA_DSS_3DES_EDE_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: RSA_AES_128_CBC_SHA256
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: RSA_AES_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: RSA_CAMELLIA_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: RSA_AES_256_CBC_SHA256
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: RSA_AES_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: RSA_CAMELLIA_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: RSA_3DES_EDE_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: RSA_ARCFOUR_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: RSA_ARCFOUR_MD5
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: PSK_SHA_AES_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: PSK_SHA_AES_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: PSK_SHA_3DES_EDE_CBC_SHA1
|<3>| HSK[0x10e1b30]: Keeping ciphersuite: PSK_SHA_ARCFOUR_SHA1
|<3>| HSK[0x10e1b30]: Removing ciphersuite: SRP_SHA_AES_128_CBC_SHA1
|<3>| HSK[0x10e1b30]: Removing ciphersuite: SRP_SHA_AES_256_CBC_SHA1
|<3>| HSK[0x10e1b30]: Removing ciphersuite: SRP_SHA_3DES_EDE_CBC_SHA1
|<2>| EXT[0x10e1b30]: Sending extension CERT_TYPE
|<2>| EXT[0x10e1b30]: Sending extension SERVER_NAME
|<2>| EXT[0x10e1b30]: Sending extension SAFE_RENEGOTIATION
|<2>| EXT[0x10e1b30]: Send...

[James M. Leddy (jm-leddy)]

Interestingly, I can't reproduce with openssl. As such, this _may_ be a different bug than in the description. One important difference between openssl and both firefox/gnutls is that openssl uses SSLv3, whereas both firefox and gnutls use TLSv1. So it would appear that hexr handles all SSLv3 connections well, but not all TLSv1 connections.

I have a bunch of packet dumps and cmdline output I'll upload presently.

[James M. Leddy (jm-leddy)]

here is the tshark version of a failure when using gnutls

[James M. Leddy (jm-leddy)]

Packet capture of a few attempts to reproduce the failure via 'gnutls-cli -d 5 hexr.canonical.com'. There will be a few successful handshakes before there is a failure. You'll notice at frame 1086 we have "Alert (Level: Fatal, Description: Bad Record MAC)" sent from the client to the server, notifying that it can't decode the certificate.

[James M. Leddy (jm-leddy)]

Packet capture of a few attempts to reproduce the failure with firefox. There will be a few successful handshakes before there is a failure. You'll notice at frame 1086 we have "TLSv1 Record Layer: Alert (Level: Fatal, Description: Decrypt Error)" sent from the client to the server, notifying that it can't decode the certificate.

[James M. Leddy (jm-leddy)]

See previous comment.

One thing I forgot to mention is both of these captures look pretty dirty on the TCP protocol level, with dup acks from the server to me and other weirdness.

[James M. Leddy (jm-leddy)]

Capture illustrating several successful connections with "openssl s_client -connect hexr.canonical.com:443". The large difference here is that openssl uses SSLv3 as opposed to TLSv1. No duplicate acks here for whatever reason.

[James M. Leddy (jm-leddy)]

another report, with openssl 0.9.8g http://www.hiawatha-webserver.org/weblog/24

[Kevin Krafthefer (krafthefer)]

workaround in place

[Andreas Hasenack (ahasenack)]

What's the workaround? You disabled TLSv1?

[James M. Leddy (jm-leddy)]

The workaround is to use apache2-mpm-prefork. We haven't seen this outside of the internal environment. Also, we're upgrading to 12.04 soon, and have been using the workaround for a while now. Hence the low priority.

[Jason Robinson (jaywink)]

This bug has been driving me insane lately - or if it is not the same bug then the symptoms are identical.

What I have is a 14.04.2 LTS server that has had a Rails app running for some time without this problem. Now, approx a month ago, I started seeing this problem where on Firefox I intermittently get the 'sec_error_bad_signature' error. I have not tried to reproduce it in Chrom(e/ium) since as said it happens only sometimes. When it happens on Firefox, a few reloads solve the problem.

BUT it's not only Firefox. I have a monitoring system by my host which gives me downtime on an HTTPS check approx 20 times a day, typically in two batches - probably because at some point Apache restarts and the error goes away for a while. The error it gets from the check is:

> Exception: #<OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server key exchange B: bad signature>

Also, I monitor sites with Uptimerobot too, which also gives a failure at same time. During a few hours it gives 10-20 up/down notifications as the web site is flaky.

In the Apache error log, I can see this:

> [Sun May 10 06:25:20.149372 2015] [ssl:warn] [pid 32429:tid 140264803096448] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
> [Sun May 10 06:25:20.149467 2015] [mpm_event:notice] [pid 32429:tid 140264803096448] AH00489: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
> [Sun May 10 06:25:20.149478 2015] [core:notice] [pid 32429:tid 140264803096448] AH00094: Command line: '/usr/sbin/apache2'
> [Sun May 10 07:29:00.154618 2015] [core:notice] [pid 32429:tid 140264803096448] AH00051: child pid 19109 exit signal Segmentation fault (11), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': double free or corruption (!prev): 0x00007f91b80096c0 ***
> [Sun May 10 07:42:06.978787 2015] [core:notice] [pid 32429:tid 140264803096448] AH00051: child pid 20347 exit signal Aborted (6), possible coredump in /etc/apache2
> [Sun May 10 14:53:43.497375 2015] [mpm_event:notice] [pid 32429:tid 140264803096448] AH00491: caught SIGTERM, shutting down
> [Sun May 10 14:53:44.534805 2015] [mpm_event:notice] [pid 28764:tid 140143992375168] AH00489: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
> [Sun May 10 14:53:44.534899 2015] [core:notice] [pid 28764:tid 140143992375168] AH00094: Command line: '/usr/sbin/apache2'
> [Sun May 10 15:39:34.413988 2015] [core:notice] [pid 28764:tid 140143992375168] AH00051: child pid 28768 exit signal Segmentation fault (11), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': double free or corruption (!prev): 0x00007f75b0006f80 ***
> [Sun May 10 16:31:14.678905 2015] [core:notice] [pid 28764:tid 140143992375168] AH00051: child pid 30167 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': double free or corruption (!prev): 0x00007f75ac009780 ***
> [Sun May 10 19:46:56.937817 2015] [core:notice] [pid 28764:tid 140143992375168] AH00051: child pid 32308 exit signal Aborted (6), possible coredump in /...

[Seth Arnold (seth-arnold)]

Jason, it'd probably be best to file a new bug report, openssl and apache have both changed a fair amount in the last five years.

Thanks