Crash with SSL transfers since upgraded to OpenSSL 1.0 beta

[eMTee]

To reproduce the crash
 start an SSL download from someone with high upload bandwith. 200-300KiB/sec or more
 while the high speed download runs, select 20-30 filelists from an adc hub and download them at once
 this should trigger the crash within seconds, at least at me

It can crash with lower speed transfers as well, but it can take up to several days to crash then. I experience this since the upgrade to Beta 1 of OpenSSL 1.0.0.

bzr 1773, WinXP, also tested with another build with gcc (given by poy) so it shouldn't be a compiling issue.
I can't reproduce the crash with the MSVC build of the same revision.

--

Connected to SSL client using DHE-RSA-AES256-SHA
Thrown: SocketException: Kapcsolat lez+írva
BufferedSocket::run() end 0xbf4e6a0
threadAccBeufpf[New thread 5400.0x924]
BufferedtSocket::run() start
 0xbf4e6a0
eredSocket::accept() 0xbf4e6a0
BufferedSocket::accept()[New thread 5400.0x13c4]
BufCfCeored Soc0noxknnebenftec:etc9:ter9de8ud n0t
(to) So CS osSLntS ancLer ltcict el0enidxteb n tfuteos 9ui9SnsS8ig0L n
gDtcH lhDEriH-eeREan-StdARA S-ucAAcs-EieSApnE2gt5S
 26D5-H6-ESS-HHRAAS

A-AES256-SH
Program received signal SIGSEGV, Segmentation fault.
[Switching to thread 5400.0x1460]
0x77c46fa3 in msvcrt!memcpy () from C:\WINDOWS\system32\msvcrt.dll
(gdb) bt full
#0 0x77c46fa3 in msvcrt!memcpy () from C:\WINDOWS\system32\msvcrt.dll
No symbol table info available.
#1 0x006a3005 in EVP_MD_CTX_copy_ex (out=0xb8bb9f0, in=0xc417aa8)
    at ./crypto/evp/digest.c:291
        tmp_buf = <value optimized out>
#2 0x006af074 in HMAC_CTX_copy (dctx=0xb8bb9bc, sctx=0xc417a74)
    at ./crypto/hmac/hmac.c:171
No locals.
#3 0x0b8bb9a8 in ?? ()
No symbol table info available.
#4 0x006f63e1 in pkey_hmac_copy (dst=0xb8bb968, src=0xc1b0d68)
    at ./crypto/hmac/hm_pmeth.c:103
        sctx = (HMAC_PKEY_CTX *) 0xc417a60
#5 0x006b41ad in EVP_PKEY_CTX_dup (pctx=0xc1b0d68)
    at ./crypto/evp/pmeth_lib.c:279
        rctx = (EVP_PKEY_CTX *) 0xb8bb968
#6 0x129ffd14 in ?? ()
No symbol table info available.
#7 0x006a3021 in EVP_MD_CTX_copy_ex (out=0xa658e0, in=0x129ffcfc)
    at ./crypto/evp/digest.c:298
        tmp_buf = (
    unsigned char *) 0xb8bb8e8 "\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­
s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦\r­s¦zzz
zzzzzt¦t¦"
#8 0x006b700e in EVP_DigestSignFinal (ctx=0x129ffcfc, sigret=0x129ffd88 "",
    siglen=0x129ffd24) at ./crypto/evp/m_sigver.c:144
        md = "\037r\237\022ZVl\000\f¦\213\v\037r\237\022\005\000\000\000\b\000\0
00\000\223FE\017\000\000\000\000\004¦\213\vRR\237\022R°/\fÓ?A\fý¨/\f\024bo\000RS
\213\v\020py\v"
        mdlen = <value optimized out>
        r = <value optimized out>
        tmp_ctx = {digest = 0x0, engine = 0x0, flags = 0, md_data = 0x0,
  pctx = 0x0, update = 0}
#9 0x0068750b in tls1_mac (ssl=0xc0371d0, md=0x129ffd88 "", send=0)
    at ./ssl/t1_enc.c:899
        rec = (SSL3_RECORD *) 0xc2ff9ec
        seq = (unsigned char *) 0xc2ff8e8 ""
        hash = (EVP_MD_CTX *) 0xcc6fae0
        md_size = 20
        hmac = {digest = 0x9f4b28, engine = 0x0, flags = 8, md_data = 0x0,
  pctx = 0xc1b0d68, update = 0x6f61f4 <int_update>}
        mac_ctx = (EVP_MD_CTX *) 0x129ffcfc
        buf = "\027\003\001\000"
        stream_mac = 0
        t = <value optimized out>
#10 0x00680123 in ssl3_read_bytes (s=0xc0371d0, type=23,
    buf=0x129ffe97 "\022@P+\f", len=1, peek=1) at ./ssl/s3_pkt.c:447
        al = <value optimized out>
        i = <value optimized out>
        ret = <value optimized out>
        n = 17736
        cb = (void (*)(const SSL *, int, int)) 0
#11 0x0067dc8d in ssl3_read_internal (s=0xc0371d0, buf=0x129ffe97, len=1,
    peek=1) at ./ssl/s3_lib.c:3254
        ret = <value optimized out>
#12 0x00608260 in dcpp::SSLSocket::wait (this=0xb3d04c0, millis=250,
    waitFor=2) at dcpp/SSLSocket.cpp:171
        c = 18 '\022'
#13 0x00558885 in dcpp::BufferedSocket::checkSocket (this=0xcc35008)
    at dcpp/BufferedSocket.cpp:432
        waitFor = -2142251686
#14 0x0055977a in dcpp::BufferedSocket::run (this=0xcc35008)
    at dcpp/BufferedSocket.cpp:451
        e = (const Exception &) @0x7c810669: {<exception> = {
    _vptr$exception = 0x5053ed33, data_ = {px_ = 0xeee9006a},
    throw_function_ = 0x0, throw_file_ = 0x0, throw_line_ = 786438}, error = {
    static npos = 4294967295,
    _M_dataplus = {<allocator<char>> = {<new_allocator<char>> = {<No data fields
>}, <No data fields>},
      _M_p = 0xeee9006a <Address 0xeee9006a out of bounds>}}}
#15 0x007f6502 in dcpp::Thread::starter (p=0xcc35040) at dcpp//Thread.h:132
        t = (Thread *) 0xcc35040
#16 0x7c80b699 in KERNEL32!GetModuleFileNameA ()
   from C:\WINDOWS\system32\kernel32.dll
No symbol table info available.
#17 0x00000000 in ?? ()
No symbol table info available.
(gdb)

[Edward Sheldrake (ejs1920)]

This could have something to do with CRYPTO_set_locking_callback() not being anywhere in DC++, which may be needed to set up openssl for thread safety:
http://www.openssl.org/support/faq.html#PROG1
http://www.openssl.org/docs/crypto/threads.html

It was added to StrongDC's CryptoManager a while ago, it should be easy to copy and paste the relevant code into DC++ to test this idea.

[eMTee (realprogger)]

Adding the related changes from StrongDC++ 2.22 source didn't solved the problem. It crashes the same way...

[poy (poy)]

updated trace with rev1978:

Program received signal SIGSEGV, Segmentation fault.
[Switching to thread 1168.0xf40]
0x006b44ed in EVP_MD_CTX_copy_ex (out=0x9e36a88, in=0xa1be6e8)
    at ./crypto/evp/digest.c:307
307 ./crypto/evp/digest.c: No such file or directory.
        in ./crypto/evp/digest.c
(gdb) bt full
#0 0x006b44ed in EVP_MD_CTX_copy_ex (out=0x9e36a88, in=0xa1be6e8)
    at ./crypto/evp/digest.c:307
        tmp_buf = (unsigned char *) 0x0
#1 0x006be531 in HMAC_CTX_copy (dctx=0x9e36a6c, sctx=0xa1be6cc)
    at ./crypto/hmac/hmac.c:169
No locals.
#2 0x00714799 in pkey_hmac_copy (dst=0xae0f570, src=0xf391f30)
    at ./crypto/hmac/hm_pmeth.c:103
        sctx = (HMAC_PKEY_CTX *) 0xa1be6b8
#3 0x006c4e4a in EVP_PKEY_CTX_dup (pctx=0xf391f30)
    at ./crypto/evp/pmeth_lib.c:287
        rctx = (EVP_PKEY_CTX *) 0xae0f570
#4 0x006b4475 in EVP_MD_CTX_copy_ex (out=0xab9c10, in=0xd7ffcd4)
    at ./crypto/evp/digest.c:314
        tmp_buf = (unsigned char *) 0x0
#5 0x006cb8f4 in EVP_DigestSignFinal (ctx=0xd7ffcd4,
    sigret=0xb0f41f2 " ▒▒t\v▒0(▒?▒\a▒:\226\203\037▒.\221▒\232▒▒'▒▒s\026▒Ra\027\0
03\001", siglen=0xd7ffcfc) at ./crypto/evp/m_sigver.c:144
        tmp_ctx = {digest = 0x0, engine = 0x0, flags = 0, md_data = 0x0,
  pctx = 0x0, update = 0}
        md = "\000\000\001\000\000\000\000\000▒\202▒\t▒\202▒\tuDk\000\030▒\t\v\0
00\000\000\000(▒▒\nP▒▒\n\000\000\000\000(▒▒\n▒}d\nL▒\r\v\030\020▒\n(▒\r\v▒▒\177\
r"
        mdlen = <value optimized out>
---Type <return> to continue, or q <return> to quit---
        r = <value optimized out>
#6 0x006903f1 in tls1_mac (ssl=0xaf81018,
    md=0xb0f41f2 " ▒▒t\v▒0(▒?▒\a▒:\226\203\037▒.\221▒\232▒▒'▒▒s\026▒Ra\027\003\0
01", send=1) at ./ssl/t1_enc.c:899
        rec = (SSL3_RECORD *) 0xb0df128
        seq = (unsigned char *) 0xb0df04c ""
        hash = (EVP_MD_CTX *) 0xd7ffcd4
        md_size = 20
        hmac = {digest = 0xa4bea0, engine = 0x0, flags = 256, md_data = 0x0,
  pctx = 0xf391f30, update = 0x714590 <int_update>}
        mac_ctx = (EVP_MD_CTX *) 0xd7ffcd4
        buf = "\025\003\001\000\002"
        stream_mac = 0
        t = <value optimized out>
#7 0x00683aa2 in do_ssl3_write (s=0xaf81018, type=<value optimized out>,
    buf=0xb0df188 "\001", len=2, create_empty_fragment=0)
    at ./ssl/s3_pkt.c:771
        p = (unsigned char *) 0xb0f41f0 "\001"
        i = <value optimized out>
        mac_size = 20
        prefix_len = <value optimized out>
        align = <value optimized out>
#8 0x00683b77 in ssl3_dispatch_alert (s=0xaf81018) at ./ssl/s3_pkt.c:1429
        i = <value optimized out>
---Type <return> to continue, or q <return> to quit---
        cb = <value optimized out>
#9 0x0067ff3c in ssl3_shutdown (s=0xaf81018) at ./ssl/s3_lib.c:3166
        ret = <value optimized out>
#10 0x0060cec5 in dcpp::SSLSocket::shutdown (this=0xac0c828)
    at dcpp\SSLSocket.cpp:212
No locals.
#11 0x005ebd8e in dcpp::Socket::disconnect (this=0xac0c828)
    at dcpp\Socket.cpp:627
No locals.
#12 0x00564b78 in dcpp::BufferedSocket::fail (this=0xac0c518,
    aError=@0xab0946c) at dcpp\BufferedSocket.cpp:477
No locals.
#13 0x00564abf in dcpp::B...

[poy (poy)]

fixed by switching to the 0.9 branch of openssl. when compiling 1.0 with MinGW i had to disable the "capieng" option because MinGW is lacking some of the functions necessary to fully compile that module; its absence was inducing the crash.
i don't know the exact reason behind this issue but since the 0.9 branch can be compiled with asm whereas the 1.0 one was also failing there, sticking with the 0.9 builds is fine for now.

[poy (poy)]

reviving this bug as we have upgraded to the 1.0 branch and have been experiencing these crashes again.

the problem wasn't that some modules had to be disabled, since the latest version is now being compiled with all default modules. the actual reason was a missing flag in the MinGW compilation that was making all MinGW builds be "optimized for MS-DOS". this affected previous versions of OpenSSL too, but seems to be causing a crash only in 1.0.

fixed in rev 2304.
i am now going to report this to OpenSSL...

[poy (poy)]

Fixed in DC++ 0.780.