mutex deadlock in pyOpenSSL under PyPy

[Vasily Evseenko]

pyOpenSSL deadlocks while creating SSL connection under PyPy
OS: Fedora 14 x86_64
PyPy-1.5
pyOpenSSL: 0.12
See example in attachment

gdb stack dump:
-------
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
#1 0x0000000000410158 in RPyThreadAcquireLock ()
#2 0x00000000004db01b in pypy_g_after_external_call ()
#3 0x00000000008edfb7 in pypy_g_wrapper ()
#4 0x0000003107a39dcf in SSL_get_ex_data_X509_STORE_CTX_idx ()
    at ssl_cert.c:140
#5 0x0000003107a3768b in SSL_CTX_new (meth=0x3107c557e0) at ssl_lib.c:1532
#6 0x00007ffff73cd33f in ssl_Context_init ()
   from /home/svpcom/work/pypy_exp/qikengine.pypy/env/site-packages/OpenSSL/SSL.pypy-15.so
#7 0x00007ffff73cd503 in ssl_Context_new ()
   from /home/svpcom/work/pypy_exp/qikengine.pypy/env/site-packages/OpenSSL/SSL.pypy-15.so
#8 0x0000000000f6afab in pypy_asm_stackwalk ()
#9 0x0000000000a0ab03 in pypy_g_generic_cpy_call__StdObjSpaceConst_funcPtr_PyTyp_2 ()
#10 0x0000000000944b91 in pypy_g_tp_new_wrapper ()
#11 0x00000000008ee43d in cpyext_tp_new_wrapper ()
#12 0x0000000000f6afab in pypy_asm_stackwalk ()
#13 0x000000000073a0dd in pypy_g_generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI_3 ()
#14 0x000000000050994f in pypy_g_BuiltinCodePassThroughArguments1_funcrun_obj
    ()
#15 0x0000000000508864 in pypy_g_get_and_call_args ()
#16 0x00000000004d303d in pypy_g_call_args ()
#17 0x000000000079beeb in pypy_g_call__Type ()
#18 0x00000000006657d9 in pypy_g__mm_call_typeS0_perform_call ()
#19 0x000000000050994f in pypy_g_BuiltinCodePassThroughArguments1_funcrun_obj
    ()
#20 0x0000000000508864 in pypy_g_get_and_call_args ()
#21 0x00000000004d303d in pypy_g_call_args ()
#22 0x00000000008aa743 in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
#23 0x0000000000833876 in pypy_g_dispatch_bytecode__AccessDirect_None ()
#24 0x00000000007dda83 in pypy_g_handle_bytecode__AccessDirect_None ()
#25 0x0000000000715e65 in pypy_g_portal_1 ()
#26 0x00000000006a9e64 in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter ()
#27 0x0000000000538e38 in pypy_g_PyFrame_execute_frame ()
#28 0x000000000050982c in pypy_g_PyFrame_run ()
#29 0x00000000008aa743 in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
#30 0x0000000000833876 in pypy_g_dispatch_bytecode__AccessDirect_None ()
#31 0x00000000007dda83 in pypy_g_handle_bytecode__AccessDirect_None ()
#32 0x0000000000715e65 in pypy_g_portal_1 ()
#33 0x00000000006a9e64 in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter ()
#34 0x0000000000538e38 in pypy_g_PyFrame_execute_frame ()
.....

[Jean-Paul Calderone (exarkun)]

pyOpenSSL 0.12 doesn't even build on PyPy, so I wonder how this could happen. Running the attached example using pyOpenSSL trunk tip using Twisted 10.0.0, the connection never gets set up, but there is no deadlock. Ctrl-C will interrupt the client and the server, and the client times out on its own after a while. Using Twisted trunk@HEAD, the connection is set up and the transfer succeeds.

[Vasily Evseenko (svpcom)]

Hi,
But we successfully built it with pypy-1.5:
---------
[svpcom@localhost pyOpenSSL-0.12]$ ./env/bin/python
Python 2.7.1 (?, May 25 2011, 11:05:43)
[PyPy 1.5.0-alpha0 with GCC 4.5.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``A language is a dialect with
an
army and navy''
>>>>
---------
See build.log in attachment
"env" directory has been created by virtualenv

About twisted: We use trunk with additional fixes (some are related to
pypy)
Sources are on git://github.com/smira/twisted.git (pypy branch)

2011/5/30 Jean-Paul Calderone <email address hidden>

> pyOpenSSL 0.12 doesn't even build on PyPy, so I wonder how this could
> happen. Running the attached example using pyOpenSSL trunk tip using
> Twisted 10.0.0, the connection never gets set up, but there is no
> deadlock. Ctrl-C will interrupt the client and the server, and the
> client times out on its own after a while. Using Twisted trunk@HEAD,
> the connection is set up and the transfer succeeds.
>
>
> ** Changed in: pyopenssl
> Status: New => Incomplete
>
> ** Changed in: pyopenssl
> Assignee: (unassigned) => Vasily Evseenko (svpcom)
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/790076
>
> Title:
> mutex deadlock in pyOpenSSL under PyPy
>
> Status in pyOpenSSL:
> Incomplete
>
> Bug description:
> pyOpenSSL deadlocks while creating SSL connection under PyPy
> OS: Fedora 14 x86_64
> PyPy-1.5
> pyOpenSSL: 0.12
> See example in attachment
>
> gdb stack dump:
> -------
> #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
> #1 0x0000000000410158 in RPyThreadAcquireLock ()
> #2 0x00000000004db01b in pypy_g_after_external_call ()
> #3 0x00000000008edfb7 in pypy_g_wrapper ()
> #4 0x0000003107a39dcf in SSL_get_ex_data_X509_STORE_CTX_idx ()
> at ssl_cert.c:140
> #5 0x0000003107a3768b in SSL_CTX_new (meth=0x3107c557e0) at
> ssl_lib.c:1532
> #6 0x00007ffff73cd33f in ssl_Context_init ()
> from
> /home/svpcom/work/pypy_exp/qikengine.pypy/env/site-packages/OpenSSL/
> SSL.pypy-15.so
> #7 0x00007ffff73cd503 in ssl_Context_new ()
> from
> /home/svpcom/work/pypy_exp/qikengine.pypy/env/site-packages/OpenSSL/
> SSL.pypy-15.so
> #8 0x0000000000f6afab in pypy_asm_stackwalk ()
> #9 0x0000000000a0ab03 in
> pypy_g_generic_cpy_call__StdObjSpaceConst_funcPtr_PyTyp_2 ()
> #10 0x0000000000944b91 in pypy_g_tp_new_wrapper ()
> #11 0x00000000008ee43d in cpyext_tp_new_wrapper ()
> #12 0x0000000000f6afab in pypy_asm_stackwalk ()
> #13 0x000000000073a0dd in
> pypy_g_generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI_3 ()
> #14 0x000000000050994f in
> pypy_g_BuiltinCodePassThroughArguments1_funcrun_obj
> ()
> #15 0x0000000000508864 in pypy_g_get_and_call_args ()
> #16 0x00000000004d303d in pypy_g_call_args ()
> #17 0x000000000079beeb in pypy_g_call__Type ()
> #18 0x00000000006657d9 in pypy_g__mm_call_typeS0_perform_call ()
> #19 0x000000000050994f in
> pypy_g_BuiltinCodePassThroughArguments1_funcrun_obj
> ()
> #20 0x0000000000508864 in pypy_g_get_and_call_args...

[Jean-Paul Calderone (exarkun)]

Please test with pyOpenSSL trunk at tip instead. There were many bug fixes required to get pyOpenSSL even close to working on PyPy, many involving random memory corruption which will lead to unpredicable, difficult to reproduce behavior.

[Scott Lyons (scottalyons)]

We're seeing this issue (or something closely related to it) running the following:

PyPy 1.7
Twisted 11.1
PyOpenSSL 0.13 / trunk (bug occurs in both)

Traceback from when the lock occurs:

#0 0x00007ffff62877c4 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ffff62825b4 in _L_lock_944 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007ffff62823ca in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x0000000000413b87 in RPyGilAcquire ()
#4 0x0000000000f9af0f in ?? ()
#5 0x0000000000699a30 in ?? ()
#6 0x00007ffff73d6fd7 in CRYPTO_add_lock () from /lib/libcrypto.so.0.9.8
#7 0x00007ffff773517c in SSL_free () from /lib/libssl.so.0.9.8
#8 0x00007ffff4866023 in ssl_Connection_dealloc ()