import OpenSSL segfaults pypy

[Ivan Kozik]

I built a 32-bit -Ojit pypy on Ubuntu 10.04. The pypy revision was r78580 (2010-10-31 09:54). I installed pyOpenSSL r138, but importing it segfaults pypy. I don't know if this also happens in pypy 1.3, because I haven't been able to build any modules with the released version.

# pypy
Python 2.5.2 (78628, Oct 31 2010, 23:53:58)
[PyPy 1.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``pypy is more stable than debian''
>>>> import OpenSSL
<InvalidPointerException object at 0x89302d8>
RPython traceback:
  File "implement.c", line 77916, in PyDict_SetItemString
  File "implement.c", line 175602, in from_ref
  File "implement_4.c", line 129883, in CpyTypedescr_type_realize
  File "implement_1.c", line 37655, in type_realize
  File "implement_1.c", line 117483, in W_PyCTypeObject___init__
  File "implement.c", line 175602, in from_ref
  File "implement_1.c", line 17640, in CpyTypedescr_realize
zsh: segmentation fault pypy

[Ivan Kozik (ludios)]

Same results on a release build, pypy 1.4 32-bit on Ubuntu 10.04 (64-bit):

<InvalidPointerException object at 0x893c554>
RPython traceback:
  File "implement.c", line 79968, in PyDict_SetItemString
  File "implement.c", line 179834, in from_ref
  File "implement_4.c", line 132364, in CpyTypedescr_type_realize
  File "implement_1.c", line 44168, in type_realize
  File "implement_1.c", line 123481, in W_PyCTypeObject___init__
  File "implement.c", line 179834, in from_ref
  File "implement_1.c", line 24349, in CpyTypedescr_realize
Segmentation fault

The exact same thing happens in a 32-bit Ubuntu 10.04 chroot.

[Jean-Paul Calderone (exarkun)]

You should try with the branch linked from this ticket, rather than pyOpenSSL trunk. It still probably won't work, but maybe it will fail less completely.

[Ivan Kozik (ludios)]

I just tried the run-on-pypy branch. It doesn't segfault pypy on import, but none of the tests pass.

elf@linux640:/tmp$ /opt/pypy-1.4/bin/pypy `which trial` OpenSSL.test.test_crypto
OpenSSL.test.test_crypto
  FunctionTests
    test_dump_certificate ... RPython traceback:
  File "implement.c", line 199776, in _PyObject_New
  File "implement.c", line 241951, in _PyObject_NewVar
  File "implement.c", line 179834, in from_ref
  File "implement_4.c", line 132364, in CpyTypedescr_type_realize
  File "implement_1.c", line 44168, in type_realize
  File "implement_1.c", line 123481, in W_PyCTypeObject___init__
  File "implement.c", line 179783, in from_ref
Fatal RPython error: AssertionError
Aborted
elf@linux640:/tmp$ /opt/pypy-1.4/bin/pypy `which trial` OpenSSL.test.test_rand
OpenSSL.test.test_rand
  RandTests
    test_add ... [ERROR]
    test_bytes ... [ERROR]
    test_files ... [ERROR]
    test_seed ... [ERROR]
    test_status ... [ERROR]

===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/opt/pypy-latest/lib-python/2.5.2/unittest.py", line 270, in run
    self.tearDown()
  File "/opt/pypy-latest/site-packages/OpenSSL/test/util.py", line 37, in tearDown
    _exception_from_error_queue()
OpenSSL.SSL.Error: []

OpenSSL.test.test_rand.RandTests.test_add
OpenSSL.test.test_rand.RandTests.test_bytes
OpenSSL.test.test_rand.RandTests.test_files
OpenSSL.test.test_rand.RandTests.test_seed
OpenSSL.test.test_rand.RandTests.test_status
-------------------------------------------------------------------------------
Ran 5 tests in 0.206s

FAILED (errors=5)
elf@linux640:/tmp$ /opt/pypy-1.4/bin/pypy `which trial` OpenSSL.test.test_ssl
OpenSSL.test.test_ssl
  ConnectionTests
    test_get_context ... [ERROR]
                                               [ERROR]
    test_get_context_wrong_args ... [ERROR]
                                    [ERROR]
    test_type ... [ERROR]
                                                      [ERROR]
  ConstantsTests
    test_op_cookie_exchange ... [ERROR]
    test_op_no_query_mtu ... [ERROR]
    test_op_no_ticket ... [ERROR]
  ContextTests
    test_add_extra_chain_cert ... [ERROR]
                                      [ERROR]
    test_add_extra_chain_cert_invalid_cert ... [ERROR]
                         [ERROR]
    test_load_verify_directory ... [ERROR]
                                     [ERROR]
    test_load_verify_file ... [ERROR]
                                          [ERROR]
   ...

[Ivan Kozik (ludios)]

Listening on an SSL endpoint with Twisted (trunk r30354) results in this:

Traceback (most recent call last):
  File "app_main.py", line 33, in run_toplevel
  File "/usr/local/bin/twistd", line 18, in <module>
    run()
  File "/opt/pypy-latest/site-packages/twisted/scripts/twistd.py", line 27, in run
    app.run(runApp, ServerOptions)
  File "/opt/pypy-latest/site-packages/twisted/application/app.py", line 631, in run
    runApp(config)
  File "/opt/pypy-latest/site-packages/twisted/scripts/twistd.py", line 23, in runApp
    _SomeApplicationRunner(config).run()
  File "/opt/pypy-latest/site-packages/twisted/application/app.py", line 374, in run
    self.application = self.createOrGetApplication()
  File "/opt/pypy-latest/site-packages/twisted/application/app.py", line 434, in createOrGetApplication
    ser = plg.makeService(self.config.subOptions)
  File "/home/elf/Projects/Minerva/minerva/tap.py", line 91, in makeService
    httpServer = strports.service(httpStrport, httpSite)
  File "/opt/pypy-latest/site-packages/twisted/application/strports.py", line 77, in service
    endpoints._serverFromStringLegacy(reactor, description, default),
  File "/opt/pypy-latest/site-packages/twisted/internet/endpoints.py", line 719, in _serverFromStringLegacy
    nameOrPlugin, args, kw = _parseServer(description, None, default)
  File "/opt/pypy-latest/site-packages/twisted/internet/endpoints.py", line 710, in _parseServer
    return (endpointType.upper(),) + parser(factory, *args[1:], **kw)
  File "/opt/pypy-latest/site-packages/twisted/internet/endpoints.py", line 576, in _parseSSL
    cf = ssl.DefaultOpenSSLContextFactory(privateKey, certKey, **kw)
  File "/opt/pypy-latest/site-packages/twisted/internet/ssl.py", line 89, in __init__
    self.cacheContext()
  File "/opt/pypy-latest/site-packages/twisted/internet/ssl.py", line 94, in cacheContext
    ctx = self._contextFactory(self.sslmethod)
SystemError: <StackOverflow object at 0x8931114>

[Jean-Paul Calderone (exarkun)]

I have a PyPy branch which partially implements "s*" and "t#" for PyArg_ParseTuple. Against that, the run-on-pypy branch now mostly works. There are still a few failing tests, but the majority of the functionality (according to the test suite) works.

[Jean-Paul Calderone (exarkun)]

As of r150 and PyPy 1.5, this should no longer segfault. There are still a few other issues, but they don't get in the way of most uses.

[rlotun (rlotun)]

On r150 and PyPy 1.5 (on MacOSX snow leopard) I get this:

Python 2.7.1 (b590cf6de419, Apr 30 2011, 03:30:00)
[PyPy 1.5.0-alpha0 with GCC 4.0.1] on darwin
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``it's not a hack, it's a
workaround''
>>>> from OpenSSL import Crypto
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/Cellar/pypy/1.5.0/site-packages/OpenSSL/__init__.py", line 13, in <module>
    from OpenSSL import crypto
ImportError: unable to load extension module '/usr/local/Cellar/pypy/1.5.0/site-packages/OpenSSL/crypto.pypy-15.so': dlopen(/usr/local/Cellar/pypy/1.5.0/site-packages/OpenSSL/crypto.pypy-15.so, 6): Symbol not found: _PyExc_AttributeError
  Referenced from: /usr/local/Cellar/pypy/1.5.0/site-packages/OpenSSL/crypto.pypy-15.so
  Expected in: flat namespace
 in /usr/local/Cellar/pypy/1.5.0/site-packages/OpenSSL/crypto.pypy-15.so
>>>>

Any ideas?

[Jean-Paul Calderone (exarkun)]

Note that there is no "OpenSSL.Crypto" module, but instead a "OpenSSL.crypto" module. Nevertheless, I wasn't able to reproduce this on my Snow Leopard machine using the same build of PyPy. I don't know what the "Expected in: flat namespace" part of the error means. Here's my transcript (containing few, if any, surprises):

    $ pypy
    Python 2.7.1 (b590cf6de419, Apr 30 2011, 03:30:00)
    [PyPy 1.5.0-alpha0 with GCC 4.0.1] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    And now for something completely different: ``we definitely believe in meta''
    >>>> from OpenSSL import Crypto
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
    ImportError: cannot import name 'Crypto'
    >>>> from OpenSSL import crypto
    >>>> crypto
    <module 'OpenSSL.crypto' from '/Users/exarkun/Downloads/pypy-c-jit-43780-b590cf6de419-osx64/site- packages/OpenSSL/crypto.pypy-15.so'>
    >>>>