PyOpenSSL breaks python-paste on python-2.7 (due to memoryview)

[Toshio Kuratomi]

fedora 14 and higher use python-2.7 as the default python. On this version, someone discovered that paste wasn't able to serve over https:
  https://bugzilla.redhat.com/show_bug.cgi?id=642809

After some investigation it seems that python-2.7's socket module is now using memoryviews and pyOpenSSL is not able to receive those. I'll attach a patch that rectifies this situation. Note that the patch can only work on python-2.6+ (But is necessary on python-2.7+). I'm not familiar with what #ifdef's exist to make this only apply on python >= 2.6.

[Toshio Kuratomi (toshio)]

The patched version is currently in Fedora's testing repository where people are reporting using it successfully:

  https://admin.fedoraproject.org/updates/pyOpenSSL-0.9-3.fc14

[Jean-Paul Calderone (exarkun)]

Uh, I guess I failed to use the Launchpad comment form correctly last time I was here. I'm *pretty* sure that along with that re-assignment, I was also trying to add a comment asking for instruction about how to reproduce the issue so that I can test the fix. Following the bug link to the python.org bug report, it appears to have something to do with using a file-like interface to sockets. Off the top of my head, I'm sure not sure you might do this with an OpenSSL.SSL.Connection, since that type does not support the `makefile` method.

[Toshio Kuratomi (toshio)]

Memoryviews are more akin to a new type of array than a file::
    http://docs.python.org/library/stdtypes.html#memoryview-type

I'm not sure how to reproduce this outside of paste but here's a way to reproduce within paste:

* Install python-2.7, pylons, and pyOpenSSL
* paster quickstart tester
  - Accept the default values for everything
* cd tester
* edit development.ini so that the [server:main] section looks like this:

[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 8080
ssl_pem = host.pem

* Create a host key:
$ openssl genrsa 1024 > host.key
$ chmod 400 host.key
$ openssl req -new -x509 -nodes -sha1 -days 365 -key host.key > host.cert
$ cat host.cert host.key > host.pem
$ chmod 400 host.pem

* paster serve development.ini
* In another shell, curl -k https://localhost:8080/

Without the attached patch, curl will say that the server returned no data. The server's shell will show a traceback: that ends with (apologies, I ran this in a virtual machine so I can't cut and paste the whole thing. The bugzilla.redhat.com bug has the traceback, though):
    TypeError: must be string or read-only buffer, not memoryview

If you patch pyOpenSSL with the attached patch and rerun paster serve development.ini and curl -k https://localhost:8080/ the page will be returned correctly.

[Matti Lahtinen (worros)]

I ran to this problem simply by trying this example with python 2.7.1:

http://code.activestate.com/recipes/442473/

Traceback:
---------------------------------------
Exception happened during processing of request from ('XXX.XXX.XX.XX', 57597)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 641, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 694, in finish
    self.wfile.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
TypeError: must be string or read-only buffer, not memoryview
----------------------------------------
Traceback (most recent call last):
  File "ssl_example_py2.py", line 54, in <module>
    test()
  File "ssl_example_py2.py", line 50, in test
    httpd.serve_forever()
  File "/usr/lib/python2.7/SocketServer.py", line 227, in serve_forever
    self._handle_request_noblock()
  File "/usr/lib/python2.7/SocketServer.py", line 287, in _handle_request_noblock
    self.shutdown_request(request)
  File "/usr/lib/python2.7/SocketServer.py", line 459, in shutdown_request
    request.shutdown(socket.SHUT_WR)
TypeError: shutdown() takes exactly 0 arguments (1 given)

[Oliver Marks (oly)]

Not sure where to ask this, this bug is marked as fix back in january but i am still effected by this in recent natty is there anyway to make sure the fix makes it into natty or somewhere to request / monitor its progress into natty

[Jean-Paul Calderone (exarkun)]

This is the upstream bug tracker. It sounds like you're looking for the Ubuntu pyOpenSSL bug tracker is at <https://bugs.launchpad.net/ubuntu/natty/+source/pyopenssl>. I'm not really involved with the Ubuntu packaging work.

[Oliver Marks (oly)]

cheers yeah im not sure how it all works i just noticed the bug and search for pyopenssl and ended up here, reported it now in the link you provided, hopefully some one will be able to get the fix in cheers :)

[Erik L (erik-lat)]

I'm running Fedora 14 x86_64 and still getting this error with pyOpenSSL-0.9-3.fc14.x86_64. Am I missing something here?

[Erik L (erik-lat)]

Nevermind, I was mistaken.

[Boris Petkov (boris-petkov)]

I am seeing similar problem with Kubuntu 11.04, python 2.7 and python-openssl 0.10-1ubuntu 3 (the latest by now).

Here is the trace I see:

----------------------------------------
Exception happened during processing of request from ('192.168.168.186', 44237)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 311, in process_request
    self.shutdown_request(request)
  File "/usr/lib/python2.7/SocketServer.py", line 459, in shutdown_request
    request.shutdown(socket.SHUT_WR)
TypeError: shutdown() takes exactly 0 arguments (1 given)
----------------------------------------
Traceback (most recent call last):
  File "server.py", line 161, in <module>
    server.serve_forever()
  File "/usr/lib/python2.7/SocketServer.py", line 227, in serve_forever
    self._handle_request_noblock()
  File "/usr/lib/python2.7/SocketServer.py", line 287, in _handle_request_noblock
    self.shutdown_request(request)
  File "/usr/lib/python2.7/SocketServer.py", line 459, in shutdown_request
    request.shutdown(socket.SHUT_WR)
TypeError: shutdown() takes exactly 0 arguments (1 given)

And here is the code I execute (see attachment), which is running with no errors on Kubntu 10.10 with python 2.6

[Jean-Paul Calderone (exarkun)]

This is fixed in pyOpenSSL. Report bugs against Ubuntu's pyOpenSSL package at https://bugs.launchpad.net/ubuntu/+source/pyopenssl

[Jean-Sebastien Moreau (j-s-moreau)]

I have this bug using version 0.12 with OpenSSL 0.9.8e. Do you mean it is fixed for 0.13?

[Jean-Sebastien Moreau (j-s-moreau)]

I'm running on python 2.7.

[Jean-Paul Calderone (exarkun)]

I believe the bug was fixed by r140 on January 22, 2011. pyOpenSSL 0.12 was released several months later and includes the fix. If a problem still exists, please provide more details about the misbehavior and how to reproduce it. Thanks.

[sifu (mathieu-roche-site)]

I have the same problem with on Windows Seven:
- Python 2.7
- PyOpenSSL (pyOpenSSL-0.13-py2.7-win32.egg)

I'm trying with SimpleSecureHTTPServer.py from http://code.activestate.com/recipes/442473-simple-http-server-supporting-ssl-secure-communica/

Serving HTTPS on 0.0.0.0 port 9000 ...
mymachine - - [06/Dec/2011 15:37:52] code 404, message File
not found
mymachine - - [06/Dec/2011 15:37:52] "GET /test HTTP/1.1"
04 -
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 54408)
Traceback (most recent call last):
  File "C:\Python27\lib\SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\Python27\lib\SocketServer.py", line 311, in process_request
    self.shutdown_request(request)
  File "C:\Python27\lib\SocketServer.py", line 459, in shutdown_request
    request.shutdown(socket.SHUT_WR)
TypeError: shutdown() takes exactly 0 arguments (1 given)
----------------------------------------
Traceback (most recent call last):
  File "C:\Users\mylogin\Desktop\test.py", line 60, in <module>
    test()
  File "C:\Users\mylogin\Desktop\test.py", line 56, in test
    httpd.serve_forever()
  File "C:\Python27\lib\SocketServer.py", line 227, in serve_forever
    self._handle_request_noblock()
  File "C:\Python27\lib\SocketServer.py", line 287, in _handle_request_noblock
    self.shutdown_request(request)
  File "C:\Python27\lib\SocketServer.py", line 459, in shutdown_request
    request.shutdown(socket.SHUT_WR)
TypeError: shutdown() takes exactly 0 arguments (1 given)

Thanks.

[Jean-Paul Calderone (exarkun)]

This exception is fixed now:

    TypeError: must be string or read-only buffer, not memoryview

However, this exception remains:

    TypeError: shutdown() takes exactly 0 arguments (1 given)

These are two unrelated bugs. I filed https://bugs.launchpad.net/pyopenssl/+bug/900792 for the latter.