IOError: ' sys.stdout access restricted by mod_wsgi'

[Michael Nelson]

We had over 30k oopses last night on staging rnr, which seems to be caused by twisted accessing a stdout attribute within our wsgi process. Here's an example oops:

https://pastebin.canonical.com/80384/

and a description of the problem can be found at:

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Apache_Error_Log_Files

We should be able to reproduce this locally to find the cause.

[Michael Nelson (michael.nelson)]

AFAICT, the cause is a change between twisted releases 11.0.0 and 11.1.0.

Our staging server (waxapple) had 11.1.0 installed yesterday [1], although I can't be certain from which version it was upgraded from the info that I have.

The line causing the IOError is accessing stdout in the 11.1.0 twisted release [2], which is not accessed at all in the twisted 11.0.0 release [3].

Our production service is still running with python-twisted-core 10.0.0 [4].

As this will effect any wsgi service using txstatsd, I'll mark it as affected also, although we can probably just use the WSGIRestrictStdout directive to avoid the exception.

[1] https://pastebin.canonical.com/80387/
[2] http://twistedmatrix.com/trac/browser/tags/releases/twisted-11.1.0/twisted/python/log.py#L658
[3] http://twistedmatrix.com/trac/browser/tags/releases/twisted-11.0.0/twisted/python/log.py#L621
[4] https://pastebin.canonical.com/80391/

[Michael Nelson (michael.nelson)]

We had one lonely oops yesterday on staging reviews - thanks Sidnei for the fix, and for the quick rollout achuni.