Comment 3 for bug 1934937

Revision history for this message
Herve Beraud (herveberaud) wrote :

Sean said setting `[oslo_messaging_rabbit]/heartbeat_in_pthread=false` apprently fixed the issue.

However, for now this parameter is deprecated, so we should reconsider the current depreaction of this one and the future removing of this option, especially if it help to solve the problem.

Indeed, native threads and green threads are mixed there.

This option can help us to restore the green thread and avoid us to mixing different types of threads, however, the problem is that the heartbeat won't work with green thread.

Indeed, with a heartbeat in a green thread, with mod_wsgi the python interpreter in the worker process does not run except when actively processing a request so:
(1) request comes in to httpd,
(2) python interpreter is woken up and handed the request,
(3) python interpreter processes request
and returns result back to httpd,
(4) httpd parks the interpreter until the next request comes in now.

What happens when there is a long delay between (4) and then next request at (1)?

The interpreter doesn't run, so the heartbeat thread doesn't run either, and thus the client connection doesn't heartbeat and then you get rabbitmq closing the connection on the other end meanwhile, the next time a request *does* come in, python wakes up and tries to use the old connection but it gets ECONNRESET or whatever from the os at that point in time so it logs an error and then reconnects.

oslo.messaging suffer from the this architecture (mod_wsgi, gevent), I'm not sure that we could solve this problem on the oslo.messaging side. Either we will suffer from the deconnections or from the mixed types of threads and their side effects (this described bug).

Do you have any idea?