AMQP concurrency problem in API server

[Soren Hansen]

It seems that both our use of Redis and Carrot (or py-amqplib) are suffering somewhat under Eventlet. This bug is about Carrot/py-amqplib:

I created a simple script that would start an instance, and keep trying to connect to its ssh port for a while, and once succesful, it would terminate the instance again and start over. I ran this script in four terminals at the same time. Quite soon, I would start getting errors like this from euca-run-instances:

RuntimeError: Second simultaneous read on fileno 17 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_multiple_reader_prevention(False)

Running the api server under strace revealed that that file descriptor in question is the connection to the AMQP server. Since I don't know what I'm doing, I didn't call eventlet.debug.hub_multiple_reader_prevention(FaIse).

Again, if I stop reusing connections (like in the Redis bug) things work fine, apparantly.

[Eric Day (eday)]

Yeah, a couple have these have already been fixed. I changed the consumer code to actually wait rather than busy poll every 100ms, which means each consumer needs to have it's own connection. Since these are short lived we can keep a pool of cached connections to reuse so we don't have the connection setup cost on every use. We can add a pool anywhere we are currently using singleton connections.

[Eric Day (eday)]

This particular issue was resolved before Austin. The rpc Connection class was modified to allow multiple connections to be created when needed (usually for temporary return queues). Marking this as fix released.