Redis concurrency problems in API server
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Invalid
|
Low
|
Unassigned |
Bug Description
It seems that both our use of Redis and Carrot (or py-amqplib) are suffering somewhat under Eventlet. This bug is about Redis:
Trying something like this:
$ euca-describe-
Causes at least two (sometimes all) of these calls to fail with an HTTP 403 error from the API server. My analysis strongly suggested this to be due to multiple calls to Redis going through the same socket at the same time, making all of them fail.
Replacing the Redis singleton with a class that just returns a fresh Redis connection each time fixes it, but may not be acceptable (When doing 200 concurrent requests I got a /lot/ of Redis connections at the same time).
Changed in nova: | |
importance: | Undecided → High |
We are running into this occasionally, but I suspect the AuthManager singleton, since we aren't using redis anymore. I'll use your repro to see if i can stop the ldap version of the problem. Is someone working on an implementation of an AuthDriver that doesn't use redis?
On Sep 28, 2010, at 6:19 AM, Soren Hansen wrote:
> Public bug reported: instances & euca-describe- instances & euca-describe- instances & euca-describe- instances & /bugs.launchpad .net/bugs/ 649807 instances & euca-describe- instances & euca-describe- instances & euca-describe- instances &
>
> It seems that both our use of Redis and Carrot (or py-amqplib) are
> suffering somewhat under Eventlet. This bug is about Redis:
>
> Trying something like this:
> $ euca-describe-
>
> Causes at least two (sometimes all) of these calls to fail with an HTTP
> 403 error from the API server. My analysis strongly suggested this to be
> due to multiple calls to Redis going through the same socket at the same
> time, making all of them fail.
>
> Replacing the Redis singleton with a class that just returns a fresh
> Redis connection each time fixes it, but may not be acceptable (When
> doing 200 concurrent requests I got a /lot/ of Redis connections at the
> same time).
>
> ** Affects: nova
> Importance: Undecided
> Status: New
>
> --
> Redis concurrency problems in API server
> https:/
> You received this bug notification because you are a member of Nova
> Bugs, which is subscribed to OpenStack Compute (nova).
>
> Status in OpenStack Compute (Nova): New
>
> Bug description:
> It seems that both our use of Redis and Carrot (or py-amqplib) are suffering somewhat under Eventlet. This bug is about Redis:
>
> Trying something like this:
> $ euca-describe-
>
> Causes at least two (sometimes all) of these calls to fail with an HTTP 403 error from the API server. My analysis strongly suggested this to be due to multiple calls to Redis going through the same socket at the same time, making all of them fail.
>
> Replacing the Redis singleton with a class that just returns a fresh Redis connection each time fixes it, but may not be acceptable (When doing 200 concurrent requests I got a /lot/ of Redis connections at the same time).
>
>