memcache client race

[Peter Feiner]

Nova uses thread-unsafe memcache client objects in multiple threads. For instance, nova-api's metadata WSGI server uses the same nova.api.metadata.handler.MetadataRequestHandler._cache object for every request. A memcache client object is thread unsafe because it has a single open socket connection to memcached. Thus the multiple threads will read from & write to the same socket fd.

Keystoneclient has the same bug. See https://bugs.launchpad.net/python-keystoneclient/+bug/1289074 for a patch to fix the problem.

[Dan Smith (danms)]

Have you seen this actually happen in nova or are you assuming the same problem exists? Do you have a traceback equivalent to the keystone one for nova?

[Peter Feiner (pete5)]

Hi Dan,

I can't recall if I actually saw it in nova. I don't have a stack trace.

Peter

On Wed, Mar 19, 2014 at 12:56 PM, Dan Smith <email address hidden> wrote:
> Have you seen this actually happen in nova or are you assuming the same
> problem exists? Do you have a traceback equivalent to the keystone one
> for nova?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1291637
>
> Title:
> memcache client race
>
> Status in OpenStack Identity (Keystone):
> New
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Nova uses thread-unsafe memcache client objects in multiple threads.
> For instance, nova-api's metadata WSGI server uses the same
> nova.api.metadata.handler.MetadataRequestHandler._cache object for
> every request. A memcache client object is thread unsafe because it
> has a single open socket connection to memcached. Thus the multiple
> threads will read from & write to the same socket fd.
>
> Keystoneclient has the same bug. See https://bugs.launchpad.net
> /python-keystoneclient/+bug/1289074 for a patch to fix the problem.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/keystone/+bug/1291637/+subscriptions

[Tracy Jones (tjones-i)]

well then i think we'll remove it from rc1 as we need more info here.

[Thierry Carrez (ttx)]

Is there anything more to fix in keystone than was just fixed ?

[Andres Lagar-Cavilla (andreslc-x)]

You have a cross-linked bug (with stack traces!) which has been deemed of critical importance and linked to a CVE security vulnerability.

You are reusing *identical* code.

You choose to ignore the risk because there is no stack trace.

(mindblown)

[Dolph Mathews (dolph)]

Admittedly I didn't look too deeply into this on nova's side, but it appears that nova would not be affected due to nova's implicit monkey patching of thread:

  https://github.com/openstack/nova/blob/0861fc2d22ae40b82dfe6fc0469db84339464923/nova/cmd/__init__.py

Similar to nova, keystone only avoids monkey patching thread for debugging purposes.

If there's a way to reproduce an issue here, I'm not aware of it. Please enlighten us!

[Launchpad Janitor (janitor)]

[Expired for Keystone because there has been no activity for 60 days.]