instance deletion fails due to neutron client error: "simultaneous read on fileno N detected"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
New
|
Undecided
|
Unassigned |
Bug Description
Doing performance testing with the latest OpenStack code, I regularly have a problem when deleting instances. When attempting to query neutron during instance deallocation, nova-compute apparently makes a greenthread-unsafe use of its neutron client:
{u'message':
"Connection to neutron failed: Second simultaneous read on fileno 25 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and",
u'code': 500,
u'details': u'
File "/opt/stack/
return function(self, context, *args, **kwargs)
File "/opt/stack/
do_
File "/opt/stack/
return f(*args, **kwargs)
File "/opt/stack/
reservatio
File "/opt/stack/
rv = f(*args, **kwargs)
File "/opt/stack/
user_
File "/opt/stack/
self.
File "/opt/stack/
self.
File "/opt/stack/
self.
File "/opt/stack/
self.
File "/opt/stack/
context, instance, requested_
File "/opt/stack/
data = neutronv2.
File "/opt/stack/
ret = self.function(
File "/opt/stack/
**_params)
File "/opt/stack/
for r in self._paginatio
File "/opt/stack/
res = self.get(path, params=params)
File "/opt/stack/
headers=
File "/opt/stack/
headers=
File "/opt/stack/
resp, replybody = self.httpclient
File "/opt/stack/
**kwargs)
File "/opt/stack/
raise exceptions.
', u'created': u'2013-
Note that I had to make a minor hack to my python-
diff --git a/neutronclient
index 733f339..d2ac237 100644
--- a/neutronclient
+++ b/neutronclient
@@ -1224,6 +1224,8 @@ class Client(object):
if i < self.retries:
+ else:
+ raise
raise exceptions.
Without that patch, you just see the error "Maximum attempts reached," which doesn't tell you anything.
Apparently the cause of this error is very straightforward: http:// blog.eventlet. net/2010/ 03/18/safety/. I bet that nova-compute has a single neutron client instance that's being used by all of its greenthreads. I'll check ...