instance deletion fails due to neutron client error: "simultaneous read on fileno N detected"

[Peter Feiner]

Doing performance testing with the latest OpenStack code, I regularly have a problem when deleting instances. When attempting to query neutron during instance deallocation, nova-compute apparently makes a greenthread-unsafe use of its neutron client:

{u'message':
"Connection to neutron failed: Second simultaneous read on fileno 25 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and",
u'code': 500,
u'details': u'
   File "/opt/stack/nova/nova/compute/manager.py", line 259, in decorated_function
     return function(self, context, *args, **kwargs)
   File "/opt/stack/nova/nova/compute/manager.py", line 1984, in terminate_instance
     do_terminate_instance(instance, bdms)
   File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 248, in inner
     return f(*args, **kwargs)
   File "/opt/stack/nova/nova/compute/manager.py", line 1976, in do_terminate_instance
     reservations=reservations)
   File "/opt/stack/nova/nova/hooks.py", line 105, in inner
     rv = f(*args, **kwargs)
   File "/opt/stack/nova/nova/compute/manager.py", line 1949, in _delete_instance
     user_id=user_id)
   File "/opt/stack/nova/nova/compute/manager.py", line 1919, in _delete_instance
     self._shutdown_instance(context, db_inst, bdms)
   File "/opt/stack/nova/nova/compute/manager.py", line 1854, in _shutdown_instance
     self._try_deallocate_network(context, instance, requested_networks)
   File "/opt/stack/nova/nova/compute/manager.py", line 1814, in _try_deallocate_network
     self._set_instance_error_state(context, instance[\'uuid\'])
   File "/opt/stack/nova/nova/compute/manager.py", line 1809, in _try_deallocate_network
     self._deallocate_network(context, instance, requested_networks)
   File "/opt/stack/nova/nova/compute/manager.py", line 1492, in _deallocate_network
     context, instance, requested_networks=requested_networks)
   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 404, in deallocate_for_instance
     data = neutronv2.get_client(context).list_ports(**search_opts)
   File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 111, in with_params
     ret = self.function(instance, *args, **kwargs)
   File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 306, in list_ports
     **_params)
   File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 1253, in list
     for r in self._pagination(collection, path, **params):
   File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 1266, in _pagination
     res = self.get(path, params=params)
   File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 1239, in get
     headers=headers, params=params)
   File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 1221, in retry_request
     headers=headers, params=params)
   File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 1156, in do_request
     resp, replybody = self.httpclient.do_request(action, method, body=body)
   File "/opt/stack/python-neutronclient/neutronclient/client.py", line 190, in do_request
     **kwargs)
   File "/opt/stack/python-neutronclient/neutronclient/client.py", line 152, in _cs_request
     raise exceptions.ConnectionFailed(reason=e)
', u'created': u'2013-11-19T19:36:11Z'}

Note that I had to make a minor hack to my python-neutronclient to get this exception:

diff --git a/neutronclient/v2_0/client.py b/neutronclient/v2_0/client.py
index 733f339..d2ac237 100644
--- a/neutronclient/v2_0/client.py
+++ b/neutronclient/v2_0/client.py
@@ -1224,6 +1224,8 @@ class Client(object):
                 if i < self.retries:
                     _logger.debug(_('Retrying connection to Neutron service'))
                     time.sleep(self.retry_interval)
+ else:
+ raise

         raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))

Without that patch, you just see the error "Maximum attempts reached," which doesn't tell you anything.

[Peter Feiner (pete5)]

Apparently the cause of this error is very straightforward: http://blog.eventlet.net/2010/03/18/safety/. I bet that nova-compute has a single neutron client instance that's being used by all of its greenthreads. I'll check ...

[Peter Feiner (pete5)]

I was wrong. There's a per-greenthread client.

[Peter Feiner (pete5)]

Given how simple nova-compute's allocation and use of the client object is, I'm inclined to think that the bug lies in python-neutronclient. I will do some stress testing on the client code to see if I can isolate the cause.

[Davanum Srinivas (DIMS) (dims-v)]

Please make sure you have the proper neutronclient with this patch

https://github.com/openstack/python-neutronclient/commit/e6a9f79066943b75e2c60b0165533d5886cc9cd8

[Peter Feiner (pete5)]

Davanum, please see bug #1229475. The problem was in nova-compute's use of thread-local storage. The solution to that bug fixes this problem as well.