openstack hangs on message passing

[Magosányi Árpád]

I can reproduce it both with quantum ('quantum net-create --tenant_id 31fd997c87c04cae9cf15e4f2f305def demo-net --provider:network_type gre --provider:segmentation_id 1') and nova ('nova boot --image 617ce1fd-884d-49cf-94c6-083dfad6ac95 test').

When I run the command, it hangs forever. In one instance I left it for hours.

The underlying problem lies somewhere around quantum/openstack/common/rpc/amqp.py in notify(),
maybe in the underlying kombu or amqplib.
The teardown method of the ConnectionContext waits for some answer on the connection which never arrives.

A stack trace to right before some read that blocks (off with a couple of lines because debug info inserted):
  File "/usr/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl
    func(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 567, in process_request
    proto = self.protocol(socket, address, self)

  File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()

  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()

  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 278, in handle_one_request
    self.handle_one_response()

  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 382, in handle_one_response
    result = self.application(self.environ, start_response)

  File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)

  File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 278, in __call__
    return self.app(env, start_response)

  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)

  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)

  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)

  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)

  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)

  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)

  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)

  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/quantum/api/v2/resource.py", line 97, in resource
    result = method(request=request, **args)

  File "/usr/lib/python2.7/dist-packages/quantum/api/v2/base.py", line 255, in create
    body)

  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/notifier/api.py", line 139, in notify
    driver.notify(context, msg)

  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/notifier/rabbit_notifier.py", line 43, in notify
    rpc.notify(context, topic, message)

  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/__init__.py", line 190, in notify
    return _get_impl().notify(cfg.CONF, context, topic, msg)

  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 764, in notify
    rpc_amqp.get_connection_pool(conf, Connection))

  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 424, in notify
    LOG.debug("notify sent")

  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 132, in __exit__
    self._done()

  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 117, in _done
    self.connection.reset()

  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 553, in reset
    self.channel.close()

  File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 229, in close
    super(Channel, self).close()

  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 203, in close
    (20, 41), # Channel.close_ok

  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
    self.channel_id, allowed_methods)

  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 206, in _wait_method
    AMQP_LOGGER.debug("before read. stack: %s"%(traceback.format_stack(),))

[mark sanchez (mssanch2002)]

did you come up with a solution?

[James Page (james-page)]

Magosányi

Apologies for the lack for response on this bug; I note that this may be from a much older OpenStack release (quantum -> neutron in newer released).

I'm assuming that this is probably resolved (we use neutron all of the time and I've never seen this problem), but please set back to NEW if you think this is still an issue in OpenStack >= icehouse.