OpenStack Compute (Nova)

AMQPConnectionException on launch VM after restart controller

Reported by Joe Gordon on 2012-01-04
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Unassigned

Bug Description

Environment: nova stable/diablo in two machine cluster, one controller and one compute node.

Problem: After rebooting the controller and trying to launch a new VM (a few minutes after the controller is back up), the launched VM goes into an error state. If retry launching VM again it works.

Stack Trace from nova-compute.log file:

2012-01-03 17:14:18,624 AUDIT nova.compute.manager [6b505190-e161-466e-b6ad-4b61358fda20 nova project_503] instance 32: starting...
2012-01-03 17:14:18,680 DEBUG nova.rpc [6b505190-e161-466e-b6ad-4b61358fda20 nova project_503] Making asynchronous call on network ... from (pid=21128) multicall /usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:730
2012-01-03 17:14:18,680 DEBUG nova.rpc [6b505190-e161-466e-b6ad-4b61358fda20 nova project_503] MSG_ID is 1b804a7965464cbca5ab63e42760a0f6 from (pid=21128) multicall /usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:733
2012-01-03 17:14:18,681 ERROR nova.compute.manager [6b505190-e161-466e-b6ad-4b61358fda20 nova project_503] Instance '32' failed network setup.
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 411, in _logging_error
(nova.compute.manager): TRACE: yield
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 442, in _run_instance
(nova.compute.manager): TRACE: network_info = _make_network_info()
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 378, in _make_network_info(nova.compute.manager): TRACE: requested_networks=requested_networks)(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/network/api.py", line 162, in allocate_for_instance
(nova.compute.manager): TRACE: 'args': args})(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 45, in call
(nova.compute.manager): TRACE: return get_impl().call(context, topic, msg)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 746, in call
(nova.compute.manager): TRACE: rv = multicall(context, topic, msg)(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 738, in multicall(nova.compute.manager): TRACE: conn.declare_direct_consumer(msg_id, wait_msg)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 449, in declare_direct_consumer
(nova.compute.manager): TRACE: self.declare_consumer(DirectConsumer, topic, callback)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 394, in declare_consumer
(nova.compute.manager): TRACE: self.consumer_num.next())
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 140, in __init__
(nova.compute.manager): TRACE: **options)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 61, in __init__
(nova.compute.manager): TRACE: self.reconnect(channel)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 68, in reconnect
(nova.compute.manager): TRACE: self.queue.declare()
(nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 359, in declare(nova.compute.manager): TRACE: return (self.name and self.exchange.declare(nowait),
(nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 151, in declare
(nova.compute.manager): TRACE: nowait=nowait)
(nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/syn.py", line 14, in blocking
(nova.compute.manager): TRACE: return __sync_current(fun, *args, **kwargs)
(nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/syn.py", line 40, in __eblocking__
(nova.compute.manager): TRACE: return spawn(fun, *args, **kwargs).wait()
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
(nova.compute.manager): TRACE: return self._exit_event.wait()
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait(nova.compute.manager): TRACE: return hubs.get_hub().switch()
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova.compute.manager): TRACE: return self.greenlet.switch()
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
(nova.compute.manager): TRACE: result = function(*args, **kwargs)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/channel.py", line 843, in exchange_declare
(nova.compute.manager): TRACE: (40, 11), # Channel.exchange_declare_ok
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/abstract_channel.py", line 89, in wait
(nova.compute.manager): TRACE: self.channel_id, allowed_methods)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/connection.py", line 218, in _wait_method
(nova.compute.manager): TRACE: self.wait()
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/abstract_channel.py", line 105, in wait
(nova.compute.manager): TRACE: return amqp_method(self, args)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/connection.py", line 367, in _close
(nova.compute.manager): TRACE: raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id))
(nova.compute.manager): TRACE: AMQPConnectionException: (320, u"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'", (0, 0), '')
(nova.compute.manager): TRACE:

Vish Ishaya (vishvananda) wrote :

I tried to reproduce this on essex master by rebooting rabbit and did not see the same behavior. I will try with stable/diablo

Vish Ishaya (vishvananda) wrote :

Ok i can repro the error on stable/diablo so apparently one of the recent changes to the rpc code has fixed this. I don't know if it is worth the effort to
a) find the commit that fixes it
and
b) backport the relevant changes. I think we can mark it fixed in trunk though

Changed in nova:
status: New → Fix Committed
Joe Gordon (jogo) wrote :

So this will just stay as known bug in diablo?

Thierry Carrez (ttx) on 2012-02-29
Changed in nova:
milestone: none → essex-4
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers