These are the tracebacks we're seeing when this bug is invoked.
2012-03-26 19:21:28 ERROR nova.compute.manager [-] [instance: 2967f7b5-ae4a-4511-a8b0-10f330b7d5df] Instance failed to spawn (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 592, in _spawn (nova.compute.manager): TRACE: self._legacy_nw_info(network_info), block_device_info) (nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped (nova.compute.manager): TRACE: return f(*args, **kw) (nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 926, in spawn (nova.compute.manager): TRACE: self.firewall_driver.prepare_instance_filter(instance, network_info) (nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 136, in prepare_instance_filter (nova.compute.manager): TRACE: self.add_filters_for_instance(instance) (nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 178, in add_filters_for_instance (nova.compute.manager): TRACE: ipv4_rules, ipv6_rules = self.instance_rules(instance, network_info) (nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 334, in instance_rules (nova.compute.manager): TRACE: ips.extend(net[1]['ips']) (nova.compute.manager): TRACE: KeyError: 1 (nova.compute.manager): TRACE:
Then again when trying to deallocate the network for the instance.
2012-03-26 19:21:28 DEBUG nova.compute.manager [-] [instance: 2967f7b5-ae4a-4511-a8b0-10f330b7d5df] Deallocating network for instance from (pid=32203) _deallocate_network /usr/lib/python2.7/dist-packages/nova/compute/manager.py:616 2012-03-26 19:21:28 DEBUG nova.rpc.amqp [-] Making asynchronous cast on network... from (pid=32203) cast /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:346 2012-03-26 19:21:28 ERROR nova.rpc.amqp [-] Exception during message handling (nova.rpc.amqp): TRACE: Traceback (most recent call last): (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in _process_data (nova.rpc.amqp): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped (nova.rpc.amqp): TRACE: return f(*args, **kw) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function (nova.rpc.amqp): TRACE: sys.exc_info()) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ (nova.rpc.amqp): TRACE: self.gen.next() (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 171, in decorated_function (nova.rpc.amqp): TRACE: return function(self, context, instance_uuid, *args, **kwargs) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 648, in run_instance (nova.rpc.amqp): TRACE: self._run_instance(context, instance_uuid, **kwargs) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 451, in _run_instance (nova.rpc.amqp): TRACE: self._set_instance_error_state(context, instance_uuid) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ (nova.rpc.amqp): TRACE: self.gen.next() (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 432, in _run_instance (nova.rpc.amqp): TRACE: self._deallocate_network(context, instance) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ (nova.rpc.amqp): TRACE: self.gen.next() (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 429, in _run_instance (nova.rpc.amqp): TRACE: injected_files, admin_password) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 592, in _spawn (nova.rpc.amqp): TRACE: self._legacy_nw_info(network_info), block_device_info) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped (nova.rpc.amqp): TRACE: return f(*args, **kw) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 926, in spawn (nova.rpc.amqp): TRACE: self.firewall_driver.prepare_instance_filter(instance, network_info) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 136, in prepare_instance_filter (nova.rpc.amqp): TRACE: self.add_filters_for_instance(instance) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 178, in add_filters_for_instance (nova.rpc.amqp): TRACE: ipv4_rules, ipv6_rules = self.instance_rules(instance, network_info) (nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 334, in instance_rules (nova.rpc.amqp): TRACE: ips.extend(net[1]['ips']) (nova.rpc.amqp): TRACE: KeyError: 1 (nova.rpc.amqp): TRACE:
These are the tracebacks we're seeing when this bug is invoked.
2012-03-26 19:21:28 ERROR nova.compute. manager [-] [instance: 2967f7b5- ae4a-4511- a8b0-10f330b7d5 df] Instance failed to spawn manager) : TRACE: Traceback (most recent call last): manager) : TRACE: File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 592, in _spawn manager) : TRACE: self._legacy_ nw_info( network_ info), block_device_info) manager) : TRACE: File "/usr/lib/ python2. 7/dist- packages/ nova/exception. py", line 114, in wrapped manager) : TRACE: return f(*args, **kw) manager) : TRACE: File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ connection. py", line 926, in spawn manager) : TRACE: self.firewall_ driver. prepare_ instance_ filter( instance, network_info) manager) : TRACE: File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ firewall. py", line 136, in prepare_ instance_ filter manager) : TRACE: self.add_ filters_ for_instance( instance) manager) : TRACE: File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ firewall. py", line 178, in add_filters_ for_instance manager) : TRACE: ipv4_rules, ipv6_rules = self.instance_ rules(instance, network_info) manager) : TRACE: File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ firewall. py", line 334, in instance_rules manager) : TRACE: ips.extend( net[1][ 'ips']) manager) : TRACE: KeyError: 1 manager) : TRACE:
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
Then again when trying to deallocate the network for the instance.
2012-03-26 19:21:28 DEBUG nova.compute. manager [-] [instance: 2967f7b5- ae4a-4511- a8b0-10f330b7d5 df] Deallocating network for instance from (pid=32203) _deallocate_network /usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py:616 python2. 7/dist- packages/ nova/rpc/ amqp.py: 346 python2. 7/dist- packages/ nova/rpc/ amqp.py" , line 252, in _process_data context= ctxt, **node_args) python2. 7/dist- packages/ nova/exception. py", line 114, in wrapped python2. 7/dist- packages/ nova/compute/ manager. py", line 177, in decorated_function python2. 7/contextlib. py", line 24, in __exit__ python2. 7/dist- packages/ nova/compute/ manager. py", line 171, in decorated_function python2. 7/dist- packages/ nova/compute/ manager. py", line 648, in run_instance instance( context, instance_uuid, **kwargs) python2. 7/dist- packages/ nova/compute/ manager. py", line 451, in _run_instance instance_ error_state( context, instance_uuid) python2. 7/contextlib. py", line 24, in __exit__ python2. 7/dist- packages/ nova/compute/ manager. py", line 432, in _run_instance e_network( context, instance) python2. 7/contextlib. py", line 24, in __exit__ python2. 7/dist- packages/ nova/compute/ manager. py", line 429, in _run_instance python2. 7/dist- packages/ nova/compute/ manager. py", line 592, in _spawn nw_info( network_ info), block_device_info) python2. 7/dist- packages/ nova/exception. py", line 114, in wrapped python2. 7/dist- packages/ nova/virt/ libvirt/ connection. py", line 926, in spawn driver. prepare_ instance_ filter( instance, network_info) python2. 7/dist- packages/ nova/virt/ firewall. py", line 136, in prepare_ instance_ filter filters_ for_instance( instance) python2. 7/dist- packages/ nova/virt/ firewall. py", line 178, in add_filters_ for_instance rules(instance, network_info) python2. 7/dist- packages/ nova/virt/ firewall. py", line 334, in instance_rules net[1][ 'ips'])
2012-03-26 19:21:28 DEBUG nova.rpc.amqp [-] Making asynchronous cast on network... from (pid=32203) cast /usr/lib/
2012-03-26 19:21:28 ERROR nova.rpc.amqp [-] Exception during message handling
(nova.rpc.amqp): TRACE: Traceback (most recent call last):
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: rval = node_func(
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: return f(*args, **kw)
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: sys.exc_info())
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: self.gen.next()
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: return function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: self._run_
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: self._set_
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: self.gen.next()
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: self._deallocat
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: self.gen.next()
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: injected_files, admin_password)
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: self._legacy_
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: return f(*args, **kw)
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: self.firewall_
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: self.add_
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: ipv4_rules, ipv6_rules = self.instance_
(nova.rpc.amqp): TRACE: File "/usr/lib/
(nova.rpc.amqp): TRACE: ips.extend(
(nova.rpc.amqp): TRACE: KeyError: 1
(nova.rpc.amqp): TRACE: