I have seen this happen a couple of times, recently here http://logs.openstack.org/31001/2/check/gate-tempest-devstack-vm-quantum/26260/ and here http://logs.openstack.org/31015/2/check/gate-tempest-devstack-vm-quantum/26286/
Tempest reports several of these exceptions for various tests:
BuildErrorException: Server %(server_id)s failed to build and is in ERROR status
Upon inspecting the compute log, there are multiple KeyError exceptions for 'floatingips' or 'ports' from when nova tries to setup its networking. The source of the errors is python-quantumclient.
Example traceback:
2013-05-30 12:06:17.608 ERROR nova.compute.manager [req-047d428b-e790-4174-bb42-f760371b5c50 demo demo] [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] Instance failed network setup
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] Traceback (most recent call last):
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] File "/opt/stack/new/nova/nova/compute/manager.py", line 1118, in _allocate_network
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] security_groups=security_groups)
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] File "/opt/stack/new/nova/nova/network/api.py", line 48, in wrapper
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] res = f(self, context, *args, **kwargs)
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 285, in allocate_for_instance
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] nw_info = self._get_instance_nw_info(context, instance, networks=nets)
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 377, in _get_instance_nw_info
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] nw_info = self._build_network_info_model(context, instance, networks)
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 854, in _build_network_info_model
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] network_IPs = self._nw_info_get_ips(client, port)
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 783, in _nw_info_get_ips
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] client, fixed_ip['ip_address'], port['id'])
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 721, in _get_floating_ips_by_fixed_and_port
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] port_id=port)
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] File "/opt/stack/new/python-quantumclient/quantumclient/v2_0/client.py", line 108, in with_params
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] ret = self.function(instance, *args, **kwargs)
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] File "/opt/stack/new/python-quantumclient/quantumclient/v2_0/client.py", line 441, in list_floatingips
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] **_params)
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] File "/opt/stack/new/python-quantumclient/quantumclient/v2_0/client.py", line 1022, in list
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] res.extend(r[collection])
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] KeyError: 'floatingips'
2013-05-30 12:06:17.608 23143 TRACE nova.compute.manager [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7]
2013-05-30 12:06:17.609 DEBUG nova.compute.claims [req-047d428b-e790-4174-bb42-f760371b5c50 demo demo] [instance: d9c89c2d-002d-40c0-bc50-aefeb49504f7] Aborting claim: [Claim: 64 MB memory, 0 GB disk, 1 VCPUS] abort /opt/stack/new/nova/nova/compute/claims.py:91
This is very weird. Here is the quantum api call that is being made: http:// logs.openstack. org/31015/ 2/check/ gate-tempest- devstack- vm-quantum/ 26286/logs/ screen- n-cpu.txt. gz (above the exception):
REQ: curl -i http:// 127.0.0. 1:9696/ /v2.0/floatingi ps.json? fixed_ip_ address= 10.1.0. 11&port_ id=85778a9e- d706-40b9- bfd8-d07d865212 2e -X GET -H "X-Auth-Token: <snip>=" -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python- quantumclient"
But the response is:
2013-05-30 13:16:12.702 21420 DEBUG quantumclient. client [-] RESP:{'date': 'Thu, 30 May 2013 13:16:12 GMT', 'status': '200', 'content-length': '16', 'content-type': 'application/json; charset=UTF-8', 'content-location': 'http:// 127.0.0. 1:9696/ /v2.0/floatingi ps.json? fixed_ip_ address= 10.1.0. 11&port_ id=85778a9e- d706-40b9- bfd8-d07d865212 2e'} {"networks": []}
It should be {"floatingips": []} not networks.
I'm not able to reproduce this on my box passing in invalid data though :/