OpenStack Compute (nova)

Bad error report when trying to connect VM to an overcrouded Network

Bug #1367243 reported by Yair Fried
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Expired
Undecided
Unassigned
neutron
Won't Fix
Undecided
Unassigned

Bug Description

trying to create a Port when the network has no more addresses, Neutron returns this error:
No more IP addresses available on network a4e997dc-ba2e-4444-9394-cfd89f670886.

However, when trying to create a VM in a network that has no more addresses, the VM is created with Error details:
"No valid host was found"

That's because the the compute agent is seeing the PortLimitExceeded error from neutron and reports the the scheduler.
The scheduler mistakes that for Hypervisor limit and tries another compute (which fails for the same reason) and finally reports the error as "No host found", while the error has to be "No more IP addresses available on network a4e997dc-ba2e-4444-9394-cfd89f670886"

Neutron however, doesn't register any errors for this flow.

Recreate:
1. create a subnet with 31 mask bits (which means no ports available for VMs)
2. Boot a VM. errors will be seen in nova-compute, and nova-scheduler

nova compute:
2014-09-09 14:30:11.517 123433 AUDIT nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Starting instance...
2014-09-09 14:30:11.608 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Attempting claim: memory 64 MB, disk 0 GB, VCPUs 1
2014-09-09 14:30:11.608 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total memory: 31952 MB, used: 1152.00 MB
2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] memory limit: 47928.00 MB, free: 46776.00 MB
2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total disk: 442 GB, used: 0.00 GB
2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] disk limit not specified, defaulting to unlimited
2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total CPUs: 24 VCPUs, used: 10.00 VCPUs
2014-09-09 14:30:11.610 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] CPUs limit: 384.00 VCPUs, free: 374.00 VCPUs
2014-09-09 14:30:11.610 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Claim successful
2014-09-09 14:30:12.141 123433 WARNING nova.network.neutronv2.api [-] Neutron error: quota exceeded
2014-09-09 14:30:12.142 123433 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager Traceback (most recent call last):
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager dhcp_options=dhcp_options)
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager LOG.exception(msg, port_id)
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager security_group_ids, available_macs, dhcp_opts)
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager raise exception.PortLimitExceeded()
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager PortLimitExceeded: Maximum number of ports exceeded
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager
2014-09-09 14:30:12.158 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Creating image
2014-09-09 14:30:12.261 123433 ERROR nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Instance failed to spawn
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Traceback (most recent call last):
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] block_device_info)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] admin_pass=admin_password)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] instance, network_info, admin_pass, files, suffix)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] net = netutils.get_injected_network_template(network_info)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] if not (network_info and template):
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] return self._sync_wrapper(fn, *args, **kwargs)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] self.wait()
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] self[:] = self._gt.wait()
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] return self._exit_event.wait()
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] current.throw(*self._exc)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] result = function(*args, **kwargs)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] dhcp_options=dhcp_options)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] LOG.exception(msg, port_id)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] six.reraise(self.type_, self.value, self.tb)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] security_group_ids, available_macs, dhcp_opts)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] raise exception.PortLimitExceeded()
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] PortLimitExceeded: Maximum number of ports exceeded
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]
2014-09-09 14:30:12.326 123433 AUDIT nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Terminating instance
2014-09-09 14:30:12.334 123433 ERROR nova.virt.libvirt.driver [-] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] During wait destroy, instance disappeared.
2014-09-09 14:30:12.425 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Deleting instance files /var/lib/nova/instances/8f50312c-f47e-4793-aff7-a890f20ee2bb
2014-09-09 14:30:12.425 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Deletion of /var/lib/nova/instances/8f50312c-f47e-4793-aff7-a890f20ee2bb complete
2014-09-09 14:30:12.558 123433 INFO nova.network.neutronv2.api [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Unable to reset device ID for port None
2014-09-09 14:30:12.644 123433 ERROR nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Error: Maximum number of ports exceeded
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Traceback (most recent call last):
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1329, in _build_instance
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] set_access_ip=set_access_ip)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] return function(self, context, *args, **kwargs)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1741, in _spawn
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] six.reraise(self.type_, self.value, self.tb)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] block_device_info)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] admin_pass=admin_password)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] instance, network_info, admin_pass, files, suffix)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] net = netutils.get_injected_network_template(network_info)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] if not (network_info and template):
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] return self._sync_wrapper(fn, *args, **kwargs)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] self.wait()
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] self[:] = self._gt.wait()
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] return self._exit_event.wait()
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] current.throw(*self._exc)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] result = function(*args, **kwargs)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] dhcp_options=dhcp_options)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] LOG.exception(msg, port_id)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] six.reraise(self.type_, self.value, self.tb)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] security_group_ids, available_macs, dhcp_opts)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] raise exception.PortLimitExceeded()
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] PortLimitExceeded: Maximum number of ports exceeded
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]
2014-09-09 14:30:33.435 123433 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 30736
2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 13
2014-09-09 14:30:34.137 123433 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma48.scl.lab.tlv.redhat.com:puma48.scl.lab.tlv.redhat.com
2014-09-09 14:30:51.734 123433 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2014-09-09 14:30:52.405 123433 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 30736
2014-09-09 14:30:52.405 123433 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
2014-09-09 14:30:52.406 123433 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 13
2014-09-09 14:30:52.429 123433 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma48.scl.lab.tlv.redhat.com:puma48.scl.lab.tlv.redhat.com

nova-scheduler
2014-09-09 14:30:11.382 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Attempting to build 1 instance(s) uuids: [u'8f50312c-f47e-4793-aff7-a890f20ee2bb']
2014-09-09 14:30:11.387 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Choosing host WeighedHost [host: puma48.scl.lab.tlv.redhat.com, weight: 1.0] for instance 8f50312c-f47e-4793-aff7-a890f20ee2bb
2014-09-09 14:30:12.665 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Attempting to build 1 instance(s) uuids: [u'8f50312c-f47e-4793-aff7-a890f20ee2bb']
2014-09-09 14:30:12.665 123181 ERROR nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Error from last host: puma48.scl.lab.tlv.redhat.com (node puma48.scl.lab.tlv.redhat.com): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1329, in _build_instance\n set_access_ip=set_access_ip)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1741, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn\n block_device_info)\n', u' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn\n admin_pass=admin_password)\n', u' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image\n instance, network_info, admin_pass, files, suffix)\n', u' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data\n net = netutils.get_injected_network_template(network_info)\n', u' File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template\n if not (network_info and template):\n', u' File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__\n return self._sync_wrapper(fn, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper\n self.wait()\n', u' File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait\n self[:] = self._gt.wait()\n', u' File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait\n return self._exit_event.wait()\n', u' File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait\n current.throw(*self._exc)\n', u' File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main\n result = function(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async\n dhcp_options=dhcp_options)\n', u' File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance\n LOG.exception(msg, port_id)\n', u' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance\n security_group_ids, available_macs, dhcp_opts)\n', u' File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port\n raise exception.PortLimitExceeded()\n', u'PortLimitExceeded: Maximum number of ports exceeded\n']
2014-09-09 14:30:12.669 123181 INFO nova.filters [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Filter RetryFilter returned 0 hosts
2014-09-09 14:30:12.669 123181 WARNING nova.scheduler.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Setting instance to ERROR state.

[root@puma48 ~(keystone_admin)]# nova show 8f50312c-f47e-4793-aff7-a890f20ee2bb
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | puma48.scl.lab.tlv.redhat.com |
| OS-EXT-SRV-ATTR:hypervisor_hostname | puma48.scl.lab.tlv.redhat.com |
| OS-EXT-SRV-ATTR:instance_name | instance-000003d7 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-09-09T11:30:11Z |
| fault | {"message": "No valid host was found. ", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py\", line 108, in schedule_run_instance |
| | raise exception.NoValidHost(reason=\"\") |
| | ", "created": "2014-09-09T11:30:12Z"} |
| flavor | m1.nano (42) |
| hostId | 624ae58ea64f1cfb0ea62f3b0eced7799b21a3e55414343a6352ca67 |
| id | 8f50312c-f47e-4793-aff7-a890f20ee2bb |
| image | cirros (49948054-e169-4e1b-a5b4-6441959ef75c) |
| key_name | - |
| metadata | {} |
| name | test_VM |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | a0a2f1afe57d422887b48c204d536df0 |
| updated | 2014-09-09T11:30:12Z |
| user_id | e9bb9ce3fbe344e9b49182a13dcfb9c3 |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

See original description
Tags: network
Yair Fried (yfried)
description: updated
Revision history for this message
Sean Dague (sdague) wrote :

seems like mostly a neutron issue

Changed in nova:
status: New → Confirmed
importance: Undecided → Low
Phani Pawan (ppawan)
Changed in nova:
assignee: nobody → phanipawan (ppawan)
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

That doesn't seem as neutron issue as neutron provides expected and well-understood outcome for the operation.
Marking as Incomplete for neutron, waiting submitter clarification.

Changed in neutron:
status: New → Incomplete
Revision history for this message
Yair Fried (yfried) wrote :

I agree with @enikanorov, This a problem with Nova interpreting the Neutron error.
If you had 100 Compute nodes, scheduler would try to boot the VM in all of them, even though Neutron reports there's no way to boot a VM in that network

Revision history for this message
Alexey I. Froloff (raorn) wrote :

Same thing happens when passing invalid ip_address. Neutron responds with InvalidInput error and Nova reschedules instance to next hypervisor.

Phani Pawan (ppawan)
Changed in nova:
assignee: Phani Pawan (ppawan) → nobody
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote : Cleanup EOL bug report

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON).
  Valid example: CONFIRMED FOR: LIBERTY

Changed in nova:
importance: Low → Undecided
status: Confirmed → Expired
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Bug closed due to lack of activity, please feel free to reopen if needed.

Changed in neutron:
status: Incomplete → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.