| 2014-04-16 10:01:06 |
Liusheng |
description |
consider a scenario, we have more than 1 compute node in our environment. in some cases, when booting a instance failed because a libvirt error cause spawning failed,
according the implementation of of _build_instance(), the exceptions will be accepted as "Exception", and then, reschedule the instance or set it to error state. the related code is:
except Exception:
exc_info = sys.exc_info()
# try to re-schedule instance:
# Make sure the async call finishes
if network_info is not None:
network_info.wait(do_raise=False)
rescheduled = self._reschedule_or_error(context, instance,
exc_info, requested_networks, admin_password,
injected_files_orig, is_first_time, request_spec,
filter_properties, bdms, legacy_bdm_in_spec)
if we have specified a port when booting the instance, the port['device_id'] is the uuid of the this instance. a "PortInUse" exception will be raised after rescheduling successfully and trying to build on another node.
To reproduce, we can add "raise Exception" after spawn in _build_instance().
more details see:
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1043, in _build_instance
set_access_ip=set_access_ip)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1426, in _spawn
LOG.exception(_('Instance failed to spawn'), instance=instance)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1423, in _spawn
block_device_info)
File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2083, in spawn
admin_pass=admin_password)
File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2480, in _create_image
net = netutils.get_injected_network_template(network_info)
File "/usr/lib64/python2.6/site-packages/nova/virt/netutils.py", line 74, in get_injected_network_
if not (network_info and template):
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 379, in __len__
return self._sync_wrapper(fn, *args, **kwargs)
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 366, in _sync_wrapper
self.wait()
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 398, in wait
self[:] = self._gt.wait()
File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
return self._exit_event.wait()
File "/usr/lib64/python2.6/site-packages/eventlet/event.py", line 120, in wait
current.throw(*self._exc)
File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
result = function(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1244, in _allocate_network
dhcp_options=dhcp_options)
File "/usr/lib64/python2.6/site-packages/nova/network/neutronv2/api.py", line 243, in allocate_for
raise exception.PortInUse(port_id=port_id)
PortInUse: Port faf3aa64-11f8-4fc7-81bc-084098014f4a is still in use. |
consider a scenario, we have more than 1 compute node in our environment. in some cases, when booting a instance with a port specified, if boot failed because a libvirt error cause spawning failed,
according the implementation of of _build_instance(), the exceptions will be accepted as "Exception", and then, reschedule the instance or set it to error state. the related code is:
except Exception:
exc_info = sys.exc_info()
# try to re-schedule instance:
# Make sure the async call finishes
if network_info is not None:
network_info.wait(do_raise=False)
rescheduled = self._reschedule_or_error(context, instance,
exc_info, requested_networks, admin_password,
injected_files_orig, is_first_time, request_spec,
filter_properties, bdms, legacy_bdm_in_spec)
if we have specified a port when booting the instance, the port['device_id'] is the uuid of the this instance. a "PortInUse" exception will be raised after rescheduling successfully and trying to build on another node.
To reproduce, we can add "raise Exception" after spawn in _build_instance().
more details see:
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1043, in _build_instance
set_access_ip=set_access_ip)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1426, in _spawn
LOG.exception(_('Instance failed to spawn'), instance=instance)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1423, in _spawn
block_device_info)
File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2083, in spawn
admin_pass=admin_password)
File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2480, in _create_image
net = netutils.get_injected_network_template(network_info)
File "/usr/lib64/python2.6/site-packages/nova/virt/netutils.py", line 74, in get_injected_network_
if not (network_info and template):
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 379, in __len__
return self._sync_wrapper(fn, *args, **kwargs)
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 366, in _sync_wrapper
self.wait()
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 398, in wait
self[:] = self._gt.wait()
File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
return self._exit_event.wait()
File "/usr/lib64/python2.6/site-packages/eventlet/event.py", line 120, in wait
current.throw(*self._exc)
File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
result = function(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1244, in _allocate_network
dhcp_options=dhcp_options)
File "/usr/lib64/python2.6/site-packages/nova/network/neutronv2/api.py", line 243, in allocate_for
raise exception.PortInUse(port_id=port_id)
PortInUse: Port faf3aa64-11f8-4fc7-81bc-084098014f4a is still in use. |
|
| 2014-04-16 10:06:05 |
Liusheng |
description |
consider a scenario, we have more than 1 compute node in our environment. in some cases, when booting a instance with a port specified, if boot failed because a libvirt error cause spawning failed,
according the implementation of of _build_instance(), the exceptions will be accepted as "Exception", and then, reschedule the instance or set it to error state. the related code is:
except Exception:
exc_info = sys.exc_info()
# try to re-schedule instance:
# Make sure the async call finishes
if network_info is not None:
network_info.wait(do_raise=False)
rescheduled = self._reschedule_or_error(context, instance,
exc_info, requested_networks, admin_password,
injected_files_orig, is_first_time, request_spec,
filter_properties, bdms, legacy_bdm_in_spec)
if we have specified a port when booting the instance, the port['device_id'] is the uuid of the this instance. a "PortInUse" exception will be raised after rescheduling successfully and trying to build on another node.
To reproduce, we can add "raise Exception" after spawn in _build_instance().
more details see:
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1043, in _build_instance
set_access_ip=set_access_ip)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1426, in _spawn
LOG.exception(_('Instance failed to spawn'), instance=instance)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1423, in _spawn
block_device_info)
File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2083, in spawn
admin_pass=admin_password)
File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2480, in _create_image
net = netutils.get_injected_network_template(network_info)
File "/usr/lib64/python2.6/site-packages/nova/virt/netutils.py", line 74, in get_injected_network_
if not (network_info and template):
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 379, in __len__
return self._sync_wrapper(fn, *args, **kwargs)
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 366, in _sync_wrapper
self.wait()
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 398, in wait
self[:] = self._gt.wait()
File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
return self._exit_event.wait()
File "/usr/lib64/python2.6/site-packages/eventlet/event.py", line 120, in wait
current.throw(*self._exc)
File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
result = function(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1244, in _allocate_network
dhcp_options=dhcp_options)
File "/usr/lib64/python2.6/site-packages/nova/network/neutronv2/api.py", line 243, in allocate_for
raise exception.PortInUse(port_id=port_id)
PortInUse: Port faf3aa64-11f8-4fc7-81bc-084098014f4a is still in use. |
When booting an instance with a port specified, if the instance spawn faild for libvit error, the instance will be rescheduled, and will raise a "PortInUse" exception.
To reproduce, we can add "raise Exception" after spawn in _build_instance() and restart nova-compute.
more details plz see:
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1043, in _build_instance
set_access_ip=set_access_ip)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1426, in _spawn
LOG.exception(_('Instance failed to spawn'), instance=instance)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1423, in _spawn
block_device_info)
File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2083, in spawn
admin_pass=admin_password)
File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2480, in _create_image
net = netutils.get_injected_network_template(network_info)
File "/usr/lib64/python2.6/site-packages/nova/virt/netutils.py", line 74, in get_injected_network_
if not (network_info and template):
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 379, in __len__
return self._sync_wrapper(fn, *args, **kwargs)
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 366, in _sync_wrapper
self.wait()
File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 398, in wait
self[:] = self._gt.wait()
File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
return self._exit_event.wait()
File "/usr/lib64/python2.6/site-packages/eventlet/event.py", line 120, in wait
current.throw(*self._exc)
File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
result = function(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1244, in _allocate_network
dhcp_options=dhcp_options)
File "/usr/lib64/python2.6/site-packages/nova/network/neutronv2/api.py", line 243, in allocate_for
raise exception.PortInUse(port_id=port_id)
PortInUse: Port faf3aa64-11f8-4fc7-81bc-084098014f4a is still in use. |
|
