OpenStack Compute (nova)

nova host-servers-migrate [hostname] fails

Bug #1648887 reported by Alexander Chuzhoy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Dan Smith
Newton
Fix Released
High
Lee Yarwood

Bug Description

Tried to migrate all vms as part of upgrading controllers going from OSP9 to OSP10.
Successfully migrated all vms from computes of the same version.
Upgraded one compute node.
Then tried to migrate all vms from the other machine to the upgraded one with:
nova host-servers-migrate overcloud-compute-0.localdomain

This resulted in error:
Grepping for errors the /var/log/nova/nova-compute.log file on a target compute:

2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server [req-5bbd4371-dd07-443c-884f-02937064bb8f 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server migration.instance_uuid, exc_info=True)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self._set_instance_obj_error_state(context, instance)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server disk_info, image_meta)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server old_instance_type)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server block_device_info, power_on)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server block_device_info=block_device_info)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server context)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return [device for device in pci_devices if
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [req-10021f30-ac47-495d-83b6-2fe54d4ab647 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] Setting instance vm_state to ERROR
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] Traceback (most recent call last):
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] disk_info, image_meta)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] old_instance_type)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] self.force_reraise()
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] block_device_info, power_on)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] block_device_info=block_device_info)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] context)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] return [device for device in pci_devices if
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] TypeError: 'NoneType' object is not iterable
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server [req-10021f30-ac47-495d-83b6-2fe54d4ab647 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server migration.instance_uuid, exc_info=True)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self._set_instance_obj_error_state(context, instance)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server disk_info, image_meta)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server old_instance_type)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server block_device_info, power_on)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server block_device_info=block_device_info)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server context)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return [device for device in pci_devices if
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [req-7ecfc3ee-2b17-4582-9788-f81a90de21e8 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: e9797290-6333-485f-a1c4-965355b11874] Setting instance vm_state to ERROR
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] Traceback (most recent call last):
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] disk_info, image_meta)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] old_instance_type)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] self.force_reraise()
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] block_device_info, power_on)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] block_device_info=block_device_info)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] context)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] if pci_devices is None:
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server [req-7ecfc3ee-2b17-4582-9788-f81a90de21e8 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server migration.instance_uuid, exc_info=True)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self._set_instance_obj_error_state(context, instance)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server disk_info, image_meta)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server old_instance_type)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server block_device_info, power_on)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server block_device_info=block_device_info)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server context)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server if pci_devices is None:
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [req-81456147-61d3-4d1e-ac49-6becc999f201 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] Setting instance vm_state to ERROR
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] Traceback (most recent call last):
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] disk_info, image_meta)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] old_instance_type)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] self.force_reraise()
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] block_device_info, power_on)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] block_device_info=block_device_info)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] context)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] if pci_devices is None:
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server [req-81456147-61d3-4d1e-ac49-6becc999f201 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server migration.instance_uuid, exc_info=True)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self._set_instance_obj_error_state(context, instance)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server disk_info, image_meta)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server old_instance_type)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server block_device_info, power_on)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server block_device_info=block_device_info)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server context)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server if pci_devices is None:
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/409338

Changed in nova:
assignee: nobody → Dan Smith (danms)
status: New → In Progress
Sylvain Bauza (sylvain-bauza)
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/409338
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=15564eb355657fcf2594d5060cb36f25e28bc0c9
Submitter: Jenkins
Branch: master

commit 15564eb355657fcf2594d5060cb36f25e28bc0c9
Author: Dan Smith <email address hidden>
Date: Fri Dec 9 13:23:44 2016 -0800

    Fix crashing during guest config with pci_devices=None

    The Instance.pci_devices field is nullable, but the get_instance_pci_devs()
    function does not account for that. If it is passed an instance (from an
    older deployment) with no pci_devices, it will crash trying to iterate
    None as a list.

    Change-Id: I3d535e01ac31db7804347c3938c0d88a28ba67f5
    Closes-Bug: #1648887

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/409821

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/409821
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9642cd339f1f520569dfea49495fec79ad7a8195
Submitter: Jenkins
Branch: stable/newton

commit 9642cd339f1f520569dfea49495fec79ad7a8195
Author: Dan Smith <email address hidden>
Date: Fri Dec 9 13:23:44 2016 -0800

    Fix crashing during guest config with pci_devices=None

    The Instance.pci_devices field is nullable, but the get_instance_pci_devs()
    function does not account for that. If it is passed an instance (from an
    older deployment) with no pci_devices, it will crash trying to iterate
    None as a list.

    Change-Id: I3d535e01ac31db7804347c3938c0d88a28ba67f5
    Closes-Bug: #1648887
    (cherry picked from commit 15564eb355657fcf2594d5060cb36f25e28bc0c9)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.0.0b2

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.3

This issue was fixed in the openstack/nova 14.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.