2019-02-13 13:41:33.890 56155 INFO nova.compute.manager [req-13e8d467-b162-42f4-8c0a-c0fae9e32df7 a3bee416cf67420995855d602d2bccd3 a564613210ee43708b8a7fc6274ebd63 - default default] [BRUCE](M-4): post_live_migration_at_destination: migrate_instance_finish(context, instance(Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=False,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2019-02-12T11:40:46Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=,disable_terminate=False,display_description='osa_compute10_1',display_name='osa_compute10_1',ec2_ids=,ephemeral_gb=0,ephemeral_key_uuid=None,fault=,flavor=Flavor(7),host='cc-compute10-kna1',hostname='osa-compute10-1',id=282634,image_ref='f14f070e-8634-485b-b822-4b12a5bf87a0',info_cache=InstanceInfoCache,instance_type_id=7,kernel_id='',key_data='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJvlhaBlEg4cE2rzqr8XU5O/MWxgR2ERPeJEgMZJ0U8QOqKU1WvQUhdvY1aK+ZcIeiqX3HYcUwD3M7FgGCYD1eiKO7wuaBWC0fOxH3l7dx8WMOXW3kYswFgYen565arPu75ETaI4+tz3GFh9vnA+d6rkiCTUL1fvNs7Px37tN7J74Pxox17AuBGnyXtht2Vl0RGbbuHn1vbQ4Kk+Xw3AOgOonltOPW63IuprsiK1brAvwKn8XgLn5eT8/ovpLre2FQSiuzArHA15n+ppSY0du+MlZUQ2y63aKfZmdR765ORlXOiUToGfDxj0oowiTFvJO6VX0rHlkXU4CNlS7h1CsN /usr/local/lib/opensc-pkcs11.so',key_name='rcolvin_yubi',keypairs=,launch_index=0,launched_at=2019-02-12T11:40:53Z,launched_on='cc-compute11-kna1',locked=False,locked_by=None,memory_mb=1024,metadata={},migration_context=,new_flavor=None,node='cc-compute10-kna1',numa_topology=,old_flavor=None,os_type=None,pci_devices=,pci_requests=,power_state=1,progress=80,project_id='a4cac02597bc4013b69a9cd9373f181c',ramdisk_id='',reservation_id='r-xqc9c0u0',root_device_name='/dev/vda',root_gb=20,security_groups=SecurityGroupList,services=,shutdown_terminate=False,system_metadata={boot_roles='_member_,swiftoperator',image_base_image_ref='f14f070e-8634-485b-b822-4b12a5bf87a0',image_container_format='bare',image_disk_format='qcow2',image_min_disk='20',image_min_ram='0',owner_project_name='CN Eng',owner_user_name='CCP_User_35012_b8'},tags=,task_state='migrating',terminated_at=None,trusted_certs=,updated_at=2019-02-13T13:41:22Z,user_data='I2Nsb3VkLWNvbmZpZwpzeXN0ZW1faW5mbzoKIGRlZmF1bHRfdXNlcjoKICBuYW1lOiByY29sdmluCmNocGFzc3dkOiB7IGV4cGlyZTogRmFsc2UgfQpzc2hfcHdhdXRoOiBUcnVlCnBhY2thZ2VfdXBncmFkZTogdHJ1ZQphcHRfbWlycm9yOiBodHRwOi8vbWlycm9yLmNpdHluZXR3b3JrLnNlL3VidW50dS8K',user_id='d42829e49904441eb1834805fa1eb372',uuid=8fe8b490-13aa-4fc7-87a0-39706200b904,vcpu_model=,vcpus=1,vm_mode=None,vm_state='active')), migration{cc-compute10-kna1, cc-compute29-kna1}) 2019-02-13 13:41:33.953 56155 INFO nova.network.neutronv2.api [req-13e8d467-b162-42f4-8c0a-c0fae9e32df7 a3bee416cf67420995855d602d2bccd3 a564613210ee43708b8a7fc6274ebd63 - default default] [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] Updating port e314d12d-aff2-4e60-ae16-3eda2b6156b5 with attributes {'binding:profile': {}, 'binding:host_id': 'cc-compute29-kna1'} 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [req-13e8d467-b162-42f4-8c0a-c0fae9e32df7 a3bee416cf67420995855d602d2bccd3 a564613210ee43708b8a7fc6274ebd63 - default default] [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] Unable to update binding details for port e314d12d-aff2-4e60-ae16-3eda2b6156b5: InternalServerError: Request Failed: internal server error while processing your request. Neutron server returns request_ids: ['req-94584484-32b4-40c2-8740-9ba56b92a7fc'] 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] Traceback (most recent call last): 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 3076, in _update_port_binding_for_instance 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] neutron.update_port(port_id, {'port': updates}) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] ret = obj(*args, **kwargs) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 799, in update_port 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] revision_number=revision_number) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] ret = obj(*args, **kwargs) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 2375, in _update_resource 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] return self.put(path, **kwargs) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] ret = obj(*args, **kwargs) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 363, in put 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] headers=headers, params=params) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] ret = obj(*args, **kwargs) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] headers=headers, params=params) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] ret = obj(*args, **kwargs) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] self._handle_fault_response(status_code, replybody, resp) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] ret = obj(*args, **kwargs) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] exception_handler_v20(status_code, error_body) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] request_ids=request_ids) 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] InternalServerError: Request Failed: internal server error while processing your request. 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] Neutron server returns request_ids: ['req-94584484-32b4-40c2-8740-9ba56b92a7fc'] 2019-02-13 13:42:21.750 56155 ERROR nova.network.neutronv2.api [instance: 8fe8b490-13aa-4fc7-87a0-39706200b904] 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server [req-13e8d467-b162-42f4-8c0a-c0fae9e32df7 a3bee416cf67420995855d602d2bccd3 a564613210ee43708b8a7fc6274ebd63 - default default] Exception during message handling: InternalServerError: Request Failed: internal server error while processing your request. Neutron server returns request_ids: ['req-94584484-32b4-40c2-8740-9ba56b92a7fc'] 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in decorated_function 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/compute/manager.py", line 6783, in post_live_migration_at_destination 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server migration) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 2598, in migrate_instance_finish 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server migration=migration) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 3081, in _update_port_binding_for_instance 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server port_id, instance=instance) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 3076, in _update_port_binding_for_instance 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server neutron.update_port(port_id, {'port': updates}) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server ret = obj(*args, **kwargs) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 799, in update_port 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server revision_number=revision_number) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server ret = obj(*args, **kwargs) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 2375, in _update_resource 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server return self.put(path, **kwargs) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server ret = obj(*args, **kwargs) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 363, in put 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server headers=headers, params=params) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server ret = obj(*args, **kwargs) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server headers=headers, params=params) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server ret = obj(*args, **kwargs) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server self._handle_fault_response(status_code, replybody, resp) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server ret = obj(*args, **kwargs) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server exception_handler_v20(status_code, error_body) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-18.0.0/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server request_ids=request_ids) 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server InternalServerError: Request Failed: internal server error while processing your request. 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server Neutron server returns request_ids: ['req-94584484-32b4-40c2-8740-9ba56b92a7fc'] 2019-02-13 13:42:21.816 56155 ERROR oslo_messaging.rpc.server ^C root@cc-comput