Steps to reproduce ================== 1、Deployment Stein-stable version openstack. The number of compute nodes is greater than 2. 2、Add logs on nova-compute of src-node Edit nova\objects\instance.py def obj_load_attr(self, attrname): # NOTE(danms): We can't lazy-load anything without a context and a uuid if not self._context: raise exception.OrphanedObjectError(method='obj_load_attr', objtype=self.obj_name()) if 'uuid' not in self: raise exception.ObjectActionError( action='obj_load_attr', reason=_('attribute %s not lazy-loadable') % attrname) -- LOG.debug("Lazy-loading '%(attr)s' on %(name)s uuid %(uuid)s", ++ LOG.debug("Lazy-loading '%(attr)s' on %(name)s uuid %(uuid)s context %(context)s", {'attr': attrname, 'name': self.obj_name(), 'uuid': self.uuid, 'context' : self._context, }) with utils.temporary_mutation(self._context, read_deleted='yes'): self._obj_load_attr(attrname) ++ LOG.debug("Exit obj_load_attr context &(context)s" {'context' : self._context, }) 3、Open the debug output in src-node edit nova.conf debug=true 4、Restart nova-compute service of src-node. 5、Launch 25 instances in src-node. 6、nova host-evacuate-live --target-host dest-node src-node ================== Logs 2021-09-09 17:22:15.170 17078 INFO nova.compute.manager [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] [instance: 37633020-5f24-4098-9468-f356015dae39] Took 14.54 seconds for pre_live_migration on destination host computer2. 2021-09-09 17:22:16.264 17078 DEBUG nova.compute.manager [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] live_migration data is LibvirtLiveMigrateData(bdms=[LibvirtLiveMigrateBDMInfo],block_migration=False,disk_available_mb=453632,disk_over_commit=,dst_numa_info=,dst_supports_numa_live_migration=,dst_wants_file_backed_memory=False,file_backed_memory_discard=False,filename='tmpD3AV46',graphics_listen_addr_spice=127.0.0.1,graphics_listen_addr_vnc=0.0.0.0,image_type='default',instance_relative_path='37633020-5f24-4098-9468-f356015dae39',is_shared_block_storage=True,is_shared_instance_path=False,is_volume_backed=True,migration=Migration(5fbbf27a-f41d-4b1e-b90b-0137f296143f),old_vol_attachment_ids={},serial_listen_addr=None,serial_listen_ports=[],src_supports_native_luks=True,src_supports_numa_live_migration=,supported_perf_events=[],target_connect_addr=None,vifs=[VIFMigrateData],wait_for_vif_plugged=True) _do_live_migration /usr/lib/python2.7/site-packages/nova/compute/manager.py:7161 2021-09-09 17:22:16.269 17078 DEBUG nova.objects.instance [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] Lazy-loading 'flavor' on Instance uuid 37633020-5f24-4098-9468-f356015dae39 context obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1087 2021-09-09 17:22:16.272 17078 DEBUG nova.virt.libvirt.driver [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] [instance: 37633020-5f24-4098-9468-f356015dae39] Starting monitoring of live migration _live_migration /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:8921 2021-09-09 17:22:16.274 17078 DEBUG nova.objects.instance [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] Lazy-loading 'flavor' on Instance uuid 37633020-5f24-4098-9468-f356015dae39 context obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1087 2021-09-09 17:22:16.543 17078 DEBUG nova.utils [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] Fiona_debug: Exit temporary_mutation temporary_mutation /usr/lib/python2.7/site-packages/nova/utils.py:450 2021-09-09 17:22:16.543 17078 DEBUG nova.objects.instance [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] Fiona_debug:Exit obj_load_attr context obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1094 2021-09-09 17:22:16.552 17078 DEBUG nova.virt.libvirt.migration [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] Find same serial number: pos=0, serial=f28d9811-6294-4e5c-be68-4434f1442fd6 _update_volume_xml /usr/lib/python2.7/site-packages/nova/virt/libvirt/migration.py:218 2021-09-09 17:22:16.555 17078 DEBUG nova.virt.libvirt.vif [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=True,config_drive='',created_at=2021-09-06T08:31:59Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=,disable_terminate=False,display_description=None,display_name='test-for-livemigrate-3',ec2_ids=,ephemeral_gb=0,ephemeral_key_uuid=None,fault=,flavor=Flavor(4),hidden=False,host='computer1',hostname='test-for-livemigrate-3',id=761,image_ref='',info_cache=InstanceInfoCache,instance_type_id=4,kernel_id='',key_data=None,key_name=None,keypairs=,launch_index=2,launched_at=2021-09-06T08:33:32Z,launched_on='computer1',locked=False,locked_by=None,memory_mb=100,metadata={},migration_context=,new_flavor=None,node='computer1',numa_topology=None,old_flavor=None,os_type=None,pci_devices=,pci_requests=InstancePCIRequests,power_state=1,progress=0,project_id='a3bdbc69f4d94ca488ebb24d00b3d18c',ramdisk_id='',reservation_id='r-28izfhjj',resources=,root_device_name='/dev/vda',root_gb=1,security_groups=SecurityGroupList,services=,shutdown_terminate=False,system_metadata={boot_roles='reader,member,admin',clean_attempts='2',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_min_disk='1',image_min_ram='0',owner_project_name='admin',owner_user_name='admin'},tags=,task_state='migrating',terminated_at=None,trusted_certs=,updated_at=2021-09-09T03:10:07Z,user_data=None,user_id='e3c29f7fa78a46d1ba2cd6e8cacf6610',uuid=37633020-5f24-4098-9468-f356015dae39,vcpu_model=,vcpus=1,vm_mode=None,vm_state='active') vif={"profile": {}, "ovs_interfaceid": "50d5c411-0b7d-4c93-8de5-365760c1701a", "preserve_on_delete": false, "network": {"bridge": "br-int", "label": "selfservice01", "meta": {"injected": false, "tunneled": true, "mtu": 1450, "physical_network": null, "tenant_id": "a3bdbc69f4d94ca488ebb24d00b3d18c"}, "id": "3bec06d0-e00d-49d1-bdbb-c44b9b241a0b", "subnets": [{"ips": [{"meta": {}, "type": "fixed", "version": 4, "address": "10.0.0.150", "floating_ips": []}], "version": 4, "meta": {"dhcp_server": "10.0.0.2"}, "dns": [], "routes": [], "cidr": "10.0.0.0/24", "gateway": {"meta": {}, "type": "gateway", "version": 4, "address": "10.0.0.1"}}]}, "devname": "tap50d5c411-0b", "qbh_params": null, "vnic_type": "normal", "meta": {}, "details": {"ovs_hybrid_plug": true, "connectivity": "l2", "bridge_name": "br-int", "datapath_type": "system", "port_filter": true}, "address": "fa:16:3e:77:94:9f", "active": true, "type": "ovs", "id": "50d5c411-0b7d-4c93-8de5-365760c1701a", "qbg_params": null} virt_type=kvm get_config /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:571 2021-09-09 17:22:16.556 17078 DEBUG nova.network.os_vif_util [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] Converting VIF {"profile": {}, "ovs_interfaceid": "50d5c411-0b7d-4c93-8de5-365760c1701a", "preserve_on_delete": false, "network": {"bridge": "br-int", "label": "selfservice01", "meta": {"injected": false, "tunneled": true, "mtu": 1450, "physical_network": null, "tenant_id": "a3bdbc69f4d94ca488ebb24d00b3d18c"}, "id": "3bec06d0-e00d-49d1-bdbb-c44b9b241a0b", "subnets": [{"ips": [{"meta": {}, "type": "fixed", "version": 4, "address": "10.0.0.150", "floating_ips": []}], "version": 4, "meta": {"dhcp_server": "10.0.0.2"}, "dns": [], "routes": [], "cidr": "10.0.0.0/24", "gateway": {"meta": {}, "type": "gateway", "version": 4, "address": "10.0.0.1"}}]}, "devname": "tap50d5c411-0b", "qbh_params": null, "vnic_type": "normal", "meta": {}, "details": {"ovs_hybrid_plug": true, "connectivity": "l2", "bridge_name": "br-int", "datapath_type": "system", "port_filter": true}, "address": "fa:16:3e:77:94:9f", "active": true, "type": "ovs", "id": "50d5c411-0b7d-4c93-8de5-365760c1701a", "qbg_params": null} nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:516 2021-09-09 17:22:16.557 17078 DEBUG nova.network.os_vif_util [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] Converted object VIFBridge(active=True,address=fa:16:3e:77:94:9f,bridge_name='qbr50d5c411-0b',has_traffic_filtering=True,id=50d5c411-0b7d-4c93-8de5-365760c1701a,network=Network(3bec06d0-e00d-49d1-bdbb-c44b9b241a0b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap50d5c411-0b') nova_to_osvif_vif /usr/lib/python2.7/site-packages/nova/network/os_vif_util.py:553 2021-09-09 17:22:16.564 17078 DEBUG nova.utils [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] Fiona_debug: Exit temporary_mutation temporary_mutation /usr/lib/python2.7/site-packages/nova/utils.py:450 2021-09-09 17:22:16.565 17078 DEBUG nova.objects.instance [req-6d7e1979-2d0c-4df0-809c-ba9af187532b - - - - -] Fiona_debug:Exit obj_load_attr context obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1094 ================== analysis of logs and code Instance didn’t load flavor attr before calling nova/virt/libvirt/driver.py LibvirtDriver _live_migration in response to live-migrate calling. The assignment operation of inst_type=instance.flavor will call Instance.obj_load_attr in _live_migration_operation.As obj_load_attr calls RPC back to controller node to read data from the database, and that RPC is an IO, thread switching happen. The function _live_migration_monitor will be scheduled, assigning ram_gb call obj_load_attr again. After two calls of obj_load_attr may lead to the value of context.read_deleted is true. This doesn’t cause errors, but subsequent operations of objects.InstanceList.get_by_host_and_node are time-consuming. The vaule of context.read_deleted shouldn’t be changed after calling obj_load_attr. Because of T version of nova/compute/api.py:API.live_migrate is decorated by reject_sev_intances, which loaded instance.flavor，we need to remove decorator to reproduce the problem in Train .