OpenStack Compute (nova)

Live-migration failing with error NotImplementedError: Cannot load 'old_resources' in the base class

Bug #1955013 reported by Mitya Eremeev
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Undecided
Mitya Eremeev

Bug Description

Steps to reproduce
==================
While live-migrating the instances from the compute few instances failed with error
NotImplementedError: Cannot load 'old_resources' in the base class.

Actual result
=============
the state of failed instance:

$ openstack server show 6f01e3fe-7591-45a7-8246-59b8bd7aa224 --fit-width
+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:instance_name | instance-0013db33 |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2021-07-22T10:46:34.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | True |
| created | 2021-07-22T10:46:06Z |
| fault | {'code': 500, 'created': '2021-12-03T11:20:38Z', 'message': 'NotImplementedError', 'details': 'Traceback (most recent call last):\n File "/var/lib/openstack/lib/python3.6/site- |
| | packages/nova/compute/manager.py", line 206, in decorated_function\n return function(self, context, *args, **kwargs)\n File "/var/lib/openstack/lib/python3.6/site- |
| | packages/nova/compute/manager.py", line 8765, in _post_live_migration\n migrate_data, migr_ctxt=instance.migration_context)\n File "/var/lib/openstack/lib/python3.6/site- |
| | packages/nova/compute/manager.py", line 8600, in _live_migration_cleanup_flags\n if migr_ctxt and migr_ctxt.old_resources:\n File "/var/lib/openstack/lib/python3.6/site- |
| | packages/oslo_versionedobjects/base.py", line 67, in getter\n self.obj_load_attr(name)\n File "/var/lib/openstack/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 601, in |
| | obj_load_attr\n _("Cannot load \'%s\' in the base class") % attrname)\nNotImplementedError: Cannot load \'old_resources\' in the base class\n'} |
| status | ERROR |
| updated | 2021-12-03T11:25:25Z |
| user_id | 1fcd7470ddce40bdb0ede82745479450 |
| volumes_attached | |
+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ nova migration-list --host <HOST> --changes-since 2021-12-03 |grep 6f01e3fe-7591-45a7-8246-59b8bd7aa224
| 21789 | bb1aaaf8-b6ee-4f80-bddf-4abb8a543bfb | 8524947d-8ed6-405e-aa37-0a6852d32480.b8a63eb1e25d4a64ace6c25e0a390c4d | e84d0c19-d14e-4981-bfa6-c0dd26fc306d.b8a63eb1e25d4a64ace6c25e0a390c4d | 8524947d-8ed6-405e-aa37-0a6852d32480 | e84d0c19-d14e-4981-bfa6-c0dd26fc306d | - | error | 6f01e3fe-7591-45a7-8246-59b8bd7aa224 | 118 | 118 | 2021-12-03T11:17:18.000000 | 2021-12-03T11:20:31.000000 | live-migration | 10479799101a4fe4ada17daa105707c5 | 1df8a7d5ebb5414b9e29cf581098681c |

VM’s that failed to live-migrate with mentioned error have their disk copied successfully to the destination compute and virsh list shows VM is running on the destination compute but the openstack server show still shows the VM is on source compute.
If we hard reboot the VM then the VM shows as running on both source and destination computes.

RCA
---
"Old_resources" was introduced in Train
https://opendev.org/openstack/nova/commit/384d783fc65223d5684d0fb7ea1e7fb0c44eeae1

DB migration was performed
MariaDB [nova]> describe instance_extra;
+-------------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+-------------+------+-----+---------+----------------+
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| deleted_at | datetime | YES | | NULL | |
| deleted | int(11) | YES | | NULL | |
| id | int(11) | NO | PRI | NULL | auto_increment |
| instance_uuid | varchar(36) | NO | MUL | NULL | |
| numa_topology | text | YES | | NULL | |
| pci_requests | text | YES | | NULL | |
| flavor | text | YES | | NULL | |
| vcpu_model | text | YES | | NULL | |
| migration_context | text | YES | | NULL | |
| keypairs | text | YES | | NULL | |
| device_metadata | text | YES | | NULL | |
| trusted_certs | text | YES | | NULL | |
| vpmems | text | YES | | NULL | |
| resources | text | YES | | NULL | |
+-------------------+-------------+------+-----+---------+----------------+

2021-12-03 11:19:07.691 1117 INFO nova.virt.libvirt.migration [-] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] Increasing downtime to 50 ms after 0 sec elapsed time
2021-12-03 11:19:07.985 1117 INFO nova.virt.libvirt.driver [-] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] Migration running for 0 secs, memory 100% remaining (bytes processed=0, remaining=0, total=0); disk 100% remaining (bytes processed=0, remaining=0, total=0).
2021-12-03 11:19:08.047 1117 WARNING nova.compute.manager [req-f7455d0d-9c07-4936-a1a7-fd98eaee1722 a04bf8c78dab4f6c92d6cea3763f485a da7a56c3c1cd4111a997d7fc5f01cc59 - ec2d6bea230a4938a3013d2cf30b7a38 ec2d6bea230a4938a3013d2cf30b7a38] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] Received unexpected event network-vif-plugged-cab4e7d5-12ac-497d-abd2-8d1f7f7f4cac for instance with vm_state active and task_state migrating.
2021-12-03 11:19:08.257 1117 WARNING nova.compute.manager [req-a9d84808-82fa-4df2-85ff-67333684977f a04bf8c78dab4f6c92d6cea3763f485a da7a56c3c1cd4111a997d7fc5f01cc59 - ec2d6bea230a4938a3013d2cf30b7a38 ec2d6bea230a4938a3013d2cf30b7a38] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] Received unexpected event network-vif-plugged-7fff54c8-7599-4cab-a19f-7dfe2851a244 for instance with vm_state active and task_state migrating.
2021-12-03 11:19:39.576 1117 INFO nova.virt.libvirt.driver [-] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] Migration running for 30 secs, memory 100% remaining (bytes processed=0, remaining=0, total=0); disk 62% remaining (bytes processed=5182193664, remaining=8463908864, total=13646102528).
2021-12-03 11:20:10.617 1117 INFO nova.virt.libvirt.driver [-] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] Migration running for 60 secs, memory 100% remaining (bytes processed=0, remaining=0, total=0); disk 24% remaining (bytes processed=10332209152, remaining=3315400704, total=13647609856).
2021-12-03 11:20:32.676 1117 INFO nova.compute.manager [req-a5fe4d7c-46b4-4ab8-af45-1b7c85e6abb6 - - - - -] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] VM Paused (Lifecycle Event)
2021-12-03 11:20:32.800 1117 INFO nova.compute.manager [req-a5fe4d7c-46b4-4ab8-af45-1b7c85e6abb6 - - - - -] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] During sync_power_state the instance has a pending task (migrating). Skip.
2021-12-03 11:20:33.775 1117 INFO nova.virt.libvirt.driver [-] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] Migration operation has completed
2021-12-03 11:20:33.776 1117 INFO nova.compute.manager [-] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] _post_live_migration() is started..
2021-12-03 11:20:37.099 1117 WARNING nova.compute.manager [req-b26cd838-5b69-4249-b21c-93fbb7b2749b a04bf8c78dab4f6c92d6cea3763f485a da7a56c3c1cd4111a997d7fc5f01cc59 - ec2d6bea230a4938a3013d2cf30b7a38 ec2d6bea230a4938a3013d2cf30b7a38] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] Received unexpected event network-vif-plugged-79d5e238-fc1d-46b4-85b7-63ee6dc2a0de for instance with vm_state active and task_state migrating.
2021-12-03 11:20:38.506 1117 WARNING nova.virt.libvirt.driver [-] [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] Error monitoring migration: Cannot load 'old_resources' in the base class: NotImplementedError: Cannot load 'old_resources' in the base class
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] Traceback (most recent call last):
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 9593, in _live_migration
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] finish_event, disk_paths)
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 9504, in _live_migration_monitor
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] migrate_data)
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/nova/exception_wrapper.py", line 78, in wrapped
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] function_name, call_dict, binary)
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] self.force_reraise()
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] six.reraise(self.type_, self.value, self.tb)
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] raise value
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] return f(self, context, *args, **kw)
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 218, in decorated_function
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] kwargs['instance'], e, sys.exc_info())
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] self.force_reraise()
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] six.reraise(self.type_, self.value, self.tb)
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] raise value
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 206, in decorated_function
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] return function(self, context, *args, **kwargs)
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 8765, in _post_live_migration
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] migrate_data, migr_ctxt=instance.migration_context)
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 8600, in _live_migration_cleanup_flags
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] if migr_ctxt and migr_ctxt.old_resources:
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 67, in getter
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] self.obj_load_attr(name)
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] File "/var/lib/openstack/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 601, in obj_load_attr
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] _("Cannot load '%s' in the base class") % attrname)
2021-12-03 11:20:38.506 1117 ERROR nova.virt.libvirt.driver [instance: 6f01e3fe-7591-45a7-8246-59b8bd7aa224] NotImplementedError: Cannot load 'old_resources' in the base class

So root cause was not found.
We can add some check if migration context has member "old_resources"

Environment
===========
release: Victoria
hypervisor: Libvirt + KVM
storage: local folders
networking: Neutron

Mitya Eremeev (mitos)
Changed in nova:
assignee: nobody → Mitya Eremeev (mitos)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/821947

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.