OpenStack Compute (nova)

When post_live_migration_at_destination fails the instance is not put into ERROR/None vm_state/task_state

Bug #1818873 reported by Matt Riedemann on 2019-03-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Triaged	Medium	Unassigned

Bug Description

Seen here:

http://logs.openstack.org/43/635343/4/check/tempest-slow-py3/a2497ae/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Mar_06_05_26_25_035103

Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [-] [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Post live migration at destination ubuntu-bionic-rax-iad-0003421804 failed: AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: Traceback (most recent call last):
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: res = self.dispatcher.dispatch(message)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return self._do_dispatch(endpoint, method, ctxt, args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: result = func(ctxt, **new_args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: function_name, call_dict, binary, tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return f(self, context, *args, **kw)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/utils.py", line 1301, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: kwargs['instance'], e, sys.exc_info())
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 6942, in post_live_migration_at_destination
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2815, in migrate_instance_finish
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: context, instance, migration.dest_compute, migration=migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Traceback (most recent call last):
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 6846, in _post_live_migration
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] instance, block_migration, dest)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/rpcapi.py", line 716, in post_live_migration_at_destination
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] instance=instance, block_migration=block_migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/client.py", line 178, in call
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] retry=self.retry)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/transport.py", line 128, in _send
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] retry=retry)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] call_monitor_timeout, retry=retry)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] raise result
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Traceback (most recent call last):
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] res = self.dispatcher.dispatch(message)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return self._do_dispatch(endpoint, method, ctxt, args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] result = func(ctxt, **new_args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] function_name, call_dict, binary, tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return f(self, context, *args, **kw)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/utils.py", line 1301, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] kwargs['instance'], e, sys.exc_info())
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 6942, in post_live_migration_at_destination
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2815, in migrate_instance_finish
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] context, instance, migration.dest_compute, migration=migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]

Apparently we don't hit this except block which would set the instance to ERROR state because I don't see that traceback and message in the logs:

https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6537

So we end up with something like this (failed tempest test):

b'Details: (TestNetworkAdvancedServerOps:test_server_connectivity_live_migration) Server b78e10fb-44b8-4010-a72d-c86410950c15 failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: MIGRATING. Current task state: migrating.'

It looks like the error from post_live_migration_at_destination is logged and ignored here:

https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6844

And post_live_migration_at_destination is what sets the fields on the instance:

https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6971

So the task_state is never reset.

This is a mess which will take some thought to clean up because if we get to _post_live_migration the guest is on the destination host, but the database is all messed up if post_live_migration_at_destination fails early.

Tags:

Revision history for this message

Matt Riedemann (mriedem) wrote on 2019-03-06:

Note that the migration.status is also set to 'completed' even if post_live_migration_at_destination fails:

https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6883

Revision history for this message

Matt Riedemann (mriedem) wrote on 2019-03-06:

This might be a duplicate of bug 1628606.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.