Live migrate UnexpectedTaskStateError
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
New
|
Undecided
|
Unassigned |
Bug Description
Description
===========
Occasionally, when performing live migration, the instance goes to ERROR state with the message "UnexpectedTask
The migration is always successful so far and the instance is always found residing on the target host. Updating the instances table "node" and "host" rows in the nova database with the destination host and then resetting the instance state to ACTIVE gets things back in order.
The issue appears to occur randomly across 16 uniform compute nodes and seems like a race condition.
Steps to reproduce
==================
1. Boot an instance to Compute01
2. Issue a live-migrate command for instance targeting Compute02 (this can be done via Horizon and via python-
# openstack server migrate --shared-migration --live computehost02 e8928cb2-
3. Live-migration works, the instance remains accessible and is moved to the new host. However, ~20% of the time, the instance goes to ERROR state and some cleanup must be done in the database.
MariaDB [nova]> update instances set node='computeho
# openstack server set --state active e8928cb2-
Expected result
===============
The migrated instance should move successfully and return to ACTIVE state.
Actual result
=============
Instances occassional end up in ERROR state after a "successful" live-migration.
Environment
===========
1. This is a Newton environment with Nova Libvirt/KVM backed by Ceph. Networking is provided by Neutron ML2 linux bridge agent.
root@
14.0.8
root@
ii libvirt-bin 1.3.1-1ubuntu10.15 amd64 programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu10.15 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1.1 amd64 libvirt Python bindings
root@
ii ipxe-qemu 1.0.0+git-
ii qemu 1:2.5+dfsg-
ii qemu-block-
ii qemu-slof 20151103+
ii qemu-system 1:2.5+dfsg-
ii qemu-system-arm 1:2.5+dfsg-
ii qemu-system-common 1:2.5+dfsg-
ii qemu-system-mips 1:2.5+dfsg-
ii qemu-system-misc 1:2.5+dfsg-
ii qemu-system-ppc 1:2.5+dfsg-
ii qemu-system-sparc 1:2.5+dfsg-
ii qemu-system-x86 1:2.5+dfsg-
ii qemu-user 1:2.5+dfsg-
ii qemu-utils 1:2.5+dfsg-
root@
root@
ii ceph 10.2.10-1xenial amd64 distributed storage and file system
ii ceph-base 10.2.10-1xenial amd64 common ceph daemon libraries and management tools
ii ceph-common 10.2.10-1xenial amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mon 10.2.10-1xenial amd64 monitor server for the ceph storage system
ii ceph-osd 10.2.10-1xenial amd64 OSD server for the ceph storage system
ii libcephfs1 10.2.10-1xenial amd64 Ceph distributed file system client library
ii python-cephfs 10.2.10-1xenial amd64 Python libraries for the Ceph libcephfs library
ii python-rados 10.2.10-1xenial amd64 Python libraries for the Ceph librados library
ii python-rbd 10.2.10-1xenial amd64 Python libraries for the Ceph librbd library
root@
Logs & Configs
==============
This is the specific error message that the target host generates in nova-compute.log:
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
oslo_messaging.
tags: | added: live-migration |
Probably related to but not the same as stale bug https:/ /bugs.launchpad .net/nova/ +bug/1646841