Instance stuck in resize_prep when migrating from down compute, moves to ERROR when compute returns.

[Lee Yarwood]

Description
===========
Instance stuck in resize_prep when migrating from down compute, moves to ERROR when compute returns later.

Steps to reproduce
==================

- Launch an instance:

- Stop compute service on the host:

$ sudo systemctl stop devstack@n-cpu

$ openstack compute service list
[..]
| 3 | nova-compute | host2.example.com | nova | enabled | down | 2018-12-07T18:29:35.000000 |
[..]

- Attempt to migrate the instance to another host:

$ openstack server migrate test

- Review the task_state of the instance:

$ openstack server show test
[..]
|
| OS-EXT-STS:task_state | resize_prep [..]

- Restart the compute service:

$ sudo systemctl start devstack@n-cpu

$ openstack compute service list
[..]
| 3 | nova-compute | host2.example.com | nova | enabled | up | 2018-12-07T18:42:00.000000 |
[..]

- - Review the task_state and status of the instance:

$ openstack server show test
[..]
| OS-EXT-STS:task_state | resize_migrating [..]
| status | ERROR [..]

Expected result
===============

No attempt to migrate an instance is made while the source compute host is marked as down.

Actual result
=============

Instance left in an ERROR state after the compute service returns.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   master

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   N/A

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============

Trace when the compute service restarts after attempting the migration:

Dec 07 13:32:15 host2.example.com nova-compute[27522]: ERROR nova.compute.manager [None req-13de2cb9-0eac-4e59-b777-e83ce5bb1d3c admin admin] [instance: b86238ba-2f3b-4b59-bdcb-490fbb90fb52] Setting instance vm_state to ERROR: UnexpectedTaskStateError_Remote: Conflict upda>
Dec 07 13:32:15 host2.example.com nova-compute[27522]: Traceback (most recent call last):
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/conductor/manager.py", line 128, in _object_dispatch
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return getattr(target, method)(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return fn(self, *args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/objects/instance.py", line 843, in save
Dec 07 13:32:15 host2.example.com nova-compute[27522]: columns_to_join=_expected_cols(expected_attrs))
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/api.py", line 889, in instance_update_and_get_original
Dec 07 13:32:15 host2.example.com nova-compute[27522]: expected=expected)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 170, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return f(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: ectxt.value = e.inner_exc
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 07 13:32:15 host2.example.com nova-compute[27522]: self.force_reraise()
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 07 13:32:15 host2.example.com nova-compute[27522]: six.reraise(self.type_, self.value, self.tb)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return f(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapped
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return f(context, *args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2736, in instance_update_and_get_original
Dec 07 13:32:15 host2.example.com nova-compute[27522]: context, instance_uuid, values, expected, original=instance_ref))
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2872, in _instance_update
Dec 07 13:32:15 host2.example.com nova-compute[27522]: raise exc(**exc_props)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: UnexpectedTaskStateError: Conflict updating instance b86238ba-2f3b-4b59-bdcb-490fbb90fb52. Expected: {'task_state': [u'resize_prep']}. Actual: {'task_state': None}

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lee Yarwood (<email address hidden>) on branch: master
Review: https://review.opendev.org/623489