nova-compute could not destroy evacuated instance which was deleted

[Yang Youseok]

Description
===========
After evacuation, nova-compute where VMs were evacuated could not restart emitting InstanceNotFound exception when init_host() called.

Steps to reproduce
==================
0) service nova-compute stop at 'compute01'
1) nova evacuate 'compute01'
2) nova delete 'server at compute01'
3) service nova-compute restart at 'compute01'

Expected result
===============
nova-compute normally started

Actual result
=============
nova-compute could not started

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

  stable/ocata

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

  libvirt + kvm

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   Ceph

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)
   neutron with linux bridge

Logs & Configs
==============
2019-02-26 16:24:29.015 8901 INFO nova.virt.libvirt.driver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] [instance: 25ab917a-1506-4154-9902-5b8b550ec929] Deleting instance files /var/lib/nova/instances/25ab917a-1506-4154-9902-5b8b550ec929_del
2019-02-26 16:24:29.017 8901 INFO nova.virt.libvirt.driver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] [instance: 25ab917a-1506-4154-9902-5b8b550ec929] Deletion of /var/lib/nova/instances/25ab917a-1506-4154-9902-5b8b550ec929_del complete
2019-02-26 16:24:29.020 8901 DEBUG oslo_messaging._drivers.amqpdriver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] CALL msg_id: 9bbf4371e977450d98a519dac213592b exchange 'nova' topic 'conductor' _send /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:550
2019-02-26 16:24:29.085 8901 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 9bbf4371e977450d98a519dac213592b __call__ /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:407
2019-02-26 16:24:29.093 8901 DEBUG oslo_messaging._drivers.amqpdriver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] CAST unique_id: d5a0d3b93fbf42b6a379b9507e9c2a9a FANOUT topic 'scheduler' _send /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:539
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] Error starting thread.
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service Traceback (most recent call last):
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_service/service.py", line 722, in run_service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service service.start()
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/service.py", line 144, in start
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service self.manager.init_host()
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/compute/manager.py", line 1155, in init_host
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service self._destroy_evacuated_instances(context)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/compute/manager.py", line 674, in _destroy_evacuated_instances
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service bdi, destroy_disks)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/virt/libvirt/driver.py", line 915, in destroy
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service destroy_disks, migrate_data)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/virt/libvirt/driver.py", line 1034, in cleanup
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service instance.save()
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service ctxt, self, fn.__name__, args, kwargs)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/conductor/rpcapi.py", line 244, in object_action
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service objmethod=objmethod, args=args, kwargs=kwargs)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service retry=self.retry)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service timeout=timeout, retry=retry)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service retry=retry)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service raise result
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service InstanceNotFound_Remote: Instance 25ab917a-1506-4154-9902-5b8b550ec929 could not be found.
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service Traceback (most recent call last):
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/conductor/manager.py", line 93, in _object_dispatch
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return getattr(target, method)(*args, **kwargs)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return fn(self, *args, **kwargs)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/objects/instance.py", line 792, in save
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service columns_to_join=_expected_cols(expected_attrs))
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/api.py", line 840, in instance_update_and_get_original
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service expected=expected)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return f(*args, **kwargs)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service ectxt.value = e.inner_exc
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return f(*args, **kwargs)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return f(context, *args, **kwargs)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 2714, in instance_update_and_get_original
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service columns_to_join=columns_to_join)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 1910, in _instance_get_by_uuid
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service raise exception.InstanceNotFound(instance_id=uuid)
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service InstanceNotFound: Instance 25ab917a-1506-4154-9902-5b8b550ec929 could not be found.
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.097 8901 ERROR oslo_service.service
2019-02-26 16:24:29.102 8901 DEBUG oslo_concurrency.lockutils [req-67496f45-7809-41c4-a6a1-365574533016 - - - - -] Acquired semaphore "singleton_lock" lock /opt/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2019-02-26 16:24:29.103 8901 DEBUG oslo_concurrency.lockutils [req-67496f45-7809-41c4-a6a1-365574533016 - - - - -] Releasing semaphore "singleton_lock" lock /opt/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225

[Matt Riedemann (mriedem)]

This might already be fixed for bug 1794996:

https://review.openstack.org/#/q/I1f4b3540dd453650f94333b36d7504ba164192f7

But that was not backported to stable/ocata.

If that fixes your issue we could work on backporting https://review.openstack.org/#/q/topic:bug/1550919+branch:stable/pike to stable/ocata.

[Yang Youseok (ileixe)]

Thanks Matt!

After cherry-picking the commit you said, it was fixed.