nova-compute doesn't start if there is difference between current compute driver and driver which was used to create instance

[Rufus Deponian]

Steps to reproduce
==================
1. Create instance with (for example) qemu as nova-compute backend.
2. Change nova-compute backend to (for example) lxd.
3. Restart nova-compute service

Expected result
===============
I expected to see an error with something like this: "You have to delete all your instances, which were created with old nova-compute driver. Use 'openstack server delete instance-name' on your controller node."

Actual result
=============
nova-compute service doesn't start and there is no clear explanation in nova-compute.log (see log below).

Environment
===========
1. Version of OpenStack is Ocata:
user@compute ~> dpkg -l | grep nova
rc nova-api 2:15.0.2-0ubuntu1~cloud0 all OpenStack Compute - API frontend
ii nova-common 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - compute node base
rc nova-compute-kvm 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii nova-compute-lxd 15.0.2-0ubuntu1~cloud0 all Openstack Compute - LXD container hypervisor support
rc nova-conductor 2:15.0.2-0ubuntu1~cloud0 all OpenStack Compute - conductor service
rc nova-consoleauth 2:15.0.2-0ubuntu1~cloud0 all OpenStack Compute - Console Authenticator
rc nova-novncproxy 2:15.0.2-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
rc nova-placement-api 2:15.0.2-0ubuntu1~cloud0 all OpenStack Compute - placement API frontend
rc nova-scheduler 2:15.0.2-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-nova-lxd 15.0.2-0ubuntu1~cloud0 all OpenStack Compute Python libraries - LXD driver
ii python-novaclient 2:7.1.0-0ubuntu1~cloud0 all client library for OpenStack Compute API - Python 2.7

2. Hypervisors: qemu and lxd

3. Storage: lvm

4. Networking type: Neutron with OpenVSwitch

Log
==============
nova-compute.log:
2017-08-08 16:18:51.112 29592 INFO nova.service [-] Starting compute node (version 15.0.5)
2017-08-08 16:18:51.882 29592 INFO oslo.privsep.daemon [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--config-file', '/etc/nova/nova-compute.conf', '--privsep_context', 'vif_plug_linux_bridge.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmpGK77AD/privsep.sock']
2017-08-08 16:18:53.810 29592 INFO oslo.privsep.daemon [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Spawned new privsep daemon via rootwrap
2017-08-08 16:18:53.812 29592 INFO oslo.privsep.daemon [-] privsep daemon starting
2017-08-08 16:18:53.812 29592 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2017-08-08 16:18:53.813 29592 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none
2017-08-08 16:18:53.813 29592 INFO oslo.privsep.daemon [-] privsep daemon running as pid 29634
2017-08-08 16:18:53.957 29592 INFO os_vif [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:ae:ba:06,bridge_name='brq3b3f3c75-8f',has_traffic_filtering=True,id=4a1e4eaa-17d3-4baa-a2c5-9f6a7369272c,network=Network(3b3f3c75-8f2b-4fe0-b91e-9c3dd53fe9ec),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap4a1e4eaa-17')
2017-08-08 16:18:54.310 29592 INFO os_vif [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:7e:a4:a1,bridge_name='brq3b3f3c75-8f',has_traffic_filtering=True,id=ae4b3102-ad58-474c-bfbf-da6b083eda9b,network=Network(3b3f3c75-8f2b-4fe0-b91e-9c3dd53fe9ec),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapae4b3102-ad')
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Error starting thread.
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service Traceback (most recent call last):
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service service.start()
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service self.manager.init_host()
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1152, in init_host
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service self._init_instance(context, instance)
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 862, in _init_instance
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service current_power_state = self._get_power_state(context, instance)
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1175, in _get_power_state
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service return self.driver.get_info(instance).state
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 306, in get_info
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service container = self.client.containers.get(instance.name)
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/pylxd/models/container.py", line 104, in get
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service response = client.api.containers[name].get()
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/pylxd/client.py", line 103, in get
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service self._assert_response(response, stream=kwargs.get('stream', False))
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/pylxd/client.py", line 68, in _assert_response
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service raise exceptions.NotFound(response)
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service NotFound: not found
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service

[Matt Riedemann (mriedem)]

This is a bug in the nova-lxd driver code, which manages bugs through some other launchpad project.

[James Page (james-page)]

Tricky one this as the main nova driver has no knowledge of the nova-lxd driver (and as a result can interrogate the LXD daemon to figure things out).

I'm pretty sure that this is an issue that's common to any of the drivers in Nova as well; it you can the driver after spinning up instances, you effectively island control of an instances created with the previous driver.

[James Page (james-page)]

I'm marking this as Triaged and Low; I'm not sure how much we can realistically do being and out-of-tree driver, but this at least leaves a bug trail for anyone else tripping over this problem.