After update of overcloud with new compute image user vm are unavailable

[Michael Kerrin]

After everything coming up user VM's are left in the NO_STATE state.

Running nova console-log demo I get a 404 error and the compute log has the following error:

014-04-04 12:14:34.583 2440 AUDIT nova.compute.manager [req-1d2adef8-2130-4281-8ba7-8afdcf5ccc6a 9ff04f8d51af45d3a7929cf4aec35d78 5ee9f4b7fe7d4339b06f99ac2adc3b3e] [instance: c0cee332-837c-4f9a-b6d4-490a2bafb58d] Get console output
2014-04-04 12:14:34.586 2440 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Instance instance-00000001 could not be found.
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher return func(*args, **kwargs)
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher payload)
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 296, in decorated_function
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 3950, in get_console_output
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher output = self.driver.get_console_output(context, instance)
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2281, in get_console_output
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher virt_dom = self._lookup_by_name(instance.name)
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3473, in _lookup_by_name
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher raise exception.InstanceNotFound(instance_id=instance_name)
2014-04-04 12:14:34.586 2440 TRACE oslo.messaging.rpc.dispatcher InstanceNotFound: Instance instance-00000001 could not be found.

Running virtsh list --all I see no instances listed at all.

Note that this is using the persistent ephemeral disks as can be seen from the output of df

root@overcloud-novacompute1-nlzir7qicvfn:/var/log/upstart# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 9.4G 1.9G 7.1G 21% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 2.0G 8.0K 2.0G 1% /dev
tmpfs 396M 380K 396M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda1 30G 1.4G 27G 5% /mnt

[Michael Kerrin (michael-kerrin-w)]

Ok something weird with the update script

[Michael Kerrin (michael-kerrin-w)]

This is still happening but it just doesn't happen all the time, hence why I marked it as invalid.

[jan grant (jan-grant)]

Worthwhile checking that /etc/libvirt is symlinked to /var/state/etc/libvirt. Domain definitions get stashed under there.

[Ben Nemec (bnemec)]

We aren't doing image-based updates anymore, so this probably isn't relevant.