nova-compute is started before libvirt

[Ante Karamatić]

Ubuntu 14.04, nova 2014.1.2-0ubuntu1.1

Rebooting nova-compute node results in a non-running nova-compute service. This is due the fact that nova-compute starts before libvirt:

2014-10-19 01:49:20.686 2993 AUDIT nova.service [-] Starting compute node (version 2014.1.2)
2014-10-19 01:50:15.141 2993 WARNING nova.virt.libvirt.driver [-] URI qemu:///system does not support events: Cannot write data: Broken pipe
2014-10-19 01:50:15.147 2993 ERROR nova.openstack.common.threadgroup [-] internal error: client socket is closed
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 117, in wait
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup x.wait()
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 49, in wait
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 483, in run_servic
e
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup service.start()
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/service.py", line 163, in start
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1030, in init_host
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup self.driver.init_host(host=self.host)
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 658, in init_host
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup self._do_quality_warnings()
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 641, in _do_quality_warnings
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup caps = self.get_host_capabilities()
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2870, in get_host_capabilities
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup xmlstr = self._conn.getCapabilities()
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup rv = execute(f,*args,**kwargs)
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup rv = meth(*args,**kwargs)
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3303, in getCapabilities
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup if ret is None: raise libvirtError ('virConnectGetCapabilities() failed', conn=self)
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup libvirtError: internal error: client socket is closed
2014-10-19 01:50:15.147 2993 TRACE nova.openstack.common.threadgroup

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ante Karamatić (ivoks)]

What I believe it's happening here is that libvirt declares itself as started, but it's not functional yet. Maybe nova-compute should do some while looping in upstart's pre-start? :)

[James Page (james-page)]

Hmm - a tricky race and one I think we have seen with other tools that use libvirt; I think the latest nova code is a little more intelligent and will try a few times and mark itself disabled in the event that the connection can't be established, rather than shutting down.