Found while analyze system tests failure. Fuel 6.1 iso # 114
Steps to reproduce:
1. Create cluster
2. Add 1 node with controller role
3. Add 2 nodes with compute role
4. Deploy the cluster
5. Run OSTF
Expected result:
OSTF passed
Actual result:
OSTF fails. In logs:
ERROR (nose_storage_plugin) fuel_health.tests.sanity.test_sanity_infrastructure.SanityInfrastructureTest.test_001_services_state
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/unittest2/case.py", line 340, in run
testMethod()
File "/usr/lib/python2.6/site-packages/fuel_health/tests/sanity/test_sanity_infrastructure.py", line 81, in test_001_services_state
u'XXX' not in output, 'Step 2 failed: Some nova services '
File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 167, in verify_response_true
self.fail(message.format(failed_step_msg, msg))
File "/usr/lib/python2.6/site-packages/unittest2/case.py", line 415, in fail
raise self.failureException(msg)
AssertionError: Step 2 failed: Some nova services have not been started.. Please refer to OpenStack logs for more details.
At nova-compute log on node-3:
2015-02-11 22:08:37.138 29698 DEBUG nova.virt.libvirt.driver [-] Connecting to libvirt: qemu:///system _get_new_connection /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:724
2015-02-11 22:08:37.152 29698 ERROR nova.virt.libvirt.driver [-] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver Traceback (most recent call last):
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 839, in _connect
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver libvirt.openAuth, uri, auth, flags)
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver rv = execute(f, *args, **kwargs)
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver six.reraise(c, e, tb)
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver rv = meth(*args, **kwargs)
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver if ret is None:raise libvirtError('virConnectOpenAuth() failed')
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2015-02-11 22:08:37.152 29698 TRACE nova.virt.libvirt.driver
2015-02-11 22:08:37.236 29698 WARNING nova.virt.libvirt.driver [-] Cannot update service status on host: node-3.test.domain.local,since it is not registered.
2015-02-11 22:08:37.243 29698 ERROR nova.openstack.common.threadgroup [-] Connection to the hypervisor is broken on host: node-3.test.domain.local
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 125, in wait
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup x.wait()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 173, in wait
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 293, in switch
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 490, in run_service
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup service.start()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/service.py", line 164, in start
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1125, in init_host
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup self.driver.init_host(host=self.host)
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 696, in init_host
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup self._do_quality_warnings()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 679, in _do_quality_warnings
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup caps = self._get_host_capabilities()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3301, in _get_host_capabilities
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup xmlstr = self._conn.getCapabilities()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 775, in _get_connection
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup wrapped_conn = self._get_new_connection()
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 728, in _get_new_connection
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup wrapped_conn = self._connect(self.uri(), self.read_only)
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 848, in _connect
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup raise exception.HypervisorUnavailable(host=CONF.host)
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup HypervisorUnavailable: Connection to the hypervisor is broken on host: node-3.test.domain.local
2015-02-11 22:08:37.243 29698 TRACE nova.openstack.common.threadgroup
We have not experienced such an issues for more than a month. Seems more like a hypervisor misconfiguration on the host, e.g. nested virt turned off. Marking this bug as Invalid. If you think, that this is not right, please reopen the bug after you reproduce it.