nova_compute can not talk to nova_libvirt

[Hui Kang]

The latest HEAD of git master seems to broken for nova-compute. The error message is

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/queue.py", line 117, in switch
    self.greenlet.switch(value)
  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 645, in run_service
    service.start()
  File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start
    self.manager.init_host()
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1296, in init_host
    self.driver.init_host(host=self.host)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 562, in init_host
    self._do_quality_warnings()
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 540, in _do_quality_warnings
    caps = self._host.get_capabilities()
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 765, in get_capabilities
    xmlstr = self.get_connection().getCapabilities()
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 537, in get_connection
    raise exception.HypervisorUnavailable(host=CONF.host)
HypervisorUnavailable: Connection to the hypervisor is broken on host:

The latest success build is 6316279278dc0d71e570f1e1dc989733f56a6c9a. So did this a02bcdcacc593d709f7a25b729d931f6f8708dc1 (https://github.com/openstack/kolla/commit/a02bcdcacc593d709f7a25b729d931f6f8708dc1) break the nova-compute. Thanks.

- Hui

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/260352

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/260352
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=b54420442ef1a4d7a39e4f868a8a8d76e593f510
Submitter: Jenkins
Branch: master

commit b54420442ef1a4d7a39e4f868a8a8d76e593f510
Author: Kuo-tung Kao <email address hidden>
Date: Tue Dec 22 14:48:24 2015 +0800

    change libvirt sock group to nova

    Add `nova` user to nova-libvirt container.
    And change libvirt-socket group to nova.

    Change-Id: I183c83f4be8b1d7c75d4ac204df7b7e059626aa2
    Closes-Bug: #1525583

[Artur Zarzycki (azarzycki)]

Previous patch doesn't resolve everything. I tried to find where is the problem.
The error message:
 HypervisorUnavailable: Connection to the hypervisor is broken on host
still exist and cover real error which is:
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host [req-8f509d52-221c-45fd-b4a6-807cf560c249 - - - - -] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host Traceback (most recent call last):
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 528, in get_connection
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host conn = self._get_connection()
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 515, in _get_connection
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host wrapped_conn = self._get_new_connection()
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 467, in _get_new_connection
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only)
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 321, in _connect
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags)
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs)
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host six.reraise(c, e, tb)
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs)
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed')
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2015-12-30 09:36:49.105 1 ERROR nova.virt.libvirt.host

The main problem is that nova_compute is started from user nova and /var/run/libvirt/libvirt-sock is mounted from nova-libvirt to nova-compute with root permissions. The solution with chown doesn't work because user nova have other uid/gid on both containers and it answer error because of unknown uid/gid.
So after some discussion with Michal we agreed that the best temporary solution (until it'll be resolved with polkit) will be to use solution proposed by Sam in this CR https://review.openstack.org/#/c/261842/1 (this error e...

[Artur Zarzycki (azarzycki)]

Fixed by https://review.openstack.org/#/c/261842/