[extra-computes] Libvirt is down on Non-Ubuntu compute node after power cycle

[Timur Nurlygayanov]

Steps To Reproduce:
1. Deploy OpenStack cluster with 3 controllers and 1 compute node (Ubuntu)
2. Add RHEL-based or CentOS-based compute node to the cluster
3. Check the cluster health, it will work fine.
4. Shutdown non-Ubuntu compute node and power on this node
5. Check the hypervisor status and libvirt process status on compute node

Expected Result:
Compute node should work fine after power cycle.

Observed Result:
libvirt process is down, hyper visor status is "down":

[root@rhel-1 ~]# virsh list
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

We can see that libvirtd is enabled in autostart:
[root@rhel-1 ~]# chkconfig libvirtd
Note: Forwarding request to 'systemctl is-enabled libvirtd.service'.
enabled

[Timur Nurlygayanov (tnurlygayanov)]

Logs from rhel node, /var/log/messages:

Feb 1 08:02:59 rhel-1 nova-compute: 2016-02-01 08:02:59.749 2637 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt'
Feb 1 08:03:00 rhel-1 nova-compute: 2016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host [req-8d03d226-9d24-4ecb-af6a-3585b750f2f9 - - - - -] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host Traceback (most recent call last):#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 528, in get_connection#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host conn = self._get_connection()#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 515, in _get_connection#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host wrapped_conn = self._get_new_connection()#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 467, in _get_new_connection#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only)#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 321, in _connect#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags)#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs)#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host six.reraise(c, e, tb)#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs)#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed')#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host
Feb 1 08:03:00 rhel-1 nova-compute: Traceback (most recent call last):
Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/queue.py", line 117, in switch
Feb 1 08:03:00 rhel-1 nova-compute: self.greenlet.switch(value)
Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
Feb 1 08...

[Timur Nurlygayanov (tnurlygayanov)]

[root@rhel-1 ~]# libvirtd
2016-02-01 08:22:03.458+0000: 32122: info : libvirt version: 1.2.17, package: 13.el7_2.2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2015-11-23-07:46:04, x86-019.build.eng.bos.redhat.com)
2016-02-01 08:22:03.458+0000: 32122: warning : virDriverLoadModule:65 : Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not accessible
2016-02-01 08:22:03.508+0000: 32138: error : virSecurityDriverLookup:80 : unsupported configuration: Security driver selinux not enabled
2016-02-01 08:22:03.508+0000: 32138: error : qemuSecurityInit:453 : Failed to initialize security drivers
2016-02-01 08:22:03.508+0000: 32138: error : virStateInitialize:783 : Initialization of QEMU state driver failed: unsupported configuration: Security driver selinux not enabled
2016-02-01 08:22:03.508+0000: 32138: error : daemonRunStateInit:950 : Driver state initialization failed

[Sergey Kolekonov (skolekonov)]

The issue should be fixed by this patch https://review.openstack.org/#/c/270184/ as it's related to enabled selinux driver in libvirt while selinux itself was disabled. Newer images will have it enabled by default, also the patch will enable it if selinux was disabled initially.

[Ivan Berezovskiy (iberezovskiy)]

Victor Ryzhenkin is working on fix: patch https://review.openstack.org/#/c/270184/

[Roman Podoliaka (rpodolyaka)]

^ patches above are to fuel-qa repo and thus this must not be a part of 8.0 release artifacts

[Timur Nurlygayanov (tnurlygayanov)]

Sergey, it looks like we need to fix it in puppet manifests, not in image which we use for testing, because if real customer will reboot the server libvirt will not work again because of selinux.

[Timur Nurlygayanov (tnurlygayanov)]

The workaround:

Run on all compute nodes:
sed -i 's/SELINUX=disabled/SELINUX=enforcing/g' /etc/selinux/config
reboot now

[Victor Ryzhenkin (vryzhenkin)]

Hi! We changed our strategy about it. So, this was added to user guide. In tests we using images, which already have preconfigured SELinux. So, marking this one as invalid.