Logs from rhel node, /var/log/messages: Feb 1 08:02:59 rhel-1 nova-compute: 2016-02-01 08:02:59.749 2637 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt' Feb 1 08:03:00 rhel-1 nova-compute: 2016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host [req-8d03d226-9d24-4ecb-af6a-3585b750f2f9 - - - - -] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host Traceback (most recent call last):#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 528, in get_connection#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host conn = self._get_connection()#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 515, in _get_connection#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host wrapped_conn = self._get_new_connection()#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 467, in _get_new_connection#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only)#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 321, in _connect#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags)#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs)#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host six.reraise(c, e, tb)#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs)#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed')#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory#0122016-02-01 08:03:00.432 2637 ERROR nova.virt.libvirt.host Feb 1 08:03:00 rhel-1 nova-compute: Traceback (most recent call last): Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/queue.py", line 117, in switch Feb 1 08:03:00 rhel-1 nova-compute: self.greenlet.switch(value) Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main Feb 1 08:03:00 rhel-1 nova-compute: result = function(*args, **kwargs) Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 645, in run_service Feb 1 08:03:00 rhel-1 nova-compute: service.start() Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start Feb 1 08:03:00 rhel-1 nova-compute: self.manager.init_host() Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1297, in init_host Feb 1 08:03:00 rhel-1 nova-compute: self.driver.init_host(host=self.host) Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 563, in init_host Feb 1 08:03:00 rhel-1 nova-compute: self._do_quality_warnings() Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 541, in _do_quality_warnings Feb 1 08:03:00 rhel-1 nova-compute: caps = self._host.get_capabilities() Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 765, in get_capabilities Feb 1 08:03:00 rhel-1 nova-compute: xmlstr = self.get_connection().getCapabilities() Feb 1 08:03:00 rhel-1 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 537, in get_connection Feb 1 08:03:00 rhel-1 nova-compute: raise exception.HypervisorUnavailable(host=CONF.host) Feb 1 08:03:00 rhel-1 nova-compute: HypervisorUnavailable: Connection to the hypervisor is broken on host: rhel-1.test.domain.local Feb 1 08:03:37 rhel-1 neutron-openvswitch-agent: 2016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 331, in _report_state#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True)#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 86, in report_state#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs)#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry)#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent timeout=timeout, retry=retry)#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 533, in send#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 522, in _send#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = self._waiter.wait(msg_id, timeout)#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 397, in wait#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout)#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 296, in get#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id)#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID 58fe5739c6f543d4b8502be04d309a30#0122016-02-01 08:03:37.611 2636 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Feb 1 08:03:37 rhel-1 neutron-openvswitch-agent: 2016-02-01 08:03:37.614 2636 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 616.77 sec Feb 1 08:03:37 rhel-1 neutron-openvswitch-agent: 2016-02-01 08:03:37.748 2636 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Agent has just been revived. Doing a full sync. Feb 1 08:03:37 rhel-1 neutron-openvswitch-agent: 2016-02-01 08:03:37.848 2636 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e83b931f-0984-489e-a888-ac925fdb03d2 - - - - -] rpc_loop doing a full sync. Feb 1 08:03:37 rhel-1 neutron-openvswitch-agent: 2016-02-01 08:03:37.849 2636 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e83b931f-0984-489e-a888-ac925fdb03d2 - - - - -] Agent out of sync with plugin! Feb 1 08:03:37 rhel-1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-port br-tun vxlan-0a6d0901 -- set Interface vxlan-0a6d0901 type=vxlan options:df_default=true options:remote_ip=10.109.9.1 options:local_ip=10.109.9.3 options:in_key=flow options:out_key=flow Feb 1 08:03:37 rhel-1 neutron-openvswitch-agent: 2016-02-01 08:03:37.896 2636 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e83b931f-0984-489e-a888-ac925fdb03d2 - - - - -] Agent tunnel out of sync with plugin! Feb 1 08:03:38 rhel-1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-port br-tun vxlan-0a6d0901 -- set Interface vxlan-0a6d0901 type=vxlan options:df_default=true options:remote_ip=10.109.9.1 options:local_ip=10.109.9.3 options:in_key=flow options:out_key=flow Feb 1 08:03:38 rhel-1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-port br-tun vxlan-0a6d0902 -- set Interface vxlan-0a6d0902 type=vxlan options:df_default=true options:remote_ip=10.109.9.2 options:local_ip=10.109.9.3 options:in_key=flow options:out_key=flow Feb 1 08:03:38 rhel-1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-port br-tun vxlan-0a6d0904 -- set Interface vxlan-0a6d0904 type=vxlan options:df_default=true options:remote_ip=10.109.9.4 options:local_ip=10.109.9.3 options:in_key=flow options:out_key=flow Feb 1 08:03:39 rhel-1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-port br-tun vxlan-0a6d0902 -- set Interface vxlan-0a6d0902 type=vxlan options:df_default=true options:remote_ip=10.109.9.2 options:local_ip=10.109.9.3 options:in_key=flow options:out_key=flow Feb 1 08:03:42 rhel-1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-port br-tun vxlan-0a6d0904 -- set Interface vxlan-0a6d0904 type=vxlan options:df_default=true options:remote_ip=10.109.9.4 options:local_ip=10.109.9.3 options:in_key=flow options:out_key=flow Feb 1 08:08:01 rhel-1 systemd: Starting Cleanup of Temporary Directories... Feb 1 08:08:01 rhel-1 systemd: Started Cleanup of Temporary Directories.