both nova-compute and nova-libvirt fails to deploy

[Taeha Kim]

Both nova-compute and nova-libvirt fails to deploy in RHEL. That is, both processes are restarted infinitely as follows.

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
28465dab7609 192.168.122.33:5000/kolla/rhel-source-nova-compute:ocata "kolla_start" 3 hours ago Restarting (0) 38 seconds ago nova_compute
c7790d396a0c 192.168.122.33:5000/kolla/rhel-source-nova-libvirt:ocata "kolla_start" 3 hours ago Restarting (1) 26 seconds ago nova_libvirt

while kolla-ansible deploy target nodes, the following error was happened and stopped.

TASK [nova : Waiting for nova-compute service up] **************************************************************
FAILED - RETRYING: Waiting for nova-compute service up (20 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (19 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (18 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (17 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (16 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (15 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (14 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (13 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (12 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (11 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (10 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (9 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (8 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (7 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (6 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (5 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (4 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (3 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (2 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (1 retries left).
fatal: [192.168.122.85 -> 192.168.122.85]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["docker", "exec", "kolla_toolbox", "openstack", "--os-interface", "internal", "--os-auth-url", "http://192.168.122.100:35357/v3", "--os-identity-api-version", "3", "--os-project-domain-name", "default", "--os-tenant-name", "admin", "--os-username", "admin", "--os-password", "password", "--os-user-domain-name", "default", "compute", "service", "list", "-f", "json", "--service", "nova-compute"], "delta": "0:00:01.457728", "end": "2017-08-09 16:32:52.373968", "failed": true, "rc": 0, "start": "2017-08-09 16:32:50.916240", "stderr": "", "stderr_lines": [], "stdout": "[]", "stdout_lines": ["[]"]}

NO MORE HOSTS LEFT *********************************************************************************************
 to retry, use: --limit @/usr/share/kolla-ansible/ansible/site.retry

I found also the previous reported issue at https://bugs.launchpad.net/kolla-ansible/+bug/1683527

But the previous is different from this bug because this bug has no error in nova-api.

Figure 1. nova-compute.log
Figure 2. nova-libvirt.log
--

Figure 1. nova-compute.log

2017-08-09 13:11:57.971 7 WARNING oslo_config.cfg [req-38833a25-e692-4d40-855d-884890fbc758 - - - - -] Option "notify_on_state_change" from group "DEFAULT" is deprecated. Use option "notify_on_state_change" from group "notifications".
2017-08-09 13:11:57.974 7 INFO nova.service [-] Starting compute node (version 15.0.5)
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host [-] Connection to libvirt failed: unable to connect to server at '192.168.122.85:16509': Connection refused
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host Traceback (most recent call last):
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 452, in get_connection
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host conn = self._get_connection()
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 435, in _get_connection
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host {'msg': ex})
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host self.force_reraise()
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host six.reraise(self.type_, self.value, self.tb)
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 424, in _get_connection
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host self._wrapped_conn = self._get_new_connection()
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 369, in _get_new_connection
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only)
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 225, in _connect
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags)
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs)
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host six.reraise(c, e, tb)
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs)
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed')
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host libvirtError: unable to connect to server at '192.168.122.85:16509': Connection refused
2017-08-09 13:11:57.987 7 ERROR nova.virt.libvirt.host
2017-08-09 13:11:57.993 7 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt: unable to connect to server at '192.168.122.85:16509': Connection refused'
2017-08-09 13:11:58.002 7 WARNING nova.virt.libvirt.driver [req-717e7223-b27d-453f-be85-583e7862c016 - - - - -] Cannot update service status on host "rhel-node1" since it is not registered.
2017-08-09 13:11:58.004 7 ERROR oslo_service.service [req-fe8cdff5-8708-4085-9173-1e6c3c5127f7 - - - - -] Error starting thread.
2017-08-09 13:11:58.004 7 ERROR oslo_service.service Traceback (most recent call last):
2017-08-09 13:11:58.004 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_service/service.py", line 722, in run_service
2017-08-09 13:11:58.004 7 ERROR oslo_service.service service.start()
2017-08-09 13:11:58.004 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/service.py", line 144, in start
2017-08-09 13:11:58.004 7 ERROR oslo_service.service self.manager.init_host()
2017-08-09 13:11:58.004 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 1138, in init_host
2017-08-09 13:11:58.004 7 ERROR oslo_service.service self.driver.init_host(host=self.host)
2017-08-09 13:11:58.004 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 494, in init_host
2017-08-09 13:11:58.004 7 ERROR oslo_service.service self._do_quality_warnings()
2017-08-09 13:11:58.004 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 471, in _do_quality_warnings
2017-08-09 13:11:58.004 7 ERROR oslo_service.service caps = self._host.get_capabilities()
2017-08-09 13:11:58.004 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 634, in get_capabilities
2017-08-09 13:11:58.004 7 ERROR oslo_service.service xmlstr = self.get_connection().getCapabilities()
2017-08-09 13:11:58.004 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 461, in get_connection
2017-08-09 13:11:58.004 7 ERROR oslo_service.service raise exception.HypervisorUnavailable(host=CONF.host)
2017-08-09 13:11:58.004 7 ERROR oslo_service.service HypervisorUnavailable: Connection to the hypervisor is broken on host: rhel-node1
2017-08-09 13:11:58.004 7 ERROR oslo_service.service

Figure 2. nova-libvirt.log

2017-08-09 04:12:40.951+0000: 4688: info : hostname: rhel-node1
2017-08-09 04:12:40.951+0000: 4688: error : virPidFileAcquirePath:422 : Failed to acquire pid file '/var/run/libvirtd.pid': Resource temporarily unavailable

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/492032

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/492033

[OpenStack Infra (hudson-openstack)]

Change abandoned by Taeha Kim (<email address hidden>) on branch: master
Review: https://review.openstack.org/492032
Reason: useless patch

[OpenStack Infra (hudson-openstack)]

Change abandoned by Taeha Kim (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/492033
Reason: useless patch