nova-compute fails to start when libvirtd connection is not available

[Jiri Suchomel]

It is possible that starting libvirtd takes a little more time and if nova-compute is being started at the same time, it just failes because it cannot create a connection to libvirtd.

It is reproducible by calling

systemctl restart openstack-nova-compute && systemctl restart libvirtd

It seems that while libvirtd is being restarted, nova tries to connect and fails. I think it would help to retry the connection several times to prevent such cases.

This seems to happen for Newton release, see packages bellow. I've just checked with Liberty and it is not possible to reproduce this behavior with Liberty. Is it caused by some thread-checking that was removed after Liberty?

# rpm -qa | grep nova
openstack-nova-12.0.6~a0~dev2-11.17.noarch
python-novaclient-2.30.1-12.1.noarch
openstack-nova-compute-12.0.6~a0~dev2-11.17.noarch
python-nova-12.0.6~a0~dev2-11.17.noarch

# rpm -qa | grep libvirt
libvirt-daemon-driver-secret-1.2.18.4-18.2.x86_64
libvirt-daemon-driver-lxc-1.2.18.4-18.2.x86_64
libvirt-client-1.2.18.4-18.2.x86_64
libvirt-daemon-driver-interface-1.2.18.4-18.2.x86_64
libvirt-python-1.2.18-1.2.x86_64
libvirt-daemon-1.2.18.4-18.2.x86_64
libvirt-daemon-driver-nodedev-1.2.18.4-18.2.x86_64
libvirt-daemon-config-nwfilter-1.2.18.4-18.2.x86_64
libvirt-1.2.18.4-18.2.x86_64
libvirt-daemon-driver-storage-1.2.18.4-18.2.x86_64
libvirt-daemon-driver-network-1.2.18.4-18.2.x86_64
libvirt-daemon-driver-qemu-1.2.18.4-18.2.x86_64
libvirt-daemon-driver-libxl-1.2.18.4-18.2.x86_64
^[[Alibvirt-daemon-driver-nwfilter-1.2.18.4-18.2.x86_64
libvirt-daemon-config-network-1.2.18.4-18.2.x86_64

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/416928

[Jiri Suchomel (jsuchome)]

Hi all.
Any chance to review proposed solution?

Thanks, Jiri

[OpenStack Infra (hudson-openstack)]

Change abandoned by Balazs Gibizer (<email address hidden>) on branch: master
Review: https://review.opendev.org/416928
Reason: The patch is stale. Feel free to restore it (or ping gibi on IRC to do so) if you are still working on this.

[Balazs Gibizer (balazs-gibizer)]

Behavior is still visible in current master. -> Confirmed