nova-compute don't start in grenade job due to missing libvirt module

[Slawek Kaplonski]

I found that sometimes grenade jobs are failing and reason of that failure is that nova-compute is not started.
Error from nova-compute log:

Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver [None req-8a6cbe05-4b28-49b2-866b-9a35b1eb785d None None] Unable to load the virtualization driver: ModuleNotFoundError: No module named 'libvirt'
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver Traceback (most recent call last):
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver File "/opt/stack/old/nova/nova/virt/driver.py", line 1747, in load_compute_driver
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver virtapi)
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver File "/usr/local/lib/python3.6/dist-packages/oslo_utils/importutils.py", line 44, in import_object
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver return import_class(import_str)(*args, **kwargs)
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 320, in __init__
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver libvirt = importutils.import_module('libvirt')
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver File "/usr/local/lib/python3.6/dist-packages/oslo_utils/importutils.py", line 73, in import_module
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver __import__(import_str)
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver ModuleNotFoundError: No module named 'libvirt'
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: /usr/local/lib/python3.6/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: """)
Mar 29 14:04:39.264259 ubuntu-bionic-rax-dfw-0004533915 systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Mar 29 14:04:39.264289 ubuntu-bionic-rax-dfw-0004533915 systemd[1]: <email address hidden>: Failed with result 'exit-code'.

It is here: http://logs.openstack.org/45/638645/11/gate/neutron-grenade-dvr-multinode/4064577/logs/screen-n-cpu.txt.gz

Logstash query which I used to find such examples: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ModuleNotFoundError%3A%20No%20module%20named%20'libvirt'%5C%22

It happened more than 10 times in last 7 days so it is quite frequent issue.

[Slawek Kaplonski (slaweq)]

It may be that it is similar (same) issue like in https://bugs.launchpad.net/keystone/+bug/1822251 but I'm not sure so I reported this agains nova as separate bug for now.

[Matt Riedemann (mriedem)]

Is neutron-grenade-dvr-multinode running with bionic and py36? If so it might be a duplicate of bug 1820892.

[Matt Riedemann (mriedem)]

Yeah it's the same problem as bug 1820892, libvirt-python is installed on py27 rather than py36:

http://logs.openstack.org/45/638645/11/gate/neutron-grenade-dvr-multinode/4064577/logs/pip2-freeze.txt.gz

libvirt-python==4.6.0

[Slawek Kaplonski (slaweq)]

@Matt: thx for checking that :)