Instance failed to spawn: libvirtError: invalid argument: could not find capabilities for domaintype=kvm

[Scott Beck]

I have two servers I'm using to deploy as virtual hosts. Both have worked with Stein release before. I'm rolling out Train release as a fresh install and I'm stuck with libvirt errors about not having KVM capabilities.

The libvirt error is:
2019-11-17 21:51:45.297+0000: 23302: error : virCapabilitiesDomainDataLookupInternal:750 : invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm
2019-11-17 21:51:45.297+0000: 23302: error : virCapabilitiesDomainDataLookupInternal:750 : invalid argument: could not find capabilities for domaintype=kvm

The nova error:
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [req-562eb7bc-0c28-4826-bc75-c4b9eb059f6d e59a4b4dd5a24d2b8a26014715ddca3c 342c25e5d07f452ea67ff26ef99c8261 - default default] [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] Instance failed to spawn: libvirtError: invalid argument: could not find capabilities for domaintype=kvm
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] Traceback (most recent call last):
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2546, in _build_resources
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] yield resources
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2322, in _build_and_run_instance
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] block_device_info=block_device_info)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3474, in spawn
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] power_on=power_on)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6251, in _create_domain_and_network
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] destroy_disks_on_failure)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] self.force_reraise()
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] six.reraise(self.type_, self.value, self.tb)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6220, in _create_domain_and_network
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] post_xml_callback=post_xml_callback)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6147, in _create_domain
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] guest = libvirt_guest.Guest.create(xml, self._host)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 128, in create
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] encodeutils.safe_decode(xml))
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] self.force_reraise()
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] six.reraise(self.type_, self.value, self.tb)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 124, in create
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] guest = host.write_instance_config(xml)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 1122, in write_instance_config
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] domain = self.get_connection().defineXML(xml)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 190, in doit
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] result = proxy_call(self._autowrap, f, *args, **kwargs)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] rv = execute(f, *args, **kwargs)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] six.reraise(c, e, tb)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] rv = meth(*args, **kwargs)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3743, in defineXML
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37] libvirtError: invalid argument: could not find capabilities for domaintype=kvm
2019-11-17 14:51:46.024 6 ERROR nova.compute.manager [instance: 6174d01a-b14e-4e15-a40c-f189e0c97c37]

I've attached a text file with info from one of the systems that is failing. Please let me know if you need any more info.

Thanks!
Scott

[Radosław Piliszek (yoctozepto)]

CentOS images on Ubuntu host may fail with low-level services like libvirt which depend on kernel details.

[Radosław Piliszek (yoctozepto)]

The on-host libvirt should be disabled, the one in the container is being used.

[Mark Goddard (mgoddard)]

I'd certainly be interested to see what would happen if you redeployed at least libvirt with an ubuntu container image.

[Scott Beck (scottbeck)]

the host libvirtd is disabled on all systems.

So I moved to using ubuntu container images and made some progress. The new error was:

2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [req-2314ed5f-cdb5-4eb5-bd6a-81465d9c77e9 303faf5ba7e14dfda39359834017141e 4b0d0861bbd74771ae693119c5c7265d - default default] [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] Instance failed to spawn: libvirt.libvirtError: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied
2019-11-18T18:04:53.369564Z qemu-system-x86_64: failed to initialize KVM: Permission denied
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] Traceback (most recent call last):
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2546, in _build_resources
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] yield resources
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2322, in _build_and_run_instance
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] block_device_info=block_device_info)
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3474, in spawn
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] power_on=power_on)
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6251, in _create_domain_and_network
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] destroy_disks_on_failure)
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] self.force_reraise()
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] six.reraise(self.type_, self.value, self.tb)
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] raise value
2019-11-18 11:04:54.152 6 ERROR nova.compute.manager [instance: 7582bd31-4f2e-4832-8036-efee02c1b0fd] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py"...

[Mark Goddard (mgoddard)]

I expect that making /dev/kvm world writable is a security hole. Still, it's useful to know where the problem lies. I think you're hitting https://bugs.launchpad.net/kolla/+bug/1681461, which has been around for a long time. It would be great if we could get it fixed.

The workaround used by Lev Klimin posted in that bug report is similar to yours, but probably a bit more secure:

KERNEL=="kvm", GROUP="42427", MODE="0660"

The 42427 comes from our static user/group IDs listed in kolla/common/config.py.

[Tom Fifield (fifieldt)]

It looks like the poster found a workaround, to use ubuntu hosts and kvm permissions. This appears to be out of Kolla Ansible's scope, so closing.