Recent qemu packages for Ubuntu 22.04 missing rbd support

[Linh Vu]

As of 2024-05-01, the recent qemu package versions for Ubuntu 22.04 LTS are missing ceph rbd support. The affected versions are:

* 1:6.2+dfsg-2ubuntu6.18
* 1:6.2+dfsg-2ubuntu6.19

The earlier versions, from 1:6.2+dfsg-2ubuntu6.17 and older, are not affected.

With the affected versions, when launching either a boot-from-ceph-volume VM or attaching a cinder volume with ceph rbd backend to a VM, we would get an error like this:

```
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [req-5772bdba-919a-4d9e-9ec2-49793e3205c5 f432a17f4e154e238f22ac7f6158dc2a 0735ccd91ee94b8b820e87d279d8cb7f - default default] [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] Failed to build and run instance: libvirt.libvirtError: internal error: process exited while connecting to monitor: qemu: module block-block-rbd not found, do you want to install qemu-block-extra package?
2024-04-19T04:20:46.209432Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"volumes","image":"volume-b363c97f-2865-46e9-baa5-72b966e1d8c5","server":[{"host":"<IP_ADDRESS>","port":"6789"},{"host":"<IP_ADDRESS>","port":"6789"},{"host":"<IP_ADDRESS>","port":"6789"}],"user":"openstack","auth-client-required":["cephx","none"],"key-secret":"libvirt-2-storage-auth-secret0","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: Unknown driver 'rbd'
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] Traceback (most recent call last):
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/nova/compute/manager.py", line 2412, in _build_and_run_instance
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] self.driver.spawn(context, instance, image_meta,
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/nova/virt/libvirt/driver.py", line 4226, in spawn
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] self._create_guest_with_network(
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/nova/virt/libvirt/driver.py", line 7300, in _create_guest_with_network
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] self._cleanup(
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] self.force_reraise()
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] raise self.value
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/nova/virt/libvirt/driver.py", line 7269, in _create_guest_with_network
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] guest = self._create_guest(
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/nova/virt/libvirt/driver.py", line 7209, in _create_guest
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] guest.launch(pause=pause)
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/nova/virt/libvirt/guest.py", line 168, in launch
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] LOG.exception('Error launching a defined domain with XML: %s',
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] self.force_reraise()
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] raise self.value
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/nova/virt/libvirt/guest.py", line 165, in launch
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] return self._domain.createWithFlags(flags)
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 193, in doit
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] result = proxy_call(self._autowrap, f, *args, **kwargs)
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 151, in proxy_call
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] rv = execute(f, *args, **kwargs)
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132, in execute
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] six.reraise(c, e, tb)
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] raise value
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86, in tworker
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] rv = meth(*args, **kwargs)
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] File "/usr/lib/python3/dist-packages/libvirt.py", line 1398, in createWithFlags
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] raise libvirtError('virDomainCreateWithFlags() failed')
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] libvirt.libvirtError: internal error: process exited while connecting to monitor: qemu: module block-block-rbd not found, do you want to install qemu-block-extra package?
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e] 2024-04-19T04:20:46.209432Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"volumes","image":"volume-b363c97f-2865-46e9-baa5-72b966e1d8c5","server":[{"host":"<IP_ADDRESS>","port":"6789"},{"host":"<IP_ADDRESS>","port":"6789"},{"host":"<IP_ADDRESS>","port":"6789"}],"user":"openstack","auth-client-required":["cephx","none"],"key-secret":"libvirt-2-storage-auth-secret0","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: Unknown driver 'rbd'
2024-04-19 04:20:52.493 22 ERROR nova.compute.manager [instance: e27bdd6e-c1fb-4e7d-8048-26fab82d802e]
```

Our current workaround is to downgrade or hold qemu packages at this version 1:6.2+dfsg-2ubuntu6.16 which is our most trusted / tested one.

[Sebastien Bacher (seb128)]

Thank you for your bug report. It's weird that 6.18 would add that regression seeing the changes in the update

http://launchpadlibrarian.net/715512558/qemu_1%3A6.2+dfsg-2ubuntu6.17_1%3A6.2+dfsg-2ubuntu6.18.diff.gz

Did you correctly install the corresponding qemu-block-extra package?

[Linh Vu (linhgb)]

Hi Sebastien,

It appears to me now, after you brought that up, is that with the older version:

Package: qemu-system-x86
Version: 1:6.2+dfsg-2ubuntu6.16

when I install that, it would also install the recommended package qemu-block-extra=1:6.2+dfsg-2ubuntu6.16

However, when I install

Package: qemu-system-x86
Version: 1:6.2+dfsg-2ubuntu6.18 or 1:6.2+dfsg-2ubuntu6.19

it does not install the recommended package qemu-block-extra.

This is a new behaviour, a bit unexpected, I think. I'll just fix our playbooks to specifically install qemu-block-extra as well.

Thank you for the suggestion!

[Sergio Durigan Junior (sergiodj)]

Hi Linh,

Thanks for the bug report.

It's strange that you're noticing this discrepancy in behaviour between minor QEMU versions. I just double checked and confirmed that there were no changes to Depends/Recommends relationships between the versions you mentioned.

I also launched a LXD container here and installed qemu-system-x86 1:6.2+dfsg-2ubuntu6.19. I can confirm that qemu-block-extra is automatically installed as well.

Given that I can't reproduce the problem, I am marking this bug as Incomplete. Can you please provide more information regarding how to reproduce it?

Thank you in advance.