libvirtd failed to open the RBD image

[macchese]

using libvirtd with a ceph pool, when i iusse the command
virsh vol-list $POOLNAME
 it takes minutes to respond and into the syslog I have a list of errors like this

Sep 12 15:06:28 op3 libvirtd[6987]: failed to open the RBD image 'arss-sys': No such file or directory
Sep 12 15:06:58 op3 libvirtd[6987]: failed to open the RBD image 'A2-wp-staging': No such file or directory
Sep 12 15:07:28 op3 libvirtd[6987]: failed to open the RBD image 'FS-L': No such file or directory
Sep 12 15:14:40 op3 libvirtd[6987]: failed to open the RBD image 'DocsPA-share-C': No such file or directory
Sep 12 15:15:10 op3 libvirtd[6987]: failed to open the RBD image 'A2_ST_MIUR': No such file or directory
Sep 12 15:15:44 op3 libvirtd[6987]: failed to open the RBD image 'ndc01-1': No such file or directory
Sep 12 15:16:16 op3 libvirtd[6987]: failed to open the RBD image 'arss-sys': No such file or directory

but every image is present, for example

rbd ls SAS|grep arss-sys
arss-sys

Every failed image is in use by another server via kernel rbd module (rbd map).

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal

dpkg -l |egrep "libvirt|ceph"
ii ceph-common 15.2.17-0ubuntu0.20.04.4 amd64 common utilities to mount and interact with a ceph storage cluster
ii gir1.2-libvirt-glib-1.0:amd64 3.0.0-1 amd64 GObject introspection files for the libvirt-glib library
ii libcephfs2 15.2.17-0ubuntu0.20.04.4 amd64 Ceph distributed file system client library
ii libvirt-clients 6.0.0-0ubuntu8.16 amd64 Programs for the libvirt library
ii libvirt-daemon 6.0.0-0ubuntu8.16 amd64 Virtualization daemon
ii libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.16 amd64 Virtualization daemon QEMU connection driver
ii libvirt-daemon-driver-storage-rbd 6.0.0-0ubuntu8.16 amd64 Virtualization daemon RBD storage driver
ii libvirt-daemon-system 6.0.0-0ubuntu8.16 amd64 Libvirt daemon configuration files
ii libvirt-daemon-system-systemd 6.0.0-0ubuntu8.16 amd64 Libvirt daemon configuration files (systemd)
ii libvirt-glib-1.0-0:amd64 3.0.0-1 amd64 libvirt GLib and GObject mapping library
ii libvirt0:amd64 6.0.0-0ubuntu8.16 amd64 library for interfacing with different virtualization systems
ii python3-ceph-argparse 15.2.17-0ubuntu0.20.04.4 amd64 Python 3 utility libraries for Ceph CLI
ii python3-ceph-common 15.2.17-0ubuntu0.20.04.4 all Python 3 utility libraries for Ceph
ii python3-cephfs 15.2.17-0ubuntu0.20.04.4 amd64 Python 3 libraries for the Ceph libcephfs library
ii python3-libvirt 6.1.0-1 amd64 libvirt Python 3 bindings

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Libera.chat.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/2035225/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Christian Ehrhardt  (paelzer)]

Hi,
just saying "I use ceph and is slow" is unfortunately not much one can act one.
There are plenty of ceph setups in our openstack deployments not being affected by this, so it is not "slow in general".

I subscribed James Page (feel free to reassign) as an expert on how our OpenStack uses Ceph to compare what you might describe.

So I assume to be able to begin - it comes down to identify what is different in your setup.
Which starts at describing how you set it up and hopefully our ceph people spot a difference that can be tried by you to see if it changes things. Once that way we find what makes the difference it can either be debugged and fixed (if it is a bug) or just be configured differently (if it is a config issue).

[macchese (max-liccardo)]

well,
I try to describe in depth my setup.
I have a 4-node nautilus ceph on ubuntu 18.04-LTS and 2 other ubuntu server for VMs (op1 and op3).
op1 is an ubuntu 18.04-lts using virsh/rbdmap for vm disks while op3 is an ubuntu 20.04-lts and uses virsh/libceph (by libvirt ceph pool).
The ceph pool is the same for the 2 servers op1 and op3.
The problem I noted is always on op3 where for example It takes quite 5 minutes to have the vol list from the pool (virsh vol-list POOLNAME).
If I use virt-manager the ssh connection (qemu+ssh) takes always a lot of minutes to complete while the logs are full of

Sep 12 15:15:10 op3 libvirtd[6987]: failed to open the RBD image 'A2_ST_MIUR': No such file or directory
Sep 12 15:15:44 op3 libvirtd[6987]: failed to open the RBD image 'ndc01-1': No such file or directory
Sep 12 15:16:16 op3 libvirtd[6987]: failed to open the RBD image 'arss-sys': No such file or directory

I suppose that virt-manager try to obtain the volumes list of the pool, but it takes a lot of time and sometimes doesn't find a lot of volumes that exist but are in use by the op1 server using rbdmap.

[Launchpad Janitor (janitor)]

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for ceph (Ubuntu) because there has been no activity for 60 days.]