Nova fails to start : [libvirt] cannot list SASL mechanisms

[Bastien]

Hello,

I'm trying to deploy Ussuri in all-in-one mode, on a Centos8 host. The installation is based on the latest stable Ussuri version for Kolla https://opendev.org/openstack/kolla-ansible/src/branch/stable/ussuri.

During the deployment, the following steps fails :

TASK [nova-cell : Waiting for nova-compute services to register themselves] ********************************************************************************
FAILED - RETRYING: Waiting for nova-compute services to register themselves (20 retries left).
...
FAILED - RETRYING: Waiting for nova-compute services to register themselves (1 retries left).
ok: [localhost -> localhost]

TASK [nova-cell : Fail if nova-compute service failed to register] *****************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "The Nova compute service failed to register itself on the following hosts: my-host.lab"}

Looking at the logs, nova is not able to authenticate, and libvirt is failing :

[ /var/log/kolla/nova/nova-compute.log ]
2022-09-26 09:35:45.648 7 ERROR nova.virt.libvirt.host [-] Connection to libvirt failed: authentication failed: authentication failed: libvirt.libvirtError: authentication failed: authentication failed

[ /var/log/kolla/libvirt/libvirtd.log ]
2022-09-26 09:35:45.648+0000: 1818403: info : libvirt version: 7.6.0, package: 6.el8 (CBS <email address hidden>, 2021-11-17-01:57:30, )
2022-09-26 09:35:45.648+0000: 1818403: info : hostname: my-host.lab
2022-09-26 09:35:45.648+0000: 1818403: error : virNetSASLSessionListMechanisms:400 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4)
2022-09-26 09:35:45.648+0000: 1818403: error : remoteDispatchAuthSaslInit:3609 : authentication failed: authentication failed
2022-09-26 09:35:45.648+0000: 1818378: error : virNetSocketReadWire:1804 : End of file while reading data: Input/output error
2022-09-26 09:35:49.886+0000: 1818405: error : virNetSASLSessionListMechanisms:400 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4)
2022-09-26 09:35:49.886+0000: 1818405: error : remoteDispatchAuthSaslInit:3609 : authentication failed: authentication failed

The configured mech_list for SASL is :

[ /etc/kolla/nova-libvirt/sasl.conf ]
mech_list: DIGEST-MD5

Does anyone know what is the fix to the issue ? Thanks

* OS (e.g. from /etc/os-release): (Host) CentOS Stream release 8, the deployed containers use CentOS Linux release 8.5.2111
* Kernel (e.g. `uname -a`): 4.18.0-408.el8.x86_64
* Docker version if applicable (e.g. `docker version`): 20.10.18
* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release): Ussuri from stable branch
* Docker image Install type (source/binary): binary
* Docker image distribution: Ussuri
* Are you using official images from Docker Hub or self built? Docker Hub
* Share your inventory file, globals.yml and other configuration files if relevant

[Tino Schmeier (tis-x)]

Hi, i had the same problem. In my case the problems was the fqdn-hostname in /etc/hosts. After remove the fqdn in the corresponding line for host them selfs and only leave the hostname in there, i restarted nova_libvirt and nova_compute was able to connect to libvirt. You can also append the inventory_hostname to libvirt_sasl_authname but thats will prevent live-migration to work.

Regards
Tino

[Bastien (bclasse)]

Hi Tino, thanks, /etc/hosts was not containing any FQDN before the deployment unfortunately.

On the other hand, I found that libvirt is relying on cyrus-sasl to operate.
On the current image, the following packages are installed :

sudo docker run kolla/centos-binary-nova-compute:ussuri dnf list --installed | grep cyrus
cyrus-sasl.x86_64 2.1.27-5.el8 @baseos
cyrus-sasl-gssapi.x86_64 2.1.27-5.el8 @baseos
cyrus-sasl-lib.x86_64 2.1.27-5.el8 @System

To work with DIGEST-MD5, the image would need to also install cyrus-sasl-md5.
Maybe MD5 is no longer supported for weak encryption reasons, but the documentation lets believe it is : https://docs.openstack.org/kolla-ansible/ussuri/reference/compute/libvirt-guide.html
Any comment on this would be welcome

Regards
Bastien

[Tino Schmeier (tis-x)]

I'm using ubuntu-source-images and they contain the required libs:

ii libsasl2-2:amd64 2.1.27+dfsg-2ubuntu0.1 amd64 Cyrus SASL - authentication abstraction library
ii libsasl2-modules:amd64 2.1.27+dfsg-2ubuntu0.1 amd64 Cyrus SASL - pluggable authentication modules
ii libsasl2-modules-db:amd64 2.1.27+dfsg-2ubuntu0.1 amd64 Cyrus SASL - pluggable authentication modules (DB)
ii libsasl2-modules-gssapi-mit:amd64 2.1.27+dfsg-2ubuntu0.1 amd64 Cyrus SASL - pluggable authentication modules (GSSAPI)
ii sasl2-bin 2.1.27+dfsg-2ubuntu0.1 amd64 Cyrus SASL - administration programs for SASL users database

Maybe you can try to overwrite the image for nova_libvirt and use the ubuntu-source-image

[Bastien (bclasse)]

The Ussuri docker images have not been rebuilt for 8 months, and it turns out the support for MD5 in kolla is more recent than that.
Rebuilding those images following the admin guide fixed the issue, the deployment is now successful.

[Radosław Piliszek (yoctozepto)]

Ussuri is in Extended Maintenance so its images are not published. We assume that users of old releases already follow the best practices of building images for themselves by themselves so it's not an issue.