nova-compute fails to auth to libvirtd when `hostname` != `hostname -f`

[Radosław Piliszek]

$ cat /etc/hosts (or from the DNS)
1.2.3.4 hostname.domainname hostname

$ cat /etc/hostname
hostname

$ hostname
hostname

$ hostname -s
hostname

$ hostname -f
hostname.domainname

nova-compute will log:
2022-09-15 18:45:48.608 7 ERROR nova.virt.libvirt.host [-] Connection to libvirt failed: authentication failed: authentication failed: libvirt.libvirtError: authentication failed: authentication failed

(fun fact: the error message says "authentication failed" 4 times - 4 times! - must be a serious violation ;-) )

libvirtd will log:
2022-09-15 16:45:48.606+0000: 3900339: error : virNetSASLSessionServerStep:594 : authentication failed: Failed to start SASL negotiation: -20 (SASL(-13): user not found: unable to canonify user and get auxprops)
2022-09-15 16:45:48.606+0000: 3900339: error : remoteDispatchAuthSaslStep:3985 : authentication failed: authentication failed

The workaround is to modify /etc/hosts (or set for the first time if the original source is DNS) such that the FQDN becomes the alias:

$ cat /etc/hosts
1.2.3.4 hostname hostname.domainname

and then

$ hostname -f
hostname

and then restart nova_libvirtd (probably no need to restart nova_compute but it keeps restarting in a loop so hard to tell ;-) ).

BUT BEWARE!
If this hypervisor existed before, it will remain under the old FQDN name in Placement and previously running instances will not count against the usage!
TODO: report that bug separately

Current workaround to the workaround: shelve and unshelve the instances (though note this will obviously stop and restart them in the process).

[Radosław Piliszek (yoctozepto)]

CI reproducer added: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/858142

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/915560

[Przemysław Kuczyński (przemekkuczynski)]

workaround is also run bootstrap so it will create hosts entry

main.yml
customize_etc_hosts: true