live migration in tls-everywhere scenario fails with:
2020-09-17 15:44:09.067 8 ERROR nova.virt.libvirt.driver [req-b515c1c4-f4e0-4a0d-a240-19b2b96e71a9 3452fb519c0e4113afbb5e7da113cdbe e46388fad64144cf9417a2a7f34d4c75 - default default] [instance: aff4e47c-ab7d-4473-af6c-fe5c646335f3] Migration operation has aborted
2020-09-17 15:44:09.338 8 ERROR nova.virt.libvirt.driver [req-b515c1c4-f4e0-4a0d-a240-19b2b96e71a9 3452fb519c0e4113afbb5e7da113cdbe e46388fad64144cf9417a2a7f34d4c75 - default default] [instance: aff4e47c-ab7d-4473-af6c-fe5c646335f3] Live Migration failure: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/qemu/ca-cert.pem: No such file or directory: libvirtError: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/qemu/ca-cert.pem: No such file or directory
The issue is that the certificates for the tls nbd block migration get created during the update.
They did not exist in the libvirtd container when the existing instances were created. During
libvirt container create the certificates get merged into the container directory tree using the
kolla_config mechanism. They are not a direct bind mount from the host. Therefor the qemu
processes of the existing instances don't have that information and the nbd setup process
fails with the seen error, which we can also confirm when strace a qemu process of an instance
created before the update during a live migrate:
116406 stat("/etc/pki/qemu/ca-cert.pem", 0x7fff6f4ec390) = -1 ENOENT (No such file or directory)
116406 sendmsg(25, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="{\"id\": \"libvirt-2611\", \"error\": {\"class\": \"GenericError\", \"desc\": \"Unable to access credentials /etc/pki/qemu/ca-cert.pem: No such file or directory\"}}\r\n", iov_len=153}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 153
The immediate solution is to run an overcloud deploy and specify not to use TLS transport for
nbd, which configures the same configuration as before the minor update:
parameter_defaults:
UseTLSTransportForNbd: False
For a transition to use UseTLSTransportForNbd: true, we need enhance the THT to support the
following transition path:
1) create the required nbd certificates also with "UseTLSTransportForNbd: False", or use bind
mounts for the cert directories instead of merging them into the directory tree on container
create. This would also have the benefit that there is no action required when the nbd certs
change.
2) all instances need to be migrated once that qemu process runs with an environment which has
all the certificate information
3) enable "UseTLSTransportForNbd: True" for the overcloud deployment
After that all instances have the required information to do live migration with
"UseTLSTransportForNbd: True".
Related fix proposed to branch: master
Review: https:/