MAAS/Juju/OpenStack live-migration in multiple network spaces

[Rony Zeidan]

Hi,
i hope everyone is having a good day :).
I have an issue with an openstack deployment with multiple network spaces with regards to live-migration.
Live-migration is not working because of DNS entries in MAAS. It seems that maas dns is not handling dns records correctly please refer to https://bugs.launchpad.net/maas/+bug/1823183.

So whenever i try to live-migrate from host1 to host2 it is failing with the following error:
"Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://<HOST>/system: Cannot recv data: Host key verification failed.: Connection reset by peer"

i traced it down to the dns records in MAAS:
host1 is returning the IP address of the PXE network and the other networds are registered under bondx.x.host1 so therefore the ssh hand

i am missing something ?
I would really appreciate a way to get arround this and get live-migration working

Thx
RZ

[Chris MacNaughton (chris.macnaughton)]

I have also recently observed this change in how MAAS host addressing works. I'm not sure what the best solution will be to fix this issue is but there is definitely work to be done.

[Matthias (matthiashuether)]

I have the same problem and maybe I have a workaround:

If my interface is e.g. Eno2 with Vlan 1000 is then I have to change the file /etc/nova/nova.conf this line:

live_migration_uri = qemu + ssh: //%s/system

to

live_migration_uri = qemu + ssh: //eno2.1000.%s/system

Then live migration works how expected.
But how can I do that in Juju, so this change is fix?
Maybe a setting option for live_migration_uri would help in this charm?

[Matthias (matthiashuether)]

I have to correct: I did not make it work with my workaround suggestion. Unfortunately it does not work.

[Matthias (matthiashuether)]

Any news here? Livemigration works only if i set the correct addresses in /etc/hosts on all compute-notes

10.10.10.10 cpu-test.maas

This is only my workaround, and it is very cumbersome, because you always have to maintain the /etc/hosts file on all compute notes.

Maybe an aother idea is to set the "hypervisor_hostname" in the database (table "nova.compute_nodes") to the full name with interface: eno2.1000.cpu-test.maas? I think, an option to set the right interface (eno2.1000) in Charm would be useful. What do you think?

[Rony Zeidan (ronynov)]

Hi Mathias,
the only workaround i am using now it to manage hosts files on each compude node. Apparenttly there is a fix that shoudl address this in MAAS in 2.7 https://bugs.launchpad.net/maas/+bug/1823183
Regards

[Xav Paice (xavpaice)]

This config item is found on nova_compute_context.py, "ctxt['live_migration_uri'] = 'qemu+ssh://%s/system'".

If we use the "libvirt-migration-network" config setting via juju we do get livemigration using the 'correct' url, as the nova.conf contains live_migration_inbound_addr which is updated to the correct new address. However, due to https://bugs.launchpad.net/charm-nova-compute/+bug/1860743, the ssh key exchange is not complete for connections targeting that address.

[Alex Kavanagh (ajkavanagh)]

I think the fixes in
https://review.opendev.org/#/q/topic:bug/1874235+(status:open+OR+status:merged) may resolve the situation? Note that the config item "libvirt-migration-network" is very much deprecated and only for systems that are using spaces: the "migration" space is the preferred way of doing things.