relocated VM does not come up

[Marian Gasparovic]

During our Masakari validator test the relocated VM did not come up.

Environment is Ussuri on Focal, using Masakari stable.

Our validator workflow
- deploy OpenStack with Masakari
- deploy test OpenStack server
- kill nova-service and pacemaker_remote on a machine where instance runs
- wait for the relocation to happen

We had passing runs where it worked fine.
Here is a list of recorded events in the validator

2022-06-28-23:34:49 fce.openstack.masakari DEBUG Creating resources
2022-06-28-23:34:52 fce.openstack.masakari DEBUG Creating Masakari segment
2022-06-28-23:34:55 fce.openstack.masakari DEBUG Adding all hypervisors to the segment
2022-06-28-23:35:17 fce.openstack.masakari DEBUG Creating server
2022-06-28-23:35:19 fce.openstack.masakari DEBUG Waiting for server to become active
2022-06-28-23:36:00 fce.openstack.masakari DEBUG Creating FIP
2022-06-28-23:36:02 fce.openstack.masakari DEBUG Adding FIP 10.244.32.109 to the server
2022-06-28-23:36:05 fce.openstack.masakari DEBUG Server 7c997b5a-4814-4689-a8fd-49507cbe8623 is located on quilava.silo1.solutionsqa
2022-06-28-23:36:11 fce.openstack.masakari DEBUG Hypervisor quilava.silo1.solutionsqa is on Juju machine 0
2022-06-28-23:36:11 fce.openstack.masakari INFO Killing nova-service and pacemaker_remote on machine 0
2022-06-28-23:36:12 fce.openstack.masakari INFO Waiting for a server relocation

Then it continues waiting.

We were not able to find what went wrong, looks like Masakari tried to use machine 3 (spearow).

Logs and artifacts

https://oil-jenkins.canonical.com/artifacts/ae7fdf8d-5b5d-4c0b-8c47-a02c1100ec46/index.html

[Marian Gasparovic (marosg)]

masakari.exception.HostRecoveryFailureException: Failed to evacuate instances '7c997b5a-4814-4689-a8fd-49507cbe8623' from host 'quilava.silo1.solutionsqa'

in masakari-engine.log

[Marian Gasparovic (marosg)]

Also found `Lost connection to MySQL server during query` in placement_error.log at the time when masakari was relocating the instance