migration 2.9->3.1 failed

[Marian Gasparovic]

Migrate action failed with "migrating: aborted, removing model from target controller: validating, timed out waiting for agents to report"
However subsequent attempt says target model already exists but it does not

$ juju migrate openstack foundations-maas-3
Migration started with ID "dae5702a-200f-444c-8296-a74c6b908860:0"

$ juju status|head
Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:22:47Z migrating: performing source prechecks

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:24:51Z migrating: importing model into target controller

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:25:03Z migrating: uploading model binaries into target controller

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:26:56Z migrating: validating, waiting for agents to report back

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:29:56Z migrating: validating, waiting for agents to report back: 483 succeeded, 400 still to report

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:31:53Z migrating: validating, waiting for agents to report back: 624 succeeded, 259 still to report

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:34:14Z migrating: validating, waiting for agents to report back: 745 succeeded, 138 still to report

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:36:52Z migrating: validating, waiting for agents to report back: 821 succeeded, 62 still to report

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:38:44Z migrating: validating, waiting for agents to report back: 855 succeeded, 28 still to report

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:41:03Z migrating: validating, waiting for agents to report back: 866 succeeded, 17 still to report

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:43:01Z migrating: aborted, removing model from target controller: validating, timed out waiting for agents to report

Model Controller Cloud/Region Version SLA Timestamp Notes
openstack foundations-maas maas_cloud/default 2.9.46 unsupported 13:45:08Z migrating: aborted, removing model from target controller: validating, timed out waiting for agents to report

$ juju migrate openstack foundations-maas-3
ERROR target prechecks failed: model named "openstack" already exists

$ juju switch foundations-maas-3
foundations-maas:admin/openstack -> foundations-maas-3:admin/controller

$ juju destroy-model openstack
ERROR model foundations-maas-3:admin/openstack not found

ubuntu@lab1-silo1-cpe-7e212464-c323-4a40-ad69-b8eab0a4faf3:~$ juju models
Controller: foundations-maas-3

Model Cloud/Region Type Status Machines Cores Units Access Last connection
controller* maas_cloud/default maas available 1 2 1 admin just now

$ juju switch foundations-maas
foundations-maas-3:admin/controller -> foundations-maas:admin/openstack

$ juju migrate openstack foundations-maas-3
ERROR target prechecks failed: model named "openstack" already exists

[Ian Booth (wallyworld)]

It sounds like there's 2 issues here - one, the migration failed. Two, the rollback after the failed migration left stuff behind in the target controller.

[Marian Gasparovic (marosg)]

Hi Ian,
if it helps, I tried the same today but this time I used just one Juju 2.9 controller and I reduced Openstack deployemnt to way less agents (860 vs 267 agents). This time migration worked fine.

Could it be the problem that during failure my 2.9 Juju was three units HA and target Juju was single node only?

[Ian Booth (wallyworld)]

No, the topology of the target controller doesn't matter. I think we have an intermittent issue with the agents reconnecting to the target controller during the final validation stage.

[Heather Lanigan (hmlanigan)]

@wallyworld, Here's the bug that was fixed before where artifacts were left on the destination controller after a failed migration: https://bugs.launchpad.net/juju/+bug/2004033. Joe said it could happen again if another tear down bug was introduced.