cannot add remote model (consume offer) after model migration

[Anna Savchenko]

After migrating a COS model from 3.1.6 to 3.3.3 (k8s controller), I cannot add CMR to OpenStack (machine controller).

Debug output looks okay:
$ juju consume --debug microk8s-controller-3-3:admin/cos.loki
13:44:08 INFO juju.cmd supercommand.go:56 running juju [3.3.3 3e20d5947e407dcb4ce9c6fc29ba04b24978468e gc go1.20.14]
13:44:08 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/26652/bin/juju", "consume", "--debug", "microk8s-controller-3-3:admin/cos.loki"}
13:44:08 INFO juju.juju api.go:86 connecting to API addresses: [10.131.24.72:17070]
13:44:08 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://10.131.24.72:17070/api"
13:44:08 INFO juju.api apiclient.go:707 connection established to "wss://10.131.24.72:17070/api"
13:44:09 INFO juju.juju api.go:86 connecting to API addresses: [10.131.24.65:17070 10.131.24.63:17070 10.131.24.64:17070]
13:44:09 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://10.131.24.63:17070/model/4e79285c-4378-4523-82dd-c154c4f51031/api"
13:44:09 INFO juju.api apiclient.go:707 connection established to "wss://10.131.24.63:17070/model/4e79285c-4378-4523-82dd-c154c4f51031/api"
13:44:09 INFO cmd consume.go:184 Added microk8s-controller-3-3:admin/cos.loki as loki
13:44:09 DEBUG juju.api monitor.go:35 RPC connection died
13:44:09 DEBUG juju.api monitor.go:35 RPC connection died
13:44:09 INFO cmd supercommand.go:556 command finished

But then SAAS appears in ERROR state:
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
openstack foundations-maas maas_cloud/default 3.1.6 unsupported 13:47:25Z

SAAS Status Store URL
loki error microk8s-controller-3-3 admin/cos.loki

Juju machine controller (foundations-maas) log has this:
./admin-openstack-4e7928.log:2024-03-27 12:59:30 ERROR juju.worker.remoterelations remoteapplicationworker.go:93 error in remote application worker for cos-loki: cannot connect to external controller: cannot get controller api info for remote model: expected 1 controller with model 1d2137f6-7b39-4976-8cd9-133544844323, got 2

1d2137f6-7b39-4976-8cd9-133544844323 is the COS model, and it is linked with only one controller - microk8s-controller-3-3. microk8s-controller is the old controller of COS model.

$ juju controllers
Use --refresh option with this command to see the latest information.

Controller Model User Access Cloud/Region Models Nodes HA Version
foundations-maas* openstack admin superuser maas_cloud/default 6 109 3 3.1.6
microk8s-controller test-1 admin superuser snapped-microk8s_cloud/localhost 1 - - 3.1.6
microk8s-controller-3-3 cos admin superuser snapped-microk8s_cloud/localhost 2 - - 3.3.3

$ juju show-controller microk8s-controller-3-3
microk8s-controller-3-3:
  models:
    controller:
      model-uuid: 65d11e3c-0b73-446e-80dd-ba1e7a8e1c40
    cos:
      model-uuid: 1d2137f6-7b39-4976-8cd9-133544844323
      unit-count: 8
  current-model: admin/cos

$ juju show-controller microk8s-controller
microk8s-controller:
  models:
    controller:
      model-uuid: 5d63ecfb-1e07-4101-8491-2589c741e137
  current-model: admin/test-1
  account:
    user: admin
    access: superuser

Versions:
foundations-maas: 3.1.6
microk8s-controller: 3.1.6
microk8s-controller-3-3: 3.3.3

Clouds: maas and microk8s

Reproduction steps: remove COS saas from Openstack, migrate COS model from 3.1.6 to 3.3.3 container controller, try to consume the offer with the new URL in Openstack

[Anna Savchenko (annsavchenko)]

subscribed ~field-critical

[Ian Booth (wallyworld)]

There's a quick fix that can be done to unblock. On the consuming controller, connect to mongo and inspect the "externalControllers" collection. It seems there's 2 records for the same model uuid. Delete the one that is incorrect.

Removing the COS saas should have deleted the external controller record if it was the last saas using it. Was there another saas also using an offer from that same model?

[Joseph Phillips (manadart)]

We do not remove external controller records that have the last model entry removed. There is an explicit comment to this effect in state/externalcontrollers.go.

This might be the source of the bug if there is CMR logic that assumes this is not the case.

[Ian Booth (wallyworld)]

Given comment #3, sounds like we have a bug where adding an entry for a model should replace an existing entry and instead adds a new one.

[Anna Savchenko (annsavchenko)]

Hi Ian,

Thanks for looking into this.

The issue was related to other models (micokr8s and landscape) still having old saas. Once I removed saas from those models, I was able to consume offers from COS.

[John A Meinel (jameinel)]

Ian and Joe-
Isn't migration meant to handle telling other models that the offer that
they are consuming is moving to the new location? Thus the SAAS that you
already had in the model was meant to keep working post migration, rather
than having to manually remove it and re-add it.

I suppose that if they removed the offer before the migration, that might
have broken the link (and then you have to manually remove the SAAS and
re-add it).

John
=:->

On Thu, Mar 28, 2024 at 5:44 PM Anna Savchenko <email address hidden>
wrote:

> Hi Ian,
>
> Thanks for looking into this.
>
> The issue was related to other models (micokr8s and landscape) still
> having old saas. Once I removed saas from those models, I was able to
> consume offers from COS.
>
> --
> You received this bug notification because you are subscribed to
> Canonical Juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/2059261
>
> Title:
> cannot add remote model (consume offer) after model migration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/2059261/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=juju; status=Triaged; importance=High; assignee=
> <email address hidden>;
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: annsavchenko manadart wallyworld
> Launchpad-Bug-Reporter: Anna Savchenko (annsavchenko)
> Launchpad-Bug-Modifier: Anna Savchenko (annsavchenko)
> Launchpad-Message-Rationale: Subscriber (Canonical Juju)
> Launchpad-Message-For: jameinel
> Launchpad-Subscription: juju bugs
>
>