CMR Model Migration - Error Querying Relation Scope

[Joseph Phillips]

If a model offering CMRs is migrated to another controller, we see an error in external consumers:

machine-0: 12:19:14 ERROR juju.api.watcher error trying to stop watcher: relation scope id "ef82b9ab-ebac-4287-8462-30372f9e08eb:r#0#dummy-source": settings ef82b9ab-ebac-4287-8462-30372f9e08eb:r#0#dummy-source not found (not found)

This appears due to the relationUnitsWatcher (state/watcher.go) looking for settings using the offering model's UUID (ef82b9ab-ebac-4287-8462-30372f9e08eb).

There is a settings entry for the current model UUID:
1596c280-b27e-49fb-8bb8-fb4aa15ef892:r#0#dummy-source

[Joseph Phillips (manadart)]

This might be a general issue with relation application data. It looks like it assumes common-controller relations.

[Joseph Phillips (manadart)]

Hmm, maybe a symptom of migration quiescence. This is the log around the resumption of a consumed offer for a migrated offerer.

controller-0: 14:17:08 ERROR juju.worker.remoterelations error in relation units worker for dummy-source:sink dummy-sink:source: relation scope id "ece46915-382c-42e2-8b48-036a4112cbcd:r#0#dummy-source": settings ece46915-382c-42e2-8b48-036a4112cbcd:r#0#dummy-source not found (not found)
controller-0: 14:17:08 ERROR juju.worker.remoterelations error in relation units worker for dummy-source:sink dummy-sink:source: relation scope id "ece46915-382c-42e2-8b48-036a4112cbcd:r#0#dummy-source": settings ece46915-382c-42e2-8b48-036a4112cbcd:r#0#dummy-source not found (not found)
controller-0: 14:17:08 ERROR juju.worker.remoterelations error in remote application worker for dummy-source: relation scope id "ece46915-382c-42e2-8b48-036a4112cbcd:r#0#dummy-source": settings ece46915-382c-42e2-8b48-036a4112cbcd:r#0#dummy-source not found (not found)
controller-0: 14:18:08 DEBUG juju.worker.remoterelations remote controller API addresses: [10.161.87.110:17070]
controller-0: 14:18:08 DEBUG juju.worker.remoterelations received redirect; new API addresses: [10.161.87.87:17070]
controller-0: 14:18:08 DEBUG juju.worker.remoterelations relations changed: []string{"dummy-source:sink dummy-sink:source"}, true
controller-0: 14:18:08 DEBUG juju.worker.remoterelations relation "dummy-source:sink dummy-sink:source" changed: {Error:<nil> Result:0xc004157130}
controller-0: 14:18:08 DEBUG juju.worker.remoterelations register remote relation dummy-source:sink dummy-sink:source to local application dummy-sink
controller-0: 14:18:09 DEBUG juju.worker.remoterelations import remote application token 82b0f25a-4d2c-4538-8979-091c45e6cb5f for dummy-source
controller-0: 14:18:09 DEBUG juju.worker.remoterelations relation status changed for relation-dummy-source.sink#dummy-sink.source: {dummy-source:sink remote-1244464f0c4946a780873980619246e5:source false alive}

[John A Meinel (jameinel)]

This is now preventing a migration of Mattermost to more stable juju controllers. It isn't clear from the discussion whether this bug still exists in 2.8 or not, but there hasn't been discussion on this for 9 months.

[John A Meinel (jameinel)]

Ian thought that this might have already been addressed and we just didn't update the bug.

[Joseph Phillips (manadart)]

*Might* be in the ballpark of https://github.com/juju/juju/pull/11091.

What version of Juju is the migrating Mattermost running on? I can verify with some steps to reproduce.

[Joseph Phillips (manadart)]

I did some testing today and I believe this was indeed resolved by the patch above.

Since it landed on the 2.8 series and was not back-ported, 2.8 -> 2.9 will be the first versions where upgrade-by-migration works for this scenario.