cross model breakage

[james beedy]

Hitting a string of different issues with CMR.

1) Models who consume CMR offers never die ( I think this is because I can't seem to remove the offer)

$ for i in `juju models | grep destroying | awk '{ print $1 }'`; do juju status -m $i >> myfile; done
$ cat myfile | pastebinit
http://paste.ubuntu.com/p/q57YwFTpwx/

2) Consuming model says there is an offer even after I remove it http://paste.ubuntu.com/p/dmQhwYhNr7/

3) Providing model shows connected when no clients are connected http://paste.ubuntu.com/p/tNSZs6r3t5/

[Ian Booth (wallyworld)]

A general comment without digging in too deeply...

To remove an offer, it can't have any connections (relations) to the offer. You can supply the --force flag to remove an offer and any existing relations.

Models won't die if there are any offers, applications, units, relations etc which cannot be removed. An application can't be removed if there exist offers for the application. The offer needs to be removed first. There may need to be UX improvement here.

In issue #3, how do you know no clients are connected? What does "juju offers" print? What will list the connections for the offers in a model.

In issue #2, the consuming model's SAAS application proxy is not removed when the offer is removed. This needs to be done by the user operating that model using "juju remove-application". But I think the SAAS item does show up as "blocked" to indicate that the consumed offer is no longer valid. Note that the consuming model does not show the offer in status. The consuming model creates a proxy to the offered application which presents the exported endpoints as available for relations. This application proxy receives events from the application on the offering side.

[Peter Matulis (petermatulis)]

I hit issue #1 today. I wanted to remove two models where each had an offer connection to the other. Trying to remove a model hung.

Suggestion: Since a failure is guaranteed to occur, print an appropriate warning and exit.

[Andrea Ieri (aieri)]

I've run into the same issue, but only if the CMR moves to the 'joining' state.

A how to reproduce example:

$ juju add-model -c overcloud-controller cmr-test-model
Added 'cmr-test-model' model on os-charmlab8/RegionOne with credential 'admin' for user 'admin'

$ juju deploy -m overcloud-controller:cmr-test-model cs:~aieri/prometheus-blackbox-exporter-0 overcloud-pbe
Located charm "cs:~aieri/prometheus-blackbox-exporter-0".
Deploying charm "cs:~aieri/prometheus-blackbox-exporter-0".

$ juju offer -c overcloud-controller cmr-test-model.overcloud-pbe:blackbox-exporter
Application "overcloud-pbe" endpoints [blackbox-exporter] available at "admin/cmr-test-model.overcloud-pbe"

$ juju relate -m charmlab8:aieri-openstack overcloud-controller:admin/cmr-test-model.overcloud-pbe prometheus:blackbox-exporter

# This remains in 'joining' because the offering application isn't actually deployed (it's a subordinate)
$ juju offers -m overcloud-controller:cmr-test-model
Offer User Relation id Status Endpoint Interface Role Ingress subnets
overcloud-pbe admin 1 joining blackbox-exporter http provider 10.0.8.93/32

$ juju remove-relation -m charmlab8:aieri-openstack overcloud-pbe prometheus:blackbox-exporter
$ juju remove-saas -m charmlab8:aieri-openstack overcloud-pbe
$ juju remove-offer -c overcloud-controller cmr-test-model.overcloud-pbe
ERROR cannot delete application offer "overcloud-pbe": offer has 1 relation

At this point the cmr-test-model model can only be removed by forcibly destroying the controller.

The problematic step seems to be add-relation. If you only consume and then remove-saas/remove-offer things work as expected.

[Ian Booth (wallyworld)]

Several improvements have landed in the upcoming 2.6.9 release which hopefully address many key issues.

A key problem was that in a cmr scenario, if the other model was deleted, or there was a network or similar issue preventing both sides of the cross model relation communicating, deletion of an application or model destruction could not complete.

We have added the following:
- removing an application which is offered is now possible
- if a consumed or offered application cannot be removed normally, the --force option can be used to force through the removal even though the other end is not responding
- if a model cannot be deleted due to a stuck application, use --force
- trying to remove by id a cross model relation could fail; again, using --force should fix that

I can't reproduce the scenario in comment #3 because I don't have permissions to access the charms. But I did deploy a different subordinate charm, offer it, relate to it etc. And I validated the remove-relation --force does now clean up the relation, and hence unblock removal of the offered application and/or deletion of the model. Note that remove-relation --force does take a minute or so to push through the removal.

So given that --force can now be applied to removing stuck relations, applications, models, I'll go ahead and mark this as fix committed. You can try it out by using the 2.6.9 candidate snap. The official 2.6.9 release should happen in the next week or so.

[Andrea Ieri (aieri)]

I can confirm that juju 2.6.9 improves the handling of cross-model relations.
While operating on a pair of 2.6.9 controllers/models, I am now successful in adding and removing relations/saas/offers.
The models that were stuck in 'destroying' under juju 2.6.8 do not get unblocked after a model upgrade - perhaps due to the upgrade not actually having fully completed - but I do not see any further issues when starting from a clean setup.