cross model breakage

Bug #1768682 reported by james beedy
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Ian Booth

Bug Description

Hitting a string of different issues with CMR.

1) Models who consume CMR offers never die ( I think this is because I can't seem to remove the offer)

$ for i in `juju models | grep destroying | awk '{ print $1 }'`; do juju status -m $i >> myfile; done
$ cat myfile | pastebinit
http://paste.ubuntu.com/p/q57YwFTpwx/

2) Consuming model says there is an offer even after I remove it http://paste.ubuntu.com/p/dmQhwYhNr7/

3) Providing model shows connected when no clients are connected http://paste.ubuntu.com/p/tNSZs6r3t5/

Tags: cross-model
james beedy (jamesbeedy)
description: updated
james beedy (jamesbeedy)
description: updated
Ian Booth (wallyworld)
Changed in juju:
milestone: none → 2.3.8
importance: Undecided → High
status: New → Triaged
Revision history for this message
Ian Booth (wallyworld) wrote :

A general comment without digging in too deeply...

To remove an offer, it can't have any connections (relations) to the offer. You can supply the --force flag to remove an offer and any existing relations.

Models won't die if there are any offers, applications, units, relations etc which cannot be removed. An application can't be removed if there exist offers for the application. The offer needs to be removed first. There may need to be UX improvement here.

In issue #3, how do you know no clients are connected? What does "juju offers" print? What will list the connections for the offers in a model.

In issue #2, the consuming model's SAAS application proxy is not removed when the offer is removed. This needs to be done by the user operating that model using "juju remove-application". But I think the SAAS item does show up as "blocked" to indicate that the consumed offer is no longer valid. Note that the consuming model does not show the offer in status. The consuming model creates a proxy to the offered application which presents the exported endpoints as available for relations. This application proxy receives events from the application on the offering side.

Changed in juju:
milestone: 2.3.8 → 2.3.9
Changed in juju:
milestone: 2.3.9 → 2.3.10
Revision history for this message
Peter Matulis (petermatulis) wrote :

I hit issue #1 today. I wanted to remove two models where each had an offer connection to the other. Trying to remove a model hung.

Suggestion: Since a failure is guaranteed to occur, print an appropriate warning and exit.

Tim Penhey (thumper)
tags: added: cross-model
Changed in juju:
milestone: 2.3.10 → none
Revision history for this message
Andrea Ieri (aieri) wrote :

I've run into the same issue, but only if the CMR moves to the 'joining' state.

A how to reproduce example:

$ juju add-model -c overcloud-controller cmr-test-model
Added 'cmr-test-model' model on os-charmlab8/RegionOne with credential 'admin' for user 'admin'

$ juju deploy -m overcloud-controller:cmr-test-model cs:~aieri/prometheus-blackbox-exporter-0 overcloud-pbe
Located charm "cs:~aieri/prometheus-blackbox-exporter-0".
Deploying charm "cs:~aieri/prometheus-blackbox-exporter-0".

$ juju offer -c overcloud-controller cmr-test-model.overcloud-pbe:blackbox-exporter
Application "overcloud-pbe" endpoints [blackbox-exporter] available at "admin/cmr-test-model.overcloud-pbe"

$ juju relate -m charmlab8:aieri-openstack overcloud-controller:admin/cmr-test-model.overcloud-pbe prometheus:blackbox-exporter

# This remains in 'joining' because the offering application isn't actually deployed (it's a subordinate)
$ juju offers -m overcloud-controller:cmr-test-model
Offer User Relation id Status Endpoint Interface Role Ingress subnets
overcloud-pbe admin 1 joining blackbox-exporter http provider 10.0.8.93/32

$ juju remove-relation -m charmlab8:aieri-openstack overcloud-pbe prometheus:blackbox-exporter
$ juju remove-saas -m charmlab8:aieri-openstack overcloud-pbe
$ juju remove-offer -c overcloud-controller cmr-test-model.overcloud-pbe
ERROR cannot delete application offer "overcloud-pbe": offer has 1 relation

At this point the cmr-test-model model can only be removed by forcibly destroying the controller.

The problematic step seems to be add-relation. If you only consume and then remove-saas/remove-offer things work as expected.

Revision history for this message
Ian Booth (wallyworld) wrote :

Several improvements have landed in the upcoming 2.6.9 release which hopefully address many key issues.

A key problem was that in a cmr scenario, if the other model was deleted, or there was a network or similar issue preventing both sides of the cross model relation communicating, deletion of an application or model destruction could not complete.

We have added the following:
- removing an application which is offered is now possible
- if a consumed or offered application cannot be removed normally, the --force option can be used to force through the removal even though the other end is not responding
- if a model cannot be deleted due to a stuck application, use --force
- trying to remove by id a cross model relation could fail; again, using --force should fix that

I can't reproduce the scenario in comment #3 because I don't have permissions to access the charms. But I did deploy a different subordinate charm, offer it, relate to it etc. And I validated the remove-relation --force does now clean up the relation, and hence unblock removal of the offered application and/or deletion of the model. Note that remove-relation --force does take a minute or so to push through the removal.

So given that --force can now be applied to removing stuck relations, applications, models, I'll go ahead and mark this as fix committed. You can try it out by using the 2.6.9 candidate snap. The official 2.6.9 release should happen in the next week or so.

Changed in juju:
milestone: none → 2.6.9
status: Triaged → Fix Committed
assignee: nobody → Ian Booth (wallyworld)
Revision history for this message
Andrea Ieri (aieri) wrote :

I can confirm that juju 2.6.9 improves the handling of cross-model relations.
While operating on a pair of 2.6.9 controllers/models, I am now successful in adding and removing relations/saas/offers.
The models that were stuck in 'destroying' under juju 2.6.8 do not get unblocked after a model upgrade - perhaps due to the upgrade not actually having fully completed - but I do not see any further issues when starting from a clean setup.

Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.