destroy model fails if there's a relation to offered application

[Heather Lanigan]

Found with test-cmr-bundles-deploy, when destroying the models created by the test. The test passes for 2.8

https://jenkins.juju.canonical.com/job/test-deploy-test-deploy-bundles-aws/289/console

When run locally:

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
other ctrl-2zk8h41a aws/us-east-1 2.9.23.1 unsupported 14:45:43Z

SAAS Status Store URL
mysql terminated ctrl-2zk8h41a admin/test-cmr-bundles-deploy.mysql

App Version Status Scale Charm Store Channel Rev OS Message
wordpress active 1 wordpress charmstore stable 0 ubuntu

Unit Workload Agent Machine Public address Ports Message
wordpress/0* active idle 0 18.209.69.95 80/tcp

Machine State DNS Inst id Series AZ Message
0 started 18.209.69.95 i-0ef5700031291c278 bionic us-east-1b running

$ juju status -m test-cmr-bundles-deploy
Model Controller Cloud/Region Version SLA Timestamp Notes
test-cmr-bundles-deploy ctrl-2zk8h41a aws/us-east-1 2.9.23.1 unsupported 14:45:58Z attempt 2 to destroy model failed (will retry): model not empty, found 1 application (model not empty)

App Version Status Scale Charm Store Channel Rev OS Message
mysql unknown 0 mysql charmhub stable 58 ubuntu

Offer Application Charm Rev Connected Endpoint Interface Role
mysql mysql mysql 58 1/1 db mysql provider

[Ian Booth (wallyworld)]

The root cause might not be cmr related (EDIT: I was wrong, see below). It looks like there's been a change in behaviour (probably not intentional) where if you remove the last unit of an application, the application (with 0 units) stays behind. This is what's preventing the model from being removed.

eg
juju bootstrap
juju deploy ubuntu
juju remove-unit ubuntu/0
juju status

shows app "ubuntu" with scale 0

[Ian Booth (wallyworld)]

So the behaviour in comment #1 is unchanged between 2.8 and 2.9. Removing the last unit does leave the app behind. In some testing I got a destroy model to hang and then another time it worked as expected. Still looking...

[Ian Booth (wallyworld)]

Next scenario - deploy app, create an offer on that app, destroy model. All good.

Ah, but this scenario fails with similar symptoms to comment #1...

If there's a relation to the offer, the associated application's unit gets removed but not the app itself (stays at scale 0) and this holds up the model removal.

[Ian Booth (wallyworld)]

The issue is that the proxy application in the offering model ("remote-foo"), representing the consuming app, has a unit recorded as entering scope when the relation is created, but when the relation is removed, the leave scope for the "remote-foo" unit is never called. This keeps a relationscope record in the offering model and also doesn't allow the unitcount on the relation to drop to 0 and so the relation hangs around and the model cannot be destroyed.

[Ian Booth (wallyworld)]

Interestingly, when the relation is just removed without any model destruction, the correct cleanup is done. This works either when removing the relation from the offering model or the consuming model.

So it's just destroy model that fails to see the offering unit leave scope.

[Ian Booth (wallyworld)]

And it works if you destroy the consuming model. It's just the offering model that fails to trigger the app proxy unit to leave scope.

[Ian Booth (wallyworld)]

If you remove the offer (with --force to also remove the offer connection/relation) then things also clean up properly. And the offering model can be subsequently removed.

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/13582