Charms stored for a model are not removed when the model is destroyed

[Cheryl Jennings]

Charms cached per-model are not removed when the associated model is destroyed.

Ex:

$ juju list-models --uuid
NAME MODEL UUID OWNER STATUS LAST CONNECTION
admin 8d8078f9-e924-493c-847e-714aa12dae3b admin@local available 13 minutes ago
default* c8c2820f-106f-4214-8bde-824d420af533 admin@local available 20 minutes ago
foo f304e7b9-95d8-4be1-8f78-05de3bf66482 admin@local available 30 minutes ago

$ juju destroy-model foo

On the controller, the charms still exist in the charm-get-cache
$ ls /var/lib/juju/charm-get-cache/f304e7b9-95d8-4be1-8f78-05de3bf66482
local_3a_trusty_2f_grafana-1.zip

And the charm still exists as a managed resource:
juju:PRIMARY> db.managedStoredResources.find({_id: {$regex: "^buckets/f304e7b9-95d8-4be1-8f78-05de3bf66482/charms/"}}).pretty()
{
 "_id" : "buckets/f304e7b9-95d8-4be1-8f78-05de3bf66482/charms/local:trusty/grafana-1-493f9c68-4ceb-413e-86b0-c07355b91711",
 "bucketuuid" : "f304e7b9-95d8-4be1-8f78-05de3bf66482",
 "user" : "",
 "path" : "buckets/f304e7b9-95d8-4be1-8f78-05de3bf66482/charms/local:trusty/grafana-1-493f9c68-4ceb-413e-86b0-c07355b91711",
 "resourceid" : "aa8d778efa63b3fc5dc9ad0a699eab450cb725662421ca8f2bd687551308a8b2d6422cdf2f142daf72972bf921d59f66",
 "txn-revno" : NumberLong(2),
 "txn-queue" : [
  "5730f316832c160e9e958737_6dc1a0f8"
 ]
}

[Menno Finlay-Smits (menno.smits)]

This should be easier to fix now that charms referencing counting and clean up has been tidied up.

[Anastasia (anastasia-macmood)]

In Juju 2.1, the directories have changed. However, the issue still exists.

Since I tested using ubuntu charm, after destroying model, I can see on the controller:

$ ls /var/lib/juju/charmcache/
cs_3a_ubuntu-10.charm

[Christian Muirhead (2-xtian)]

There are two parts to this.
1. The charm isn't removed from blob storage (as shown by the managedStoredResources query).
2. The downloaded file is kept in the charmrepo package's CacheDir.

The first problem is because we explicitly don't remove non-local charms from the system in the cleanup for charms.

It's clearly deliberate - there are comments to this effect in the code (https://github.com/juju/juju/blob/6f8a8cb/state/cleanup.go#L292 and https://github.com/juju/juju/blob/9a34f2a/state/charmref.go#L123) by fwereade - but I haven't been able to find any reason for it.

Pasting the comments for discussion.
In cleanupCharm:
// No cleanup necessary or possible [for store charms].

In charmDestroyOps:
// it's not so much that it's bad to delete store
// charms; but we don't have a way to reinstate them
// once purged, so we don't allow removal in the first
// place.

If I remove the local-charm check, removing applications seems to correctly clean up the charm storage. If I then redeploy the same application, the charm comes back fine and the deploy works, so I don't understand the "don't have a way to reinstate them" comment.

Similarly, deploying the charm in two different models, then removing one model (so that it removes the model-specific reference to the charm in storage), I can still add units to the application in the other model. So the storage is clearly doing the right thing for deduplication/refcounting.

I'll discuss with the rest of the team, but if no-one knows why we need to do this I'll remove the local-charm check.

[Anastasia (anastasia-macmood)]

Removing 2.1 milestone as we will not be addressing this issue in 2.1.

[Christian Muirhead (2-xtian)]

PR to remove the local-charm-only restriction: https://github.com/juju/juju/pull/7000

[Christian Muirhead (2-xtian)]

PR to remove the charm from the cache once it's in blob storage: https://github.com/juju/juju/pull/7010

[Anastasia (anastasia-macmood)]

This has been resolved since 2.1