Local charms not de-duped when deployed multiple times

[Cheryl Jennings]

When deploying local charms multiple times, the charm is stored each time it is deployed, even if the charm has not changed.

Removing the service also does not remove the charm, so repeatedly deploying / destroying the same charm can lead to disk space exhaustion.

[Cheryl Jennings (cherylj)]

Verified that this happens on both 1.25.5 and current master.

[Ian Booth (wallyworld)]

The bundle sha is different each time. I deployed 2 charms from the same local directory:

$ juju deploy ./mysql
$ juju deploy ./mysql mysql2

The charm collection in the Juju database now has 2 charm entries with different charm urls. One of the fields in the bundleSHA. For each charm entry, this was different. So no de-duping could possibly occur.

[Ian Booth (wallyworld)]

And yet, the zips generated as part of AddLocalCharm(), and uploaded to mongo, appear the same, at least on the machine from which the deploy is done.

[Ian Booth (wallyworld)]

So, the reason is that on the server side, the charm archive is unpacked and then repacked with the revision from the charm url. The charm url revision is incremented on each deploy. This behaviour is I'm guessing as per the original design. I'm guessing this is because the charm url has the revision in it, and we want to ensure the charm archive has a matching revision.

[Ian Booth (wallyworld)]

I'm really not sure there's a bug as such here. All local charm deployments get a new incremented revision number. This is embedded in the archive. The archive thus has a different checksum.

We could maybe add a new feature to allow local charms with the same name to re-use an existing previously uploaded charm if the checksums match. Thus the same url as a previous upload would be used and no additional storage would be needed.

[Richard Harding (rharding)]

We've killed off using the revision file I thought? is this based on the local: url? I'm not getting why the zips should be different. Is it new timestamps since it's unzipped and rezipped?

[Ian Booth (wallyworld)]

The behaviour is that the revision as determined by the generated local: url is incremented server side when the charm archive is processed on upload. And that incremented revision number is set against the archive prior to sending to mongo. So what mongo gets is different each time even if the bytes on disk are the same. This behaviour has been around since forever from what I can see.

Looking at the code, the only place I can see the archive's revision attribute being used is when the archive is unpacked to a directory (and yes, the revision attr is written to the revision file). Is the revision file used? I'm not sure - how do we know what external code or tools depend on this file? eg one use is when a charm is downloaded from the charm store, via charm download, the charm is unpacked and the revision file written. If we really have killed off the revision file, we could ignore this revision attribute of the archive and the bytes would be the same (the charm url would still be incremented though). We would not write any revision file when expanding a charm.

Having said all that, maybe timestamps play a part too. We'd have to verify that if we decide to remove the revision attribute.

[Cheryl Jennings (cherylj)]

The de-dupe behavior is a non trivial change and will not be backported to 1.25. Instead, cached local charms should be deleted when the service is removed. Bug #1580418 will be used to track that work.

[Anastasia (anastasia-macmood)]

Since the issue has been reported, it has been fixed and Released. Marking this as such.