Unable to upgrade k8s charm when only resource revisions changed

[Kian Parvin]

When a k8s charm+resource is deployed, if the corresponding charm on Charmhub has a new release where only the resource revision is bumped (i.e. the charm revision remains the same), there is seemingly no way to force Juju to upgrade the OCI image in the active deployment.

It feels like a common use case to build and release new OCI images while keeping the charm revision the same, this would also seemingly relate to the k8s pull-policy.

Juju version: 2.9.35-ubuntu-amd64 (Client and controller)
Cloud: Kubernetes - Tested on microk8s

To reproduce:
- "Release" charm into a channel on charmhub with revision X for the charm and revision Y for the OCI image resource.
- Deploy the charm to k8s (tested with microk8s)
- (In my case) Tested /status endpoint which returns my Github commit hash
- "Release" the charm again, into the same channel, with revision X for the charm and revision Y+1 for the OCI image resource.
- Try `juju refresh app` and various alternatives.
- Tried restarting the pod with `microk8s kubectl get pod <pod_name> -n <namespace> -o yaml | microk8s kubectl replace --force -f -`
- Hitting the /status endpoint again confirmed the pod was not upgraded.

None of the above refresh commands or restarting the pod could get the image to upgrade.

Alternatively, if it's required that the charm revision be increased for the refresh to work then an update to the docs in https://juju.is/docs/olm/upgrade-applications would be a good alternative.

[Ian Booth (wallyworld)]

The "normal" workflow is to publish a new charm tuple containing charm and resource revision updates. The charm revision can remain the same and if there are updated resource revisions, juju is supposed to run the upgrade-charm hook.

At least that's how is used to work with the old charm store. There might be some work needed with charmhub to achieve the same behaviour.

[Heather Lanigan (hmlanigan)]

I have a stepping stone pr up to handle the general case of `juju refresh` upgrading resources where there is no new charm available:

https://github.com/juju/juju/pull/15196

Next step is to ensure that ori-images are properly upgraded.

This change is unrelated to charmhub. It's not clear based on the code I found that this functionality worked with charmstore either. One of the problems resolved was due to code in state which didn't resolve pending resources unless the charm revision changed too. Nor was the CharmModifiedVersion updated to trigger the upgrade charm hook. There were more.

[Heather Lanigan (hmlanigan)]

With my change above, and using the loki-k8s charm, this appears to be resolved. Confirmation would be helpful.

I deployed the loki-k8s charm:

$ juju deploy loki-k8s --resource loki-image=2

Revision 2 was used for the Loki-image because it had a different SHA from the most recent resource revision.

$ microk8s kubectl get pod loki-k8s-0 -n two -o json | grep "loki-image"
                "image": "registry.jujucharms.com/charm/ayt4gaga2x20gye32qthgygevco13qgq6rfff/loki-image@sha256:4e6dca46091ac8e08f8e06e734d657ee6c0000f1727050fd70ca59af7086841d",

$ juju refresh loki-k8s
charm "loki-k8s": already up-to-date
resources to be upgraded
Leaving endpoints in "alpha": alertmanager, grafana-dashboard, grafana-source, ingress, logging, metrics-endpoint

Use `juju show-status-log loki-k8s-0` to see the upgrade-charm hook after `juju refresh` run.

$ microk8s kubectl get pod loki-k8s-0 -n two -o json | grep "loki-image"
                "image": "registry.jujucharms.com/charm/ayt4gaga2x20gye32qthgygevco13qgq6rfff/loki-image@sha256:b61a65dbf591ccebebf3874c8d5e37d4ba1a8d2017625da97a83a4ced2ff252e",

The image has a new SHA.

[Tom Haddon (mthaddon)]

Deploying with an older version of the published resource and then confirming `juju refresh` upgrades the resource is a bit different to deploying what's published in charmhub, and then uploading a new resource and releasing the same charm revision with the new resource revision and confirming if `juju refresh` does the right thing in this case.

[Heather Lanigan (hmlanigan)]

@mthaddon,

This is the closest approximation to the scenario which I can easily to reproduce. I do not have an existing deployment available where I see if the new code resolves the bug.

From juju's perspective, there is no difference in the code. Once a charm is deployed with a resource, any details about how that was done is lost unless the resource was uploaded. If a resource is uploaded you cannot upgrade it to a version of the resource in charmhub. Any difference, that I'm aware of, would be in charmhub response to the refresh/refresh call that are not within juju's control.

I'll see if I can put together a staging charm for test.

[Heather Lanigan (hmlanigan)]

I created a charm for test and put in charmhub with 1 image, deployed the charm, then uploaded/released a new image, and refreshed the charm. The image was updated successfully.