Juju 3 - cannot refresh charms after revision tracking

[Matus Kosut]

we have noticed strange issue with charm refresh on Juju 3+ deployment. Tested with multiple versions of juju controller (3+), e.g. 3.0.3 and 3.1.0.

There seems to be this new feature that tracks whether there is new charm revision available in the channel. Not sure how often this update is pulled (might take an hour or few for it to pop up). If there is a new revision found charm revision number gets highlighted in juju status.

The core of the issue is that once the highlight appears we are not able to run refresh anymore. Instead an error is returned:

```
ubuntu@juju-dev--multipass-juju:~$ juju refresh rsyslog-forwarder
ERROR already exists
```

To reproduce this one has to deploy older version of a charm and then wait for juju to fetch revisions and mark that the newer one exists.

[Heather Lanigan (hmlanigan)]

The charm revision pieces has been around since the beginning of juju 2.x. What has changed in juju 3 is async download of charm and surrounding logic.

I'm surprised we're only seeing this in one place as the charm revision data is gathered, and potentially updated every 24 hours.

[Adam Dyess (addyess)]

just to add another point to this bug, i tried with python juju==3.1.0.1

```python
    for app_name, app in model.applications.items():
        log.info(f"Upgrading {app_name} from {app.charm_url} to --channel={channel}")
        await app.refresh(channel=channel)
```

and found the same error is produced. I guess this likely indicates an issue on the controller

[John A Meinel (jameinel)]

I'm guessing that the work to update the tracker to notice that there are new resources available as well as a new charm revision, ends up putting something into the database such that the 'juju refresh' action then says "oh, I can't do that, you already have it". We'll have to figure out why that is an issue, but it very much seems to be a controller side issue.

[Heather Lanigan (hmlanigan)]

The work referred to in #3 has not been done yet. Investigating the cause now.

[Heather Lanigan (hmlanigan)]

Must update the error as well, it is unhelpful: "ERROR already exists"

[Heather Lanigan (hmlanigan)]

This appears to be a day one bug in juju 3.0 caused by enabling asynchronous charm download for charmhub charms by default. In juju 2.9 it was behind a feature flag.

This issue is specific to charmhub charms only.

To download a charm asynchronously, juju inserts a charm doc with some the charm's data and indicated that it requires downloading. Naively assuming that if the charm doc already exists, that there is nothing to do, the charm has been downloaded. However if the charm revision updater worker has run and found a charm revision different that the current one, it inserts a charm doc into the juju db with a placeholder flag.

[Heather Lanigan (hmlanigan)]

https://github.com/juju/juju/pull/15382 for 3.0

[Matus Kosut (matuskosut)]

@hmlanigan I briefly tested the upgrades with newly released v3.1.2 and it seems to work. Although I noticed there is multiple failed retries before it actually downloads a new charm revision. I wasn't sure if that is expected behavior so I link the logs in case it might be useful:

https://gist.github.com/matuskosut/d01f1424ed066520c5c6b6bfcf2f7980

[Heather Lanigan (hmlanigan)]

@matuskosut

It's not expected, however not impossible either. Not a bug as it did occur in the end.