Regression in deploy charm with a forced series, refresh fails and resources cannot be downloaded.

[Corey Bryant]

We have the need to force deploy some charms to test them on kinetic, since they don't publish for 22.10. That should be possible with --force, however the deploy fails with "ERROR no releases found for channel "latest/edge"":

$ juju deploy --force --series kinetic --channel latest/edge ch:mysql-innodb-cluster
ERROR no releases found for channel "latest/edge"

This is with 2.9/stable:
2.9/stable: 2.9.37 2022-11-14 (21315) 96MB classic

[Jordan Barrett (barrettj12)]

Here's a workaround in the meantime:

Add a kinetic machine:
    juju add-machine --series kinetic

Download the charm:
    juju download mysql-innodb-cluster --channel latest/edge --series jammy

Go into the downloaded charm archive, and edit the metadata.yaml, adding "kinetic" to the list of supported series.

Deploy the charm to the added machine:
    juju deploy ./mysql-innodb-cluster_432e60e.charm --to 0 --series kinetic

[Jordan Barrett (barrettj12)]

The issue here is that we're using the value of --series for two distinct things:
- The charm revision to pull from Charmhub
- The platform to use for the new machine

Currently there is no way to supply different values for these two "series". Maybe we could have a --cross-series flag that allows this. e.g.
    juju deploy foo --cross-series jammy:kinetic
will pull the jammy version of the charm "foo", but spin up a kinetic machine and deploy "foo" to it. Naturally --cross-series would imply --force (or at least disable the series checks).

[Heather Lanigan (hmlanigan)]

@barrettj12, this is a regression, the --force flag was working afaik. We would have heard about it otherwise long before now.

It appears there is something different about this charm we need to investigate.

$ juju deploy ubuntu --series kinetic
ERROR series "kinetic" not supported by charm, supported series are: focal, focal, bionic, bionic, xenial, xenial, trusty, precise, jammy, groovy, disco, cosmic, artful, vivid. Use --force to deploy the charm anyway.

$ juju deploy ubuntu --series kinetic --force
Located charm "ubuntu" in charm-hub, revision 21
Deploying "ubuntu" from charm-hub charm "ubuntu", revision 21 in channel stable on kinetic

[Corey Bryant (corey.bryant)]

I was asked to shed some light on the importance of --force and interim release support in this bug.

Why we are even using kinetic? For the distribution side of openstack, we provide stable support for many deb packages, and the way that we regression test them is by running suites of regression tests against juju deployed clouds. For Ubuntu SRUs, all fixes must land in the latest Ubuntu series prior to being backported to N-1 and then N-2, etc. So at this point in time, we have to land all fixes in lunar, then kinetic, then jammy, etc.

We can publish 22.10 charms to get around this, but this ends up adding cycles to work that would be free if force worked.

Thanks for taking a look at this. Very much appreciated.

[Heather Lanigan (hmlanigan)]

The problem is related to "latest/edge" vs "edge" and the data we receive from charmhub. I need to have a conversation with them, as the data received has been looking odd to me for a few weeks. This related to the idea of a default track defined by the charm author. "latest" is the default is not specified.

$ juju deploy --series kinetic --channel edge ch:mysql-innodb-cluster
ERROR series "kinetic" not supported by charm, supported series are: focal, focal, jammy, impish, hirsute, groovy, eoan. Use --force to deploy the charm anyway.

Looks like there is an issue with the resources for the charm related to the old bug (fixed?) where charms with resources needed resources to be specified when releasing to a track, even if they were no updated.

$ juju deploy --series kinetic --channel edge ch:mysql-innodb-cluster --force
Located charm "mysql-innodb-cluster" in charm-hub, revision 30
ERROR listing resources for charm "ch:amd64/kinetic/mysql-innodb-cluster-30": No revision was found in the Store.

[Heather Lanigan (hmlanigan)]

The "latest/edge" vs "edge" issue with charmhub responses was filed as a bug previously: https://bugs.launchpad.net/snapstore-server/+bug/1986547

[Felipe Reyes (freyes)]

> The problem is related to "latest/edge" vs "edge" and the data we receive from charmhub.

it's worth to mention that for the case of mysql-innodb-cluster the following commands will NOT use the same charm:

```
juju deploy --series kinetic --channel edge ch:mysql-innodb-cluster --force

juju deploy --series kinetic --channel latest/edge ch:mysql-innodb-cluster --force
```

The former should use `8.0/edge`, the default's track, which is fed by the git branch "stable/jammy", while `latest/edge` is fed by builds using the git branch "master".

> Looks like there is an issue with the resources for the charm related to the old bug (fixed?) where charms with resources needed resources to be specified when releasing to a track, even if they were no updated.

Revision 30 effectively is published without resources associated.

8.0 ubuntu 20.04 (amd64) stable be2f3d1 30 -
         ubuntu 20.04 (arm64) stable be2f3d1 30 -
         ubuntu 20.04 (ppc64el) stable be2f3d1 30 -
         ubuntu 20.04 (s390x) stable be2f3d1 30 -
         ubuntu 22.04 (amd64) stable be2f3d1 30 -
         ubuntu 22.04 (arm64) stable be2f3d1 30 -
         ubuntu 22.04 (ppc64el) stable be2f3d1 30 -
         ubuntu 22.04 (s390x) stable be2f3d1 30 -

[Felipe Reyes (freyes)]

"juju deploy --channel edge --series kinetic --force ch:mysql-innodb-cluster" should work now, I fixed the issue with the resource running:

$ charmcraft release mysql-innodb-cluster --revision=30 --channel 8.0/stable --resource=mysql-shell:0
Revision 30 of charm 'mysql-innodb-cluster' released to 8.0/stable (attaching resources: 'mysql-shell' r0)
 ~ $ charmcraft status mysql-innodb-cluster | grep stable
...
8.0 ubuntu 20.04 (amd64) stable be2f3d1 30 mysql-shell (r0)
         ubuntu 20.04 (arm64) stable be2f3d1 30 mysql-shell (r0)
         ubuntu 20.04 (ppc64el) stable be2f3d1 30 mysql-shell (r0)
         ubuntu 20.04 (s390x) stable be2f3d1 30 mysql-shell (r0)
         ubuntu 22.04 (amd64) stable be2f3d1 30 mysql-shell (r0)
         ubuntu 22.04 (arm64) stable be2f3d1 30 mysql-shell (r0)
         ubuntu 22.04 (ppc64el) stable be2f3d1 30 mysql-shell (r0)
         ubuntu 22.04 (s390x) stable be2f3d1 30 mysql-shell (r0)

[Corey Bryant (corey.bryant)]

Hi,

Unfortunately using edge isn't working with --force on kinetic.

I have a simple bundle that can be deployed against the local provider in 2 scenarios to recreate this:

1) deployed applications result in jammy instances being deployed for edge channels

$ cat kinetic-mysql.yaml
series: kinetic
applications:
  mysql-innodb-cluster:
    charm: ch:mysql-innodb-cluster
    num_units: 3
    channel: edge

$ juju deploy --force ./kinetic-mysql.yaml
Located charm "mysql-innodb-cluster" in charm-hub, channel 8.0/stable
Executing changes:
- upload charm mysql-innodb-cluster from charm-hub for series kinetic from channel edge with architecture=amd64
- deploy application mysql-innodb-cluster from charm-hub on kinetic with edge
  added resource mysql-shell
- add unit mysql-innodb-cluster/0 to new machine 0
- add unit mysql-innodb-cluster/1 to new machine 1
- add unit mysql-innodb-cluster/2 to new machine 2
Deploy of bundle completed.

$ juju status --format=yaml|grep jammy
    series: jammy
    series: jammy
    series: jammy
    series: jammy

2) deploy kinetic machines first, then deploy applications to the machines

$ cat bundle.yaml
series: kinetic
machines:
  '0':
    constraints: mem=3072M
  '1':
    constraints: mem=3072M
  '2':
    constraints: mem=3072M
applications:
  mysql-innodb-cluster:
    charm: ch:mysql-innodb-cluster
    num_units: 3
    to:
      - '0'
      - '1'
      - '2'
    channel: edge

$ juju deploy --force ./kinetic-mysql.yaml
Located charm "mysql-innodb-cluster" in charm-hub, channel 8.0/stable
Executing changes:
- upload charm mysql-innodb-cluster from charm-hub for series kinetic from channel edge with architecture=amd64
- deploy application mysql-innodb-cluster from charm-hub on kinetic with edge
  added resource mysql-shell
- add new machine 0
- add new machine 1
- add new machine 2
- add unit mysql-innodb-cluster/0 to new machine 0
ERROR cannot deploy bundle: cannot add unit for application "mysql-innodb-cluster": acquiring machine to host unit "mysql-innodb-cluster/0": canno
t assign unit "mysql-innodb-cluster/0" to machine 0: series does not match

[Billy Olsen (billy-olsen)]

Note that the mysql-innodb-cluster charm does have a default channel set to 8.0/stable rather than leaving it unset and falling back to the latest/...

[Heather Lanigan (hmlanigan)]

Some of the mysql-innodb-cluster data has changed since I responded in #5. There is now a default channel. So "edge" no longer should get "latest/edge", but instead "8.0/edge", which is a closed channel, afaik.

$ juju info mysql-innodb-cluster
name: mysql-innodb-cluster
publisher: OpenStack Charmers
summary: MySQL InnoDB Cluster
description: |
  MySQL InnoDB Cluster Charm deploys and manages the lifecycle of a
  MySQL InnoDB Cluster.
store-url: https://charmhub.io/mysql-innodb-cluster
charm-id: i5HWuKoEat09854Sz0qXHOjmr4wwHLfP
supports: focal, jammy
tags: cloud, databases
subordinate: false
relations:
  provides:
    db-monitor: mysql-monitor
    db-router: mysql-router
    shared-db: mysql-shared
  requires:
    certificates: tls-certificates
channels: |
  8.0/stable: be2f3d1 2022-11-22 (30) 15MB amd64, arm64, ppc64el, s390x focal, jammy
  8.0/candidate: ↑
  8.0/beta: ↑
  8.0/edge: ↑
  latest/stable: 15 2022-02-09 (15) 13MB amd64, arm64, armhf, i386, ppc64el, s390x focal, groovy, hirsute, impish
  latest/candidate: ↑
  latest/beta: ↑
  latest/edge: 95cc985 2022-11-28 (37) 15MB amd64, arm64, ppc64el, s390x jammy

This is now a different bug than originally reported, though the same user experience.

$ sjuju deploy --channel edge --series kinetic --force ch:mysql-innodb-cluster
Located charm "mysql-innodb-cluster" in charm-hub, revision 30
ERROR listing resources for charm "ch:amd64/kinetic/mysql-innodb-cluster-30": No revision was found in the Store.

Juju is looking for resources in the 8.0/stable channel, but appears to ignore force.

[Corey Bryant (corey.bryant)]

For the individual charm, this can be worked around with:

$ touch resource.snap
$ juju deploy --force --resource mysql-shell=./resource.snap --series kinetic --channel edge ch:mysql-innodb-cluster

However with a bundle the behavior is different. It is deploying rev 30 to jammy even though the output says kinetic is getting deployed:

$ cat kinetic-mysql.yaml
series: kinetic
applications:
  mysql-innodb-cluster:
    charm: ch:mysql-innodb-cluster
    num_units: 1
    channel: edge
    resources:
      mysql-shell: /tmp/resource.snap

$ juju deploy --force ./kinetic-mysql.yaml
Located charm "mysql-innodb-cluster" in charm-hub, channel 8.0/stable
Executing changes:
- upload charm mysql-innodb-cluster from charm-hub for series kinetic from channel edge with architecture=amd64
- deploy application mysql-innodb-cluster from charm-hub on kinetic with edge
  added resource mysql-shell
- add unit mysql-innodb-cluster/0 to new machine 0
Deploy of bundle completed.

[Heather Lanigan (hmlanigan)]

There is a regression when --force is used to deploy a charm with a different series than the charm supports. `juju refresh` of a charm deployed with --force for series will also fail:

$ juju deploy ubuntu --series kinetic --force --revision 20 --channel stable
Located charm "ubuntu" in charm-hub, revision 20
Deploying "ubuntu" from charm-hub charm "ubuntu", revision 20 in channel stable on kinetic

$ juju refresh ubuntu-twenty
ERROR downloading charm "ch:amd64/jammy/ubuntu-21" from origin {charm-hub charm DksXQKAQTZfsUmBAGanZAhpoS4dpmXel 458c7bde333165ae1c3360c8eb7bad911688506dea2a768aeba56bf2eebe1b6b 0xc0039de558 stable amd64/ubuntu/22.10 }: revision-not-found: No revision was found in the Store.

The wrong base is being saved in the charm origin of the application doc. It should be the base of the charm deployed, not the base used to create the machine.

[Simon Richardson (simonrichardson)]

As we're using --force here, I think we need to work though a solution, but I don't think this is a high.

We should be attempting to use charmstore channels/tracks for providing charms without force. Similarly --channel foo/edge could be an alternative.

[Ian Booth (wallyworld)]

The next 2.9.46 candidate release will not include a fix for this bug and we don't plan on any more 2.9 releases. As such it is being removed from its 2.9 milestone.

If the bug is still important to you, let us know and we can consider it for inclusion on a 3.x milestone.