charmhub: deploying mysql from edge channel causes Juju to deploy a very old xenial stable revision

[John A Meinel]

The charm mysql has been adopted by a new team (taking over an old imported charm).
They have decided to take it out of 'stable' and only have an 'edge' channel for the new code that they are releasing.

The info for `juju info mysql` looks sane (it only lists the edge channel as available, which only supports focal), which matches https://charmhub.io/mysql

```
name: mysql
charm-id: F8CtvgcznoeRSupRrY2uEa67oQtBTF7x
summary: MySQL is a fast, stable and true multi-user, multi-threaded SQL database
publisher: Data Platform
tags: cloud, databases
subordinate: false
store-url: https://charmhub.io/mysql
description: |
  MySQL is a fast, stable and true multi-user, multi-threaded SQL database
  server. SQL (Structured Query Language) is the most popular database query
  language in the world. The main goals of MySQL are speed, robustness and
  ease of use.
relations:
  provides:
    db-router: mysql-router
    shared-db: mysql-shared
  requires: {}
channels: |
  latest/stable: –
  latest/candidate: –
  latest/beta: –
  latest/edge: 67 2022-07-08 (67) 2MB
```

If you do:
`juju deploy mysql`
You do get told that you cannot install it but there are other channels available:

```
$ juju deploy mysql
ERROR selecting releases: charm or bundle not found for channel "", platform "amd64"
available releases are:
  channel "candidate": available series are: bionic, xenial, trusty, precise, artful, zesty
  channel "beta": available series are: bionic, xenial, trusty, precise, artful, zesty
  channel "edge": available series are: focal, bionic, xenial, trusty, precise, artful, zest
```

However, the 'refresh' api is now telling us about lots of channels and os versions, which 'info' doesn't seem to want to exist.

If you then do try to deploy with --channel=edge:
$ juju deploy mysql --channel edge Located charm "mysql" in charm-hub, revision 58
Deploying "mysql" from charm-hub charm "mysql", revision 58 in channel stable on xenial

Suddenly we are redirected to xenial stable instead of edge.

I have dumps of the charmhub interaction (sorry for the canonical private links, I'm not sure if there are secrets here that we don't want to make public)
`juju deploy mysql` https://pastebin.canonical.com/p/t7sMhBH2Z5/

`juju deploy mysql --channel=edge` https://pastebin.canonical.com/p/NNwCbT2Dwv/

From what I can tell, Juju is asking charmhub about 'edge' and it returns 'no valid bases, here are some valid bases, 16.04, 18.04, 14.04.
Note that we asked about edge, but only got pointed to old stable release, ignoring the request for edge. So Focal isn't even listed. Also, Xenial is the default (even if Bionic would be available)

machine-0: 09:00:00 TRACE juju.apiserver.charms.charmhubrepo.client charmhub POST request POST /v2/charms/refresh HTTP/1.1
Host: api.charmhub.io
Accept: application/json
Content-Type: application/json
User-Agent: Juju/2.9.33

{"context":[],"actions":[{"action":"install","instance-key":"15c7513d-3358-4e06-8c59-80877b4a7b5c","name":"mysql","channel":"edge","base":{"architecture":"amd64","name":"NA","channel":"NA"}}],"fields":["bases","config-yaml","download","id","license","metadata-yaml","name","publisher","resources","revision","summary","type","version"]}

machine-0: 09:00:00 TRACE juju.apiserver.charms.charmhubrepo.client charmhub POST response HTTP/1.1 200 OK
Content-Length: 582
Content-Type: application/json
Date: Wed, 13 Jul 2022 13:00:00 GMT
Server: gunicorn/20.0.4
Snap-Store-Version: 49
X-Request-Id: AE4645A7FC560A8325F001BB62CEC1D014DC1E5
X-Vcs-Revision: 9abffbb1
X-View-Name: snapdevicegw.webapi_refresh_charm.charm_refresh

{"error-list":[],"results":[{"charm":null,"error":{"code":"invalid-charm-base","extra":{"default-bases":[{"architecture":"amd64","channel":"16.04","name":"ubuntu"},{"architecture":"amd64","channel":"18.04","name":"ubuntu"},{"architecture":"amd64","channel":"14.04","name":"ubuntu"}]},"message":"Instance key '15c7513d-3358-4e06-8c59-80877b4a7b5c' invalid 'base' in 'install' for charm_id=F8CtvgcznoeRSupRrY2uEa67oQtBTF7x name=mysql"},"id":"F8CtvgcznoeRSupRrY2uEa67oQtBTF7x","instance-key":"15c7513d-3358-4e06-8c59-80877b4a7b5c","name":"mysql","released-at":null,"result":"error"}]}

We then do a follow up to ask for the edge of xenial channel (the edge channel might be closed):
machine-0: 09:00:00 TRACE juju.apiserver.charms.charmhubrepo.client charmhub POST request POST /v2/charms/refresh HTTP/1.1
Host: api.charmhub.io
Accept: application/json
Content-Type: application/json
User-Agent: Juju/2.9.33

{"context":[],"actions":[{"action":"install","instance-key":"15c7513d-3358-4e06-8c59-80877b4a7b5c","name":"mysql","channel":"edge","base":{"architecture":"amd64","name":"NA","channel":"NA"}}],"fields":["bases","config-yaml","download","id","license","metadata-yaml","name","publisher","resources","revision","summary","type","version"]}

machine-0: 09:00:00 TRACE juju.apiserver.charms.charmhubrepo.client charmhub POST response HTTP/1.1 200 OK
Content-Length: 582
Content-Type: application/json
Date: Wed, 13 Jul 2022 13:00:00 GMT
Server: gunicorn/20.0.4
Snap-Store-Version: 49
X-Request-Id: AE4645A7FC560A8325F001BB62CEC1D014DC1E5
X-Vcs-Revision: 9abffbb1
X-View-Name: snapdevicegw.webapi_refresh_charm.charm_refresh

{"error-list":[],"results":[{"charm":{"bases":[{"architecture":"amd64","channel":"17.04","name":"ubuntu"},{"architecture":"amd64","channel":"16.04","name":"ubuntu"},{"architecture":"amd64","channel":"14.04","name":"ubuntu"}],"config-yaml":"<elided>","download":{"hash-sha-256":"eca0b3bda9ad4bc59ca774b8fcab8ab97690cf4f70ed6544caa86dcf12a3ccca","size":170004,"url":"https://api.charmhub.io/api/v1/charms/download/F8CtvgcznoeRSupRrY2uEa67oQtBTF7x_58.charm"},"id":"F8CtvgcznoeRSupRrY2uEa67oQtBTF7x","license":"","metadata-yaml":"<elided>","name":"mysql","publisher":{"display-name":"Data Platform","id":"gVVkuxw4C56POG1eszt4RPR3L5Eak8XE","username":"data-platform","validation":"unproven"},"resources":[],"revision":58,"summary":"MySQL is a fast, stable and true multi-user, multi-threaded SQL database","type":"charm","version":"58"},"effective-channel":"stable","id":"F8CtvgcznoeRSupRrY2uEa67oQtBTF7x","instance-key":"c8f6e903-bc76-4e74-852d-d2d45488f34f","name":"mysql","released-at":"2022-02-09T11:59:30.691797+00:00","result":"install"}]}

This response causes us to resolve to xenial stable:
machine-0: 09:00:01 TRACE juju.apiserver.charms.charmhubrepo Resolved CharmHub charm "ch:amd64/xenial/mysql-58" with origin {charm-hub charm 0xc0022964b0 stable amd64/ubuntu/xenial }

[Daniel Manrique (roadmr)]

first Juju does this

curl -XPOST -s https://api.charmhub.io/v2/charms/refresh -H 'Content-type: application/json' -d '{ "context": [], "actions": [{"name": "mysql", "base": {"name": "NA", "channel": "NA", "architecture": "amd64"} , "channel": "edge" , "action": "install", "instance-key": "a-test"}]}' | jq .

it gets the list of bases and picks the first one, then does this with that:

curl -XPOST -s https://api.charmhub.io/v2/charms/refresh -H 'Content-type: application/json' -d '{ "context": [], "actions": [{"name": "mysql", "base": {"name": "ubuntu", "channel": "16.04", "architecture": "amd64"} , "channel": "edge" , "action": "install", "instance-key": "a-test"}]}' | jq .

[John A Meinel (jameinel)]

Given the output of `juju info mysql` (and charmhub.io/mysql) it is quite surprising that the base detection logic would use now-closed channels to decide what series we might want, and also that it ignores that we requested `edge` which has a valid edge in focal.

[Guillermo Gonzalez (verterok)]

Hi,

This might be related to how stable is a special risk, e.g: https://bugs.launchpad.net/snapstore-server/+bug/1976301
Charmhub will use the information of the last released revision to latest/stable for some operations, e.g: suggestin bases.
While I look into this, a workaround (not tested) might be to update this meatdata by releasing the current revision on edge to stable and then closing stable. This should trigger the required update.

Cheers

[John A Meinel (jameinel)]

Hi Guillermo, you marked this as Fix Released, but I see the same behavior in another charm that has a Focal vs Jammy release:

https://bugs.launchpad.net/juju/+bug/1996590

When I do the query, I see that it is, indeed, returning the possibility of Jammy, but it is not selecting that as the default base:
$ curl -X POST -H 'Content-type: application/json' -s https://api.charmhub.io/v2/charms/refresh -d '{"context": [], "actions": [{"name": "nginx-ingress-integrator", "base": {"architecture": "amd64", "name": "NA", "channel": "NA"}, "action": "install", "channel": "edge", "instance-key": "a-test"}]}' | jq .
{
  "error-list": [],
  "results": [
    {
      "charm": null,
      "error": {
        "code": "invalid-charm-base",
        "extra": {
          "default-bases": [
            {
              "architecture": "amd64",
              "channel": "20.04",
              "name": "ubuntu"
            },
            {
              "architecture": "amd64",
              "channel": "22.04",
              "name": "ubuntu"
            }
          ]
        },
        "message": "Instance key 'a-test' invalid 'base' in 'install' for charm_id=aQVXlrbsLWBTiAiQF0lilvUFolDOpKaE name=nginx-ingress-integrator"
      },
      "id": "aQVXlrbsLWBTiAiQF0lilvUFolDOpKaE",
      "instance-key": "a-test",
      "name": "nginx-ingress-integrator",
      "released-at": null,
      "result": "error"
    }
  ]
}

Maybe this is just a misunderstanding on my part about base sorting order. I thought that it was in reverse descending LTS releases, which means that 22.04 should sort before 20.04.
But when I query a different charm (juju-controller) that does have a stable 22.04 release, it also seems to prefer focal to jammy.
(Though when I query something with a very long history, but not yet Jammy, I do see the reverse descending LTS order).

[Daniel Manrique (roadmr)]

Hi John,

Focal vs. Jammy preference is most likely actually https://bugs.launchpad.net/snapstore-server/+bug/1995483.