inconsistent base ordering in error response hints

[Heather Lanigan]

There is inconsistent release ordering in error response hints depending if the error is revision-not-found or invalid-charm-base.

Ordering of invalid-charm-base is preferred, as the first choice is what is preferred.

Ordering in revision-not-found starts with the oldest release of ubuntu and goes to the newest.

Having a consistent order is needed for juju. Grouping the charm channels supported by a base channel is fine. We'd want the top of the release list to be the preferred base.

curl -X POST https://api.snapcraft.io/v2/charms/refresh -d '{"context":[],"actions":[{"action":"install","instance-key":"a5d51df0-c3d6-4fce-88dd-5b554e5f6e87","name":"ubuntu","channel":"stable","base":{"architecture":"amd64","name":"NA","channel":"NA"}}]}' -H "User-Agent: charm-testing" -H "Content-Type: app[D[Dlication/json"
{"error-list":[],"results":[{"charm":null,"error":{"code":"invalid-charm-base","extra":{"default-bases":[{"architecture":"amd64","channel":"20.04","name":"ubuntu"},{"architecture":"amd64","channel":"18.04","name":"ubuntu"},{"architecture":"amd64","channel":"16.04","name":"ubuntu"},{"architecture":"amd64","channel":"20.10","name":"ubuntu"}]},"message":"Instance key 'a5d51df0-c3d6-4fce-88dd-5b554e5f6e87' invalid 'base' in 'install' for charm_id=DksXQKAQTZfsUmBAGanZAhpoS4dpmXel name=ubuntu"},"id":"DksXQKAQTZfsUmBAGanZAhpoS4dpmXel","instance-key":"a5d51df0-c3d6-4fce-88dd-5b554e5f6e87","name":"ubuntu","released-at":null,"result":"error"}]}

curl -X POST https://api.snapcraft.io/v2/charms/refresh -d '{"context":[],"actions":[{"action":"install","instance-key":"a5d51df0-c3d6-4fce-88dd-5b554e5f6e87","name":"ubuntu","channel":"stable","base":{"architecture":"amd64","name":"ubuntu","channel":"21.10"}}]}' -H "User-Agent: charm-testing" -H "Content-Type: app[D[Dlication/json"
{"error-list":[],"results":[{"charm":{"id":"DksXQKAQTZfsUmBAGanZAhpoS4dpmXel","license":"","name":"ubuntu","publisher":{"display-name":"charmers","id":"K08R1CtJYwkNQlpljSBruPKfHgTtAKFK","username":"charmers","validation":"unproven"},"summary":"A pristine Ubuntu Server","type":"charm"},"error":{"code":"revision-not-found","extra":{"releases":[{"base":{"architecture":"amd64","channel":"12.04","name":"ubuntu"},"channel":"stable"},{"base":{"architecture":"amd64","channel":"12.04","name":"ubuntu"},"channel":"candidate"},{"base":{"architecture":"amd64","channel":"12.04","name":"ubuntu"},"channel":"beta"},{"base":{"architecture":"amd64","channel":"12.04","name":"ubuntu"},"channel":"edge"},{"base":{"architecture":"amd64","channel":"14.04","name":"ubuntu"},"channel":"stable"},{"base":{"architecture":"amd64","channel":"14.04","name":"ubuntu"},"channel":"candidate"},{"base":{"architecture":"amd64","channel":"14.04","name":"ubuntu"},"channel":"beta"},{"base":{"architecture":"amd64","channel":"14.04","name":"ubuntu"},"channel":"edge"},{"base":{"architecture":"amd64","channel":"15.04","name":"ubuntu"},"channel":"stable"},{"base":{"architecture":"amd64","channel":"15.04","name":"ubuntu"},"channel":"candidate"},{"base":{"architecture":"amd64","channel":"15.04","name":"ubuntu"},"channel":"beta"},{"base":{"architecture":"amd64","channel":"15.04","name":"ubuntu"},"channel":"edge"},{"base":{"architecture":"amd64","channel":"16.04","name":"ubuntu"},"channel":"stable"},{"base":{"architecture":"amd64","channel":"16.04","name":"ubuntu"},"channel":"candidate"},{"base":{"architecture":"amd64","channel":"16.04","name":"ubuntu"},"channel":"beta"},{"base":{"architecture":"amd64","channel":"16.04","name":"ubuntu"},"channel":"edge"},{"base":{"architecture":"amd64","channel":"17.10","name":"ubuntu"},"channel":"stable"},{"base":{"architecture":"amd64","channel":"17.10","name":"ubuntu"},"channel":"candidate"},{"base":{"architecture":"amd64","channel":"17.10","name":"ubuntu"},"channel":"beta"},{"base":{"architecture":"amd64","channel":"17.10","name":"ubuntu"},"channel":"edge"},{"base":{"architecture":"amd64","channel":"18.04","name":"ubuntu"},"channel":"stable"},{"base":{"architecture":"amd64","channel":"18.04","name":"ubuntu"},"channel":"candidate"},{"base":{"architecture":"amd64","channel":"18.04","name":"ubuntu"},"channel":"beta"},{"base":{"architecture":"amd64","channel":"18.04","name":"ubuntu"},"channel":"edge"},{"base":{"architecture":"amd64","channel":"18.10","name":"ubuntu"},"channel":"stable"},{"base":{"architecture":"amd64","channel":"18.10","name":"ubuntu"},"channel":"candidate"},{"base":{"architecture":"amd64","channel":"18.10","name":"ubuntu"},"channel":"beta"},{"base":{"architecture":"amd64","channel":"18.10","name":"ubuntu"},"channel":"edge"},{"base":{"architecture":"amd64","channel":"19.04","name":"ubuntu"},"channel":"stable"},{"base":{"architecture":"amd64","channel":"19.04","name":"ubuntu"},"channel":"candidate"},{"base":{"architecture":"amd64","channel":"19.04","name":"ubuntu"},"channel":"beta"},{"base":{"architecture":"amd64","channel":"19.04","name":"ubuntu"},"channel":"edge"},{"base":{"architecture":"amd64","channel":"20.04","name":"ubuntu"},"channel":"stable"},{"base":{"architecture":"amd64","channel":"20.04","name":"ubuntu"},"channel":"candidate"},{"base":{"architecture":"amd64","channel":"20.04","name":"ubuntu"},"channel":"beta"},{"base":{"architecture":"amd64","channel":"20.04","name":"ubuntu"},"channel":"edge"},{"base":{"architecture":"amd64","channel":"20.10","name":"ubuntu"},"channel":"stable"},{"base":{"architecture":"amd64","channel":"20.10","name":"ubuntu"},"channel":"candidate"},{"base":{"architecture":"amd64","channel":"20.10","name":"ubuntu"},"channel":"beta"},{"base":{"architecture":"amd64","channel":"20.10","name":"ubuntu"},"channel":"edge"}]},"message":"No revision was found in the Store."},"id":"DksXQKAQTZfsUmBAGanZAhpoS4dpmXel","instance-key":"a5d51df0-c3d6-4fce-88dd-5b554e5f6e87","name":"ubuntu","released-at":null,"result":"error"}]}

[Heather Lanigan (hmlanigan)]

One place this is a challenge for juju is when a user deploys a charm to a series not supported by the charm. Which juju allows with a --force flag.

[Daniel Manrique (roadmr)]

AIEE MY EYES

for invalid base:

{
  "error-list": [],
  "results": [
    {
      "charm": null,
      "error": {
        "code": "invalid-charm-base",
        "extra": {
          "default-bases": [
            {
              "architecture": "amd64",
              "channel": "20.04",
              "name": "ubuntu"
            },
            {
              "architecture": "amd64",
              "channel": "18.04",
              "name": "ubuntu"
            },
            {
              "architecture": "amd64",
              "channel": "16.04",
              "name": "ubuntu"
            },
            {
              "architecture": "amd64",
              "channel": "20.10",
              "name": "ubuntu"
            }
          ]
        },
        "message": "Instance key 'a5d51df0-c3d6-4fce-88dd-5b554e5f6e87' invalid 'base' in 'install' for charm_id=DksXQKAQTZfsUmBAGanZAhpoS4dpmXel name=ubuntu"
      },
      "id": "DksXQKAQTZfsUmBAGanZAhpoS4dpmXel",
      "instance-key": "a5d51df0-c3d6-4fce-88dd-5b554e5f6e87",
      "name": "ubuntu",
      "released-at": null,
      "result": "error"
    }
  ]
}

For revision not found:

{
  "error-list": [],
  "results": [
    {
      "charm": {
        "id": "DksXQKAQTZfsUmBAGanZAhpoS4dpmXel",
        "license": "",
        "name": "ubuntu",
        "publisher": {
          "display-name": "charmers",
          "id": "K08R1CtJYwkNQlpljSBruPKfHgTtAKFK",
          "username": "charmers",
          "validation": "unproven"
        },
        "summary": "A pristine Ubuntu Server",
        "type": "charm"
      },
      "error": {
        "code": "revision-not-found",
        "extra": {
          "releases": [
            {
              "base": {
                "architecture": "amd64",
                "channel": "12.04",
                "name": "ubuntu"
              },
              "channel": "stable"
            },
            {
              "base": {
                "architecture": "amd64",
                "channel": "12.04",
                "name": "ubuntu"
              },
              "channel": "candidate"
            },
            {
              "base": {
                "architecture": "amd64",
                "channel": "12.04",
                "name": "ubuntu"
              },
              "channel": "beta"
            },
            {
              "base": {
                "architecture": "amd64",
                "channel": "12.04",
                "name": "ubuntu"
              },
              "channel": "edge"
            },
            {
              "base": {
                "architecture": "amd64",
                "channel": "14.04",
                "name": "ubuntu"
              },
              "channel": "stable"
            },
            {
              "base": {
                "architecture": "amd64",
                "channel": "14.04",
                "name": "ubuntu"
              },
              "channel": "candidate"
            },
            {
              "base": {
                "architecture": "amd64",
                "channel": "14.04",
                "name": "ubuntu"
              },
              "channel": "beta"
            },
            {
           ...

[Jonathan Hartley (tartley)]

In the 200 /refresh response described above, containing a 'revision-not-found' error, the list of revisions in results.error.extra.releases includes all releases made by the specified charm. It is included to show the caller what releases exist, to inform them about why none of them matched the caller's request. Presumably, therefore, every release included in that list fails to match the caller's request by at least one of:

* ubuntu series (eg 12.04 vs 21.10)
* channel (eg edge vs stable)
* architecture (eg arm64 vs amd64)

We can easily change the ordering of releases in the response to LTS followed by descending ubuntu series. This will prevent Juju in this scenario from installing the release for the oldest ubuntu series.

But, done naively, we have no assurance that the first release by this criteria matches the channel or architecture requested by the caller. If my understanding is correct, when Juju with '--force' chooses the first release in the list, it could still potentially deploy charms from incorrect channels (e.g stable, when beta was requested), or with different than requested architectures.

I guess to resolve this, I need to ask precisely what the semantics of '--force' is intended to be. (e.g. does it mean, ignore a failure to find a charm for the specified ubuntu series, but uphold the restriction on channel and architecture)

We could implement this logic in the /refresh endpoint itself, by ordering or filtering the releases to match this criteria. But to me, this feels like baking the behavior of Juju's '--force' into the endpoint. I'd prefer if this logic was localized in Juju itself (e.g. given the full list of all releases in the response, Juju would need to iterate through them to choose the one it wants, rather than just picking the first.)

Notably, if we did this, then changing the current ordering of the response is no longer necessary, although I appreciate it might be a helpful courtesy if it gives us a quick fix to prevent installs of charms for incorrectly very old ubuntu series.

Thoughts welcome.

[Jonathan Hartley (tartley)]

In discussion with Heather: In charmstore, --force was intended to deploy on a series the charm did not explicitly support. Is fine in charmhub for deploy. But for upgrade: platform, includes series, is saved when charm is deployed. When requesting refresh, juju echoes these values from the previously installed rev. If an exact match doesn't exist, then juju gets the above error, and if '--force' has been used, juju grabs the first entry from the response's list of releases **that matches the requested channel and architecture.**

This means Heather's original suggestion, of just changing the ordering to LTS-followed-by-decreasing-series, will produce absolutely optimal behavior, so I should just merge and release that.

[Jonathan Hartley (tartley)]

Fixed in MP https://code.launchpad.net/~tartley/snapdevicegw/+git/snapdevicegw/+merge/416752