Inconsistent charm revisions in --series focal vs default-series=focal

[Nobuto Murata]

$ juju version
2.9.28-ubuntu-amd64

Different revisions of charms will be picked between `--series focal` and default-series=focal without `--series focal`.

[latest - with explicit series]

$ juju deploy neutron-api --series focal --channel latest/edge neutron-api-edge
Located charm "neutron-api" in charm-hub, revision 520
Deploying "neutron-api-edge" from charm-hub charm "neutron-api", revision 520 in channel edge

^^^ revision 520

[latest - with implicit series through default-series]

$ juju model-config default-series
focal

$ juju deploy neutron-api --channel latest/edge neutron-api-edge-without-explicit-series
Located charm "neutron-api" in charm-hub, revision 498
Deploying "neutron-api-edge-without-explicit-series" from charm-hub charm "neutron-api", revision 498 in channel candidate

^^^ revision *498* <- unexpected revision

[xena - with explicit series]

$ juju deploy neutron-api --series focal --channel xena/edge neutron-api-xena-edge
Located charm "neutron-api" in charm-hub, revision 522
Deploying "neutron-api-xena-edge" from charm-hub charm "neutron-api", revision 522 in channel xena/edge

^^^ revision 522

[xena - with implicit series through default-series]

$ juju model-config default-series
focal

$ juju deploy neutron-api --channel xena/edge neutron-api-xena-edge-without-explicit-series --to 0
Located charm "neutron-api" in charm-hub, revision 501
Deploying "neutron-api-xena-edge-without-explicit-series" from charm-hub charm "neutron-api", revision 501 in channel xena/edge

^^^ revision *501* <- unexpected revision

[Nobuto Murata (nobuto)]

10:05:29 TRACE juju.rpc.jsoncodec codec.go:228 -> {"request-id":3,"type":"Charms","version":4,"request":"ResolveCharms","params":{"resolve":[{"reference":"ch:neutron-api","charm-origin":{"source":"charm-hub","type":"","id":"","risk":"edge","architecture":"amd64"}}]}}
10:05:31 TRACE juju.rpc.jsoncodec codec.go:122 <- {"request-id":3,"response":{"Results":[{"url":"ch:amd64/xenial/neutron-api-498","charm-origin":{"source":"charm-hub","type":"charm","id":"","risk":"candidate","revision":498,"architecture":"amd64","os":"ubuntu","series":"xenial"},"supported-series":["xenial","bionic","focal","groovy","hirsute","impish"]}]}}

[Nobuto Murata (nobuto)]

Interestingly enough, `juju download` gets the revision 520 properly instead of 498.

"revision":520,"summary":"OpenStack Networking API service","type":"charm","version":"520"},"effective-channel":"edge","id":"cG72uT7DkN8fkgMAprtyalAgyhBHKw4G","instance-key":"9cd8b618-fc5f-4df3-8419-2ebc2b0600a4","name":"neutron-api","released-at":"2022-03-21T11:20:31.376986+00:00","result":"install"}]}

[Alex Kavanagh (ajkavanagh)]

Just as some more information, from the "charmcraft status neutron-api" for 20.04 has this:

          ubuntu 20.04 (all) stable 501 501 policyd-override (r0)
                                  candidate 498 498 policyd-override (r0)
                                  beta ↑ ↑ ↑
                                  edge ↑ ↑ ↑
          ubuntu 20.04 (amd64) stable - - -
                                  candidate - - -
                                  beta - - -
                                  edge 520 520

The "(all)" is the import from the charmstore, whereas the "(amd64)" is a more recent upload via a recipe from launchpad. I'm guessing the charm resolution process is using (all) rather than the more specific amd64?

[Nobuto Murata (nobuto)]

Subscribing ~field-high. Picking the right channel is critical to adopt the new workflow with charmhub.

[Edward Hope-Morley (hopem)]

I tried deploying using a bundle and it seems to work:

$ juju status neutron-api
Model Controller Cloud/Region Version SLA Timestamp
xenatest hopem stsstack/stsstack 2.9.27 unsupported 10:33:46Z

App Version Status Scale Charm Channel Rev Exposed Message
neutron-api 19.1.0 active 1 neutron-api xena/edge 522 no Unit is ready
neutron-api-mysql-router 8.0.28 active 1 mysql-router 8.0.19/edge 15 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
neutron-api/0* active idle 6 10.5.0.23 9696/tcp Unit is ready
  neutron-api-mysql-router/0* active idle 10.5.0.23 Unit is ready

Machine State DNS Inst id Series AZ Message
6 started 10.5.0.23 655d562e-82d8-4e02-b8af-42466d13fdb2 focal nova ACTIVE

[Edward Hope-Morley (hopem)]

But I also get broken behaviour when using the deploy command directly e.g.

$ juju deploy --channel xena/edge neutron-api
Located charm "neutron-api" in charm-hub, revision 501
Deploying "neutron-api" from charm-hub charm "neutron-api", revision 501 in channel xena/edge

Note that in this case it chose version 501 when in the previous one it chose 522 and in fact this one is correct:

$ juju info neutron-api| grep xena/edge
  xena/edge: 501 2022-03-04 (501) 483kB

There is in fact no version 522 in juju info:

$ juju info neutron-api| grep 522

[Edward Hope-Morley (hopem)]

Further testing has shown that the problem appears to be due to the fact that juju is not properly using the default model series i.e. I have:

$ juju model-config| grep series
default-series default focal

Yet juju deploy does not use focal and I if set series explicitly it works i.e.

$ juju deploy --series=focal --channel xena/edge neutron-api
Located charm "neutron-api" in charm-hub, revision 522

And 522 is correct and the extra issue here is that we can see 522 is charmcraft status neutron-api but not juju info neutron-api. So it appears there are two critical issues here.

[Edward Hope-Morley (hopem)]

It feels like the problem is on the client side because while it is detecting the correct series in both cases:

$ juju --debug deploy --channel xena/edge neutron-api 2>&1| grep series_selector
11:20:26 INFO juju.cmd.juju.application.deployer series_selector.go:83 with the configured model default series "focal"

$ juju --debug deploy --series focal --channel xena/edge neutron-api 2>&1| grep series_selector
 11:20:37 INFO juju.cmd.juju.application.deployer series_selector.go:141 with the user specified series "focal"

Only the second method works and actually uses series focal.

[Edward Hope-Morley (hopem)]

A bit more testing shows that using juju upgrade-charm does not cause the version to regress BUT the following will:

  * juju deploy bundle.yaml where bundle contains series: <series>
  * remove series: <series> from the bundle
  * juju deploy bundle.yaml (a common way to upgrade all charms)

So this could be an issue for environments that used juju deploy --series <series> bundle.yaml who subsequently run juju deploy bundle.yaml i.e. without the series (which used to work).

[Nobuto Murata (nobuto)]

I updated the title and summary per request. And I've separated out the juju info issue to:
https://bugs.launchpad.net/juju/+bug/1967850

[Heather Lanigan (hmlanigan)]

How charms work with charmhub and charmstore are slightly different. With charmstore, juju was fully in control of the series selection. With charmhub, juju relies on base data supplied by charmhub.

It appears that the issue deploying the neutron-api charm from the xena/edge series is related to the order of the base data provided to juju. Chatting with the charmhub people on this.

Juju needs to find a way to deploy a charm more in line with user expectations. Inconsistent data keeps returning to cause issues in deployment.

One idea is to move away from the idea that charm defines an ordered list of preferred series to deploy with and always attempt deploy based on the model-series.

[Heather Lanigan (hmlanigan)]

@hopem, can you please provide a link to the bundle you were using in #9?

There should be a check in the bundle handling to prevent charm downgrade when a bundle is redeployed. Sounds like it needs to be expanded.

[Edward Hope-Morley (hopem)]

@hmlanigan sure:

$ juju add-model chtest
Added 'chtest' model on stsstack/stsstack with credential 'hopem' for user 'admin'
$ cat bundle.yaml
series: focal
applications:
  neutron-api:
    num_units: 1
    charm: ch:neutron-api
    channel: xena/edge
$ juju deploy ./bundle.yaml
Located charm "neutron-api" in charm-hub, channel xena/edge
Executing changes:
- upload charm neutron-api from charm-hub for series focal from channel xena/edge with architecture=amd64
- deploy application neutron-api from charm-hub on focal with xena/edge
  added resource policyd-override
- add unit neutron-api/0 to new machine 0
Deploy of bundle completed.
$ juju status neutron-api| grep Rev -A 1
App Version Status Scale Charm Channel Rev Exposed Message
neutron-api waiting 0/1 neutron-api xena/edge 522 no waiting for machine
$ vi bundle.yaml
$ cat bundle.yaml
#series: focal
applications:
  neutron-api:
    num_units: 1
    charm: ch:neutron-api
    channel: xena/edge
$ juju deploy ./bundle.yaml
Located charm "neutron-api" in charm-hub, channel xena/edge
Executing changes:
- upload charm neutron-api from charm-hub from channel xena/edge with architecture=amd64
- upgrade neutron-api from charm-hub using charm neutron-api from channel xena/edge
  added resource policyd-override
- set constraints for neutron-api to ""
Deploy of bundle completed.
$ juju status neutron-api| grep Rev -A 1
App Version Status Scale Charm Channel Rev Exposed Message
neutron-api waiting 0/1 neutron-api xena/edge 501 no waiting for machine

[Edward Hope-Morley (hopem)]

note that with series in bundle i get rev 522, with series removed i get rev 501

[Heather Lanigan (hmlanigan)]

Related to this issue, and juju info not returning the expected data. I've implemented --dry-run for juju deploy of charms. Previously it only worked for bundles.

https://github.com/juju/juju/pull/13950

[Peter Matulis (petermatulis)]

I've also hit this. If one is able to specify a default series for a model then it is really important that this be respected.

[Nobuto Murata (nobuto)]

I opened a separate bug with a slightly different symptoms. It might be a variant of this one though:
"juju deploy doesn't pick one default series, focal sometimes then jammy other times"
https://bugs.launchpad.net/juju/+bug/1998896