Juju fails to validate the charm OS series when --switch is used

[Alan Baghumian]

Hello OpenStack Team!

We realized this weekend that Vault 1.8/stable charm no longer supports Focal track.

$ juju info --series focal vault
name: vault
publisher: OpenStack Charmers
summary: a tool for managing secrets
description: |
  Vault secures, stores, and tightly controls access to
  tokens, passwords, certificates, API keys, and other
  secrets in modern computing. Vault handles leasing, key
  revocation, key rolling, and auditing. Through a unified
  API, users can access an encrypted Key/Value store and
  network encryption-as-a-service, or generate AWS IAM/STS
  credentials, SQL/NoSQL databases, X.509 certificates,
  SSH credentials, and more.
store-url: https://charmhub.io/vault
charm-id: u3vtXfBTJwhYCwwvkzCs0YqdCfsOBMdO
supports: jammy
tags: security
subordinate: false
relations:
  provides:
    certificates: tls-certificates
    nrpe-external-master: nrpe-external-master
    secrets: vault-kv
  requires:
    db: pgsql
    etcd: etcd
    ha: hacluster
    lb-provider: loadbalancer
    shared-db: mysql-shared
channels: |
  1.8/stable: –
  1.8/candidate: –
  1.8/beta: –
  1.8/edge: –
  latest/stable: –
  latest/candidate: –
  latest/beta: –
  latest/edge: 9bf2a4c 2022-08-02 (79) 46MB
  1.7/stable: ba1ffbf 2023-08-14 (178) 47MB
  1.7/candidate: ↑
  1.7/beta: ↑
  1.7/edge: ↑
  1.6/stable: 1648b97 2023-09-14 (182) 42MB
  1.6/candidate: ↑
  1.6/beta: ↑
  1.6/edge: ↑
  1.5/stable: 24b65b7 2023-09-14 (181) 42MB
  1.5/candidate: ↑
  1.5/beta: ↑
  1.5/edge: ↑

I have a Vault HA cluster on Focal/Yoga that was upgraded to 1.8 a few months ago and now it is not possible to receive refreshed charms:

$ juju status vault
Model Controller Cloud/Region Version SLA Timestamp
vstack home-lab-default Home-Lab/default 2.9.37 unsupported 10:01:24-07:00

App Version Status Scale Charm Channel Rev Exposed Message
lds-client-focal active 3 landscape-client stable 49 no System successfully registered
vault 1.8.8 active 3 vault 1.8/stable 179 no Unit is ready (active: true, mlock: disabled)
vault-hacluster active 3 hacluster 2.0.3/stable 113 no Unit is ready and clustered
vault-mysql-router 8.0.34 active 3 mysql-router 8.0/stable 90 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
vault/4* active idle 0/lxd/6 10.1.15.134 8200/tcp Unit is ready (active: true, mlock: disabled)
  lds-client-focal/28 active idle 10.1.15.134 System successfully registered
  vault-hacluster/4* active idle 10.1.15.134 Unit is ready and clustered
  vault-mysql-router/4* active idle 10.1.15.134 Unit is ready
vault/7 active idle 1/lxd/10 10.1.15.143 8200/tcp Unit is ready (active: false, mlock: disabled)
  lds-client-focal/814 active idle 10.1.15.143 System successfully registered
  vault-hacluster/7 active idle 10.1.15.143 Unit is ready and clustered
  vault-mysql-router/7 active idle 10.1.15.143 Unit is ready
vault/8 active idle 101/lxd/9 10.1.15.160 8200/tcp Unit is ready (active: false, mlock: disabled)
  lds-client-focal/821 active idle 10.1.15.160 System successfully registered
  vault-hacluster/8 active idle 10.1.15.160 Unit is ready and clustered
  vault-mysql-router/8 active idle 10.1.15.160 Unit is ready

Machine State Address Inst id Series AZ Message
0 started 10.1.8.31 os-controller-1 focal default Deployed
0/lxd/6 started 10.1.15.134 juju-b096f0-0-lxd-6 focal default Container started
1 started 10.1.8.33 os-controller-3 focal default Deployed
1/lxd/10 started 10.1.15.143 juju-b096f0-1-lxd-10 focal default Container started
101 started 10.1.8.32 os-controller-2 focal default Deployed
101/lxd/9 started 10.1.15.160 juju-b096f0-101-lxd-9 focal default Container started

Would it be possible to re-add Focal to 1.8/stable channel please?

Thanks much,
Alan

[Dagmawi Biru (dagbiru)]

I see this commit shows the removal of focal from the series list from the 1.8/stable branch in the git repo. Not sure if this is the commit in question but just wanted to chime in:

https://opendev.org/openstack/charm-vault/commit/0b7d04127996bd232e69f54d00e345abbd4bbc0e#diff-4d69e557ec97c6c28bb94c65e81042d669f3f034

[Felipe Reyes (freyes)]

hi Allan,

This is intentional, the vault charm available in the 1.8/stable channel is meant to support only
Jammy[0], if you are looking for a charm with focal support you can use the 1.7/stable channel.

Best,

[0]
https://github.com/openstack-charmers/charmed-openstack-info/blob/main/charmed_openstack_info/data/lp-builder-config/misc.yaml#L210

[Felipe Reyes (freyes)]

Here you can find details on what channels are meant to support what Ubuntu series
https://docs.openstack.org/charm-guide/latest/project/charm-delivery.html#tracks-for-the-openstack-charms-project

[Alan Baghumian (alanbach)]

Hi @Felipe!

Hope all is well!

What is interesting that I have been able to refresh to 1.8/stable somehow on Focal w/o using --force

Trying to figure out how!

Best,
Alan

[Alan Baghumian (alanbach)]

Here you go!!!!

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
vault-test home-lab-maas-default home-lab-maas/default 2.9.44 unsupported 11:14:29-07:00

App Version Status Scale Charm Channel Rev Exposed Message
vault blocked 1 vault latest/edge 79 no 'shared-db' or 'db' missing

Unit Workload Agent Machine Public address Ports Message
vault/0* blocked idle 0 10.1.8.41 'shared-db' or 'db' missing

Machine State Address Inst id Series AZ Message
0 started 10.1.8.41 generic-1 focal default Deployed

$ juju refresh vault --switch ch:vault --channel 1.7/stable
Added charm-hub charm "vault", revision 178 in channel 1.7/stable, to the model
Leaving endpoints in "alpha": access, certificates, cluster, db, etcd, external, ha, lb-provider, nrpe-external-master, secrets, shared-db

$ juju refresh vault --switch ch:vault --channel 1.8/stable
Added charm-hub charm "vault", revision 183 in channel 1.8/stable, to the model
Leaving endpoints in "alpha": access, certificates, cluster, db, etcd, external, ha, lb-provider, nrpe-external-master, secrets, shared-db

As you can see, my Vault is on Focal, shouldn't juju stop me here?

[Alan Baghumian (alanbach)]

Happily upgraded!

$ juju status vault
Model Controller Cloud/Region Version SLA Timestamp
vault-test home-lab-maas-default home-lab-maas/default 2.9.44 unsupported 11:17:45-07:00

App Version Status Scale Charm Channel Rev Exposed Message
vault blocked 1 vault 1.8/stable 183 no 'shared-db' or 'db' missing

Unit Workload Agent Machine Public address Ports Message
vault/0* blocked executing 0 10.1.8.41 (upgrade-charm) 'shared-db' or 'db' missing

Machine State Address Inst id Series AZ Message
0 started 10.1.8.41 generic-1 focal default Deployed

[Felipe Reyes (freyes)]

Adding a task for juju into this bug to get their thoughts on what's described in comment #5 and #6.

[Alex Kavanagh (ajkavanagh)]

@alanbach I think you got 'lucky' that the 1.8 charm is still working on focal. :) It is actually built for py3.10 + all the modules in the venv are for py3.10; so it's luck that it's still working on focal (py3.8).

We only support 1.8 on jammy, though, so if it goes wrong on focal then we'd probably not be able to help you. The 1.7 series is supported on focal and will continue to do so until focal's EOL.

You could dist-upgrade your focal unit for vault to jammy and then 1.8 will be supported on that.

[Heather Lanigan (hmlanigan)]

It's a juju bug that `juju refresh vault --switch ch:vault --channel 1.8/stable` worked here. Interestingly `juju refresh vault --channel 1.8/stable` will fail on the operating system.

When we're validating the switch, we are forgetting to check the operating system.

[Alan Baghumian (alanbach)]

Cool! So this bug report turned out not to be completely useless after all! :-D

[Alan Baghumian (alanbach)]

Just updated the bug report title so it is appropriate.

[Ian Booth (wallyworld)]

The next 2.9.46 candidate release will not include a fix for this bug and we don't plan on any more 2.9 releases. As such it is being removed from its 2.9 milestone.

If the bug is still important to you, let us know and we can consider it for inclusion on a 3.x milestone.

[Alan Baghumian (alanbach)]

@Ian It is a validation flaw that needs to be fixed!

Lack of this validation can easily lead to a broken charm due to OS incompatibility.

If there are no plans for 2.9.x (If I am correct it will still be supported until 2028), please let's get this fixed in the newer versions.

[John A Meinel (jameinel)]

This doesn't fall into the "Security and Critical" fixes only in our 2.9 support. Unless we were seeing many people hitting this, we will be fixing it in our bugfix supported releases, but not in 2.9.