Upgrade from victoria -> wallaby (15.2 -> 16.2) doesn't happen on config-changed

[Alex Kavanagh]

Deploy a cloud with cloud:focal-victoria.

Upgrade ceph-mon with: "juju config ceph-mon source=cloud:focal-wallaby"

config-changed runs.

ceph packages do not get upgraded.

Juju status afterwards:

Model Controller Cloud/Region Version SLA Timestamp
zaza-698191b355e9 tinwood2-serverstack serverstack/serverstack 2.8.10 unsupported 11:16:24Z

App Version Status Scale Charm Store Rev OS Notes
ceph-mon 15.2.14 active 3 ceph-mon jujucharms 483 ubuntu

Unit Workload Agent Machine Public address Ports Message
ceph-mon/0* active idle 0 172.20.1.154 Unit is ready and clustered
ceph-mon/1 active idle 1 172.20.1.191 Unit is ready and clustered
ceph-mon/2 active idle 2 172.20.1.177 Unit is ready and clustered

Machine State DNS Inst id Series AZ Message
0 started 172.20.1.154 80940871-e232-4132-be9b-d71947387f76 focal nova ACTIVE
1 started 172.20.1.191 a2a8a919-9a69-4b79-bd11-72647387add3 focal nova ACTIVE
2 started 172.20.1.177 2e40c654-c73b-4bf9-b966-9e454329d060 focal nova ACTIVE

juju config ceph-mon source
cloud:focal-wallaby

on ceph-mon/0:

$ cat /etc/apt/sources.list.d/cloud-archive.list
# Ubuntu Cloud Archive
deb http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby mai

$ sudo apt list --upgradable
Listing... Done
libssl1.1/focal-updates 1.1.1f-1ubuntu2.9 amd64 [upgradable from: 1.1.1f-1ubuntu2.8]
open-vm-tools/focal-updates 2:11.3.0-2ubuntu0~ubuntu20.04.2 amd64 [upgradable from: 2:11.0.5-4]
openssl/focal-updates 1.1.1f-1ubuntu2.9 amd64 [upgradable from: 1.1.1f-1ubuntu2.8]

$ sudo apt update
Get:1 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby InRelease [8769 B]
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:3 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby/main amd64 Packages [155 kB]
Hit:4 http://nova.clouds.archive.ubuntu.com/ubuntu focal InRelease
Get:5 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:6 http://nova.clouds.archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:7 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1344 kB]
Fetched 1836 kB in 1s (1461 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
33 packages can be upgraded. Run 'apt list --upgradable' to see them.

sudo apt list --upgradable
Listing... Done
ceph-base/focal-updates 16.2.6-0ubuntu0.21.04.1~cloud0 amd64 [upgradable from: 15.2.14-0ubuntu0.20.04.1]
ceph-common/focal-updates 16.2.6-0ubuntu0.21.04.1~cloud0 amd64 [upgradable from: 15.2.14-0ubuntu0.20.04.1]
ceph-mds/focal-updates 16.2.6-0ubuntu0.21.04.1~cloud0 amd64 [upgradable from: 15.2.14-0ubuntu0.20.04.1]
ceph-mgr-modules-core/focal-updates 16.2.6-0ubuntu0.21.04.1~cloud0 all [upgradable from: 15.2.14-0ubuntu0.20.04.1]
ceph-mgr/focal-updates 16.2.6-0ubuntu0.21.04.1~cloud0 amd64 [upgradable from: 15.2.14-0ubuntu0.20.04.1]
ceph-mon/focal-updates 16.2.6-0ubuntu0.21.04.1~cloud0 amd64 [upgradable from: 15.2.14-0ubuntu0.20.04.1]
ceph-osd/focal-updates 16.2.6-0ubuntu0.21.04.1~cloud0 amd64 [upgradable from: 15.2.14-0ubuntu0.20.04.1]
ceph/focal-updates 16.2.6-0ubuntu0.21.04.1~cloud0 amd64 [upgradable from: 15.2.14-0ubuntu0.20.04.1]
...
radosgw/focal-updates 16.2.6-0ubuntu0.21.04.1~cloud0 amd64 [upgradable from: 15.2.14-0ubuntu0.20.04.1]

Thoughts:

1. The source was updated.
2. "apt update" was not run
3. "apt install" ceph packages was not called to upgrade the packages.

[James Page (james-page)]

# A dict of valid ceph upgrade paths. Mapping is old -> new
UPGRADE_PATHS = collections.OrderedDict([
    ('firefly', 'hammer'),
    ('hammer', 'jewel'),
    ('jewel', 'luminous'),
    ('luminous', 'mimic'),
    ('mimic', 'nautilus'),
    ('nautilus', 'octopus'),
])

# Map UCA codenames to ceph codenames
UCA_CODENAME_MAP = {
    'icehouse': 'firefly',
    'juno': 'firefly',
    'kilo': 'hammer',
    'liberty': 'hammer',
    'mitaka': 'jewel',
    'newton': 'jewel',
    'ocata': 'jewel',
    'pike': 'luminous',
    'queens': 'luminous',
    'rocky': 'mimic',
    'stein': 'mimic',
    'train': 'nautilus',
    'ussuri': 'octopus',
}

releases past ussuri need to be added to charms.ceph to support upgrades.

[Alex Kavanagh (ajkavanagh)]

More from debug-log:

unit-ceph-mon-0: 12:45:39 DEBUG unit.ceph-mon/0.juju-log Hardening function 'config_changed'
unit-ceph-mon-0: 12:45:39 DEBUG jujuc running hook tool "config-get" for ceph-mon/0-config-changed-2325196192277328743
unit-ceph-mon-0: 12:45:39 DEBUG jujuc running hook tool "juju-log" for ceph-mon/0-config-changed-2325196192277328743
unit-ceph-mon-0: 12:45:39 DEBUG unit.ceph-mon/0.juju-log No hardening applied to 'config_changed'
unit-ceph-mon-0: 12:45:39 DEBUG jujuc running hook tool "juju-log" for ceph-mon/0-config-changed-2325196192277328743
unit-ceph-mon-0: 12:45:39 INFO unit.ceph-mon/0.juju-log old_version: None
unit-ceph-mon-0: 12:45:39 DEBUG jujuc running hook tool "juju-log" for ceph-mon/0-config-changed-2325196192277328743
unit-ceph-mon-0: 12:45:39 INFO unit.ceph-mon/0.juju-log new_version: None

So it looks like the detection logic for versions is broken on victoria at least. old and new version detected as None.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/819444

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/21.10
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/819452

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/819444
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/9e5f668299f40c0423d5a210bf87adb2caab4ab9
Submitter: "Zuul (22348)"
Branch: master

commit 9e5f668299f40c0423d5a210bf87adb2caab4ab9
Author: Alex Kavanagh <email address hidden>
Date: Fri Nov 26 12:31:43 2021 +0000

    Fix get_mon_map() for octopus and later

    The "ceph mon_status" command seems to have disappeared on octopus and
    later, and is replaced by "ceph quorum_status". This changes the
    get_mon_map() function to detect the underlying ceph version and do the
    right thing.

    Note that the fix is actually in charm-helpers, and this has been
    manually synced into the charm [1].

    [1] https://github.com/juju/charm-helpers/pull/659

    Change-Id: I59cf6fc19cf2a91b0aef37059cdb0ed37379b5cb
    Closes-Bug: #1951094

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/819452
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/33b59af45244eee33768121d6e70ee4b55297525
Submitter: "Zuul (22348)"
Branch: stable/21.10

commit 33b59af45244eee33768121d6e70ee4b55297525
Author: Alex Kavanagh <email address hidden>
Date: Fri Nov 26 12:31:43 2021 +0000

    Fix get_mon_map() for octopus and later

    The "ceph mon_status" command seems to have disappeared on octopus and
    later, and is replaced by "ceph quorum_status". This changes the
    get_mon_map() function to detect the underlying ceph version and do the
    right thing.

    Note that the fix is actually in charm-helpers, and this has been
    manually synced into the charm [1].

    Note: also picked pyparsing pin to allow tests to run.

    Cherry-picked from master.

    [1] https://github.com/juju/charm-helpers/pull/659

    Change-Id: I59cf6fc19cf2a91b0aef37059cdb0ed37379b5cb
    Closes-Bug: #1951094

[Felipe Alencastro (falencastro)]

This issue is still happening on charm version 73 during upgrade from octopus to pacific (cloud:focal-victoria -> cloud:focal-wallaby). I guess this happens due to no upgrade path defined on https://github.com/openstack/charm-ceph-mon/blob/1a24e5edc5e00e6a7adccc6f444fc723b10d750e/lib/charms_ceph/utils.py#L3165

[Chris MacNaughton (chris.macnaughton)]

It looks like this was only really a partial fix, as it didn't add support for Pacific into the upgrade paths[1].

1: https://github.com/openstack/charm-ceph-mon/blob/stable/pacific/lib/charms_ceph/utils.py#L3169

[Chuan Li (lccn)]

charm version: ch:ceph-mon channel: quincy/stable revision: 149

'juju config ceph-mon source=cloud:focal-yoga' can not generate a correct file cloud-archive.list.The cloud-archive.list will be still with "focal-updates/xena",
Thus, upgrade from Pacific to Quincy will not happen. and strange unit logs are as following:

2023-02-04 11:48:20 INFO unit.ceph-mon/0.juju-log server.go:327 old_version: octopus
2023-02-04 11:48:20 INFO unit.ceph-mon/0.juju-log server.go:327 new_version: quincy
2023-02-04 11:48:20 ERROR unit.ceph-mon/0.juju-log server.go:327 Invalid upgrade path from octopus to quincy. Valid paths are: ['firefly -> hammer', 'hammer -> jewel', 'jewel -> luminous', 'luminous -> mimic', 'mimic -> nautilus', 'nautilus -> octopus', 'octopus -> pacific', 'pacific -> quincy']

steps:
1) deploy ceph-mon onto a focal from cs:-51
ceph is Octopus as expected

2) juju refresh --switch ch:ceph-mon --channel quincy/stable ceph-mon

3) juju config ceph-mon source=cloud:focal-victoria
ceph is still Octopus as expected

4) juju config ceph-mon source=cloud:focal-wallaby
ceph is Pacific as expected

4) juju config ceph-mon source=cloud:focal-xena
ceph is still Pacific as expected

5) juju config ceph-mon source=cloud:focal-yoga
ceph is NOT Quincy, but still Pacific, it's NOT expected.

Details: https://pastebin.canonical.com/p/gRqK6YNTsW/

[Chuan Li (lccn)]

1) deply ceph-mon/radosgw/osd with from charm store and focal(source=distro)
bundle https://pastebin.canonical.com/p/Rx42Dg5Cvy/

2) switch from cs to ch
juju refresh --switch ch:ceph-mon --channel octopus/stable ceph-mon
juju refresh --switch ch:ceph-radosgw --channel octopus/stable ceph-radosgw
juju refresh --switch ch:ceph-osd --channel octopus/stable ceph-osd

3) upgrade charm from octopus/stable to pacific/stable

juju refresh --channel pacific/stable ceph-mon
juju refresh --channel pacific/stable ceph-radosgw
juju refresh --channel pacific/stable ceph-osd

4) change UCA from distro to focal-wallaby

juju config ceph-mon source=cloud:focal-wallaby

juju run -a ceph-mon cat /etc/apt/sources.list.d/cloud-archive.list
cat: /etc/apt/sources.list.d/cloud-archive.list: No such file or directory

tail -f /var/log/juju/unit-ceph-mon-0.log
2023-06-04 15:38:50 INFO unit.ceph-mon/0.juju-log server.go:316 running ceph osd set noout
2023-06-04 15:38:50 INFO unit.ceph-mon/0.juju-log server.go:316 DEPRECATION WARNING: Function action_set is being removed : moved to function_set()
2023-06-04 15:39:16 INFO unit.ceph-mon/0.juju-log server.go:316 Updating status.
2023-06-04 15:39:17 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via explicit, bespoke hook script)
2023-06-04 15:40:38 INFO unit.ceph-mon/0.juju-log server.go:316 old_version: octopus
2023-06-04 15:40:38 INFO unit.ceph-mon/0.juju-log server.go:316 new_version: None
2023-06-04 15:40:38 ERROR unit.ceph-mon/0.juju-log server.go:316 Invalid upgrade path from octopus to None. Valid paths are: ['firefly -> hammer', 'hammer -> jewel', 'jewel -> luminous', 'luminous -> mimic', 'mimic -> nautilus', 'nautilus -> octopus']

juju status
Model Controller Cloud/Region Version SLA Timestamp
ceph mycontroller stsstack/stsstack 2.9.42 unsupported 15:53:16Z

App Version Status Scale Charm Channel Rev Exposed Message
ceph-mon 15.2.17 active 1 ceph-mon pacific/stable 152 no Unit is ready and clustered
ceph-osd 15.2.17 active 2 ceph-osd pacific/stable 548 no Unit is ready (1 OSD)
ceph-radosgw 15.2.17 active 1 ceph-radosgw pacific/stable 544 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
ceph-mon/0* active idle 0 10.5.0.145 Unit is ready and clustered
ceph-osd/0* active idle 1 10.5.1.212 Unit is ready (1 OSD)
ceph-osd/1 active idle 2 10.5.1.204 Unit is ready (1 OSD)
ceph-radosgw/0* active idle 3 10.5.3.235 80/tcp Unit is ready

Based on my last test [1], if I upgrade ceph-mon's charm from cs to quincy/stable directly, source=cloud:focal-wallaby can generate a correct cloud-archive.list file, cantaining Pacific UCA
But the charm upgrade path is not supported or recommended, as the official doc [2] mentions "The Ceph charms have been tested for single-step charm upgrades only (N+1)", cs:ceph-mon to quincy/stable is N+2(skip pacific/stable)

[1] https://bugs.launchpad.net/charm-ceph-mon/+bug/1951094/comm...

[Ian Marsh (drulgaard)]

Not sure if I'm doing something wrong, but when trying to upgrade a freshly-built Victoria to Wallaby (upgrade practice before production upgrade) and therefore Ceph from Octopus to Pacific, I first upgrade the ceph-mon charm from channel octopus/stable (rev 160) to pacific/stable (rev 152) and then reconfigure the charm's source from cloud:focal-victoria to cloud:focal-wallaby, and it fails to upgrade.

If I monkey-patch each /var/lib/juju/agents/unit-ceph-mon-*/charm/lib/charms_ceph/utils.py with...

3169a3170
> ('octopus', 'pacific'),
3186a3188,3189
> 'victoria': 'octopus',
> 'wallaby': 'pacific',

... in between the charm upgrade and the source reconfiguration, then it upgrades fine. I don't think this is new news, but I'm surprised to find it still missing in the latest pacific/stable charm.

[Ian Marsh (drulgaard)]

For the record, the ceph-mon charm on channel pacific/stable is now rev 178 and has the correct entries for pacific, meaning the patch above is no longer necessary for my upgrades.

[Felipe Reyes (freyes)]

This issue was fixed by https://github.com/openstack/charm-ceph-mon/commit/9cc367f55191b106b0f6d562c7f3e3e277513445#diff-9464afc6c59500f017fa1700cd114004944d02b3e6bc7a5f4e9e423b391b80caL3169

Merged on July 14th - https://review.opendev.org/c/openstack/charm-ceph-mon/+/888584