Comment 3 for bug 1923200

In summary: I think this is likely a side effect of calling config-get too late in the previous hook execution on ceph-osd/2, capturing the updated config value and writing it to .juju-persistent-config (that's how inter-hook-execution config changes are tracked in charmhelpers). As a result the next hook execution doesn't think that the value has changed.

This is due to the behavior in Juju where the result of a config-get call gets persisted at the time when a unit calls config-get rather than at the beginning of a hook execution (this is by design in Juju).

Specifically in the ceph-osd charm case the "source" config key value is used to determine the old version and compare it to the new version.

My suggestion would be to take the installed version into account here rather than just relying on the previous config value persisted by charm-helpers in the previous hook execution.
https://github.com/openstack/charm-ceph-osd/blob/43c93d9749d49afc991b3325a96a4ccc4def75f8/hooks/ceph_hooks.py#L134-L139
https://github.com/openstack/charm-ceph-osd/blob/43c93d9749d49afc991b3325a96a4ccc4def75f8/lib/charms_ceph/utils.py#L3179-L3187

https://github.com/openstack/charm-ceph-osd/blob/43c93d9749d49afc991b3325a96a4ccc4def75f8/hooks/charmhelpers/core/hookenv.py#L392 (config.previous)
https://github.com/openstack/charm-ceph-osd/blob/43c93d9749d49afc991b3325a96a4ccc4def75f8/hooks/charmhelpers/core/hookenv.py#L345 (.juju-persistent-config)

More info from the deployment below:

ubuntu@playground-cpe-9f305f1c-537b-477e-afca-1ac364a24afd:~$ juju run --application ceph-osd 'ceph-osd --version'
- Stdout: |
    ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
  UnitId: ceph-osd/1
- Stdout: |
    ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
  UnitId: ceph-osd/2
- Stdout: |
    ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
  UnitId: ceph-osd/5
- Stdout: |
    ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
  UnitId: ceph-osd/0
- Stdout: |
    ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
  UnitId: ceph-osd/3
- Stdout: |
    ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
  UnitId: ceph-osd/4
- Stdout: |
    ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
  UnitId: ceph-osd/6

juju ssh ceph-osd/2

/var/log/juju/unit-ceph-osd-2.log
# ...
2021-04-09 12:47:28 DEBUG juju.worker.uniter.operation executor.go:132 executing operation "run config-changed hook" for ceph-osd/2
2021-04-09 12:47:28 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running config-changed hook
2021-04-09 12:47:28 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-ceph-osd-2/agent.socket <nil>}
2021-04-09 12:47:28 DEBUG jujuc server.go:211 running hook tool "juju-log" for ceph-osd/2-config-changed-6219614913432438912
2021-04-09 12:47:28 DEBUG juju-log Hardening function 'config_changed'
2021-04-09 12:47:28 DEBUG jujuc server.go:211 running hook tool "config-get" for ceph-osd/2-config-changed-6219614913432438912
2021-04-09 12:47:28 DEBUG jujuc server.go:211 running hook tool "juju-log" for ceph-osd/2-config-changed-6219614913432438912
2021-04-09 12:47:28 DEBUG juju-log No hardening applied to 'config_changed'
2021-04-09 12:47:28 DEBUG jujuc server.go:211 running hook tool "juju-log" for ceph-osd/2-config-changed-6219614913432438912
2021-04-09 12:47:28 INFO juju-log old_version: mimic
2021-04-09 12:47:28 DEBUG jujuc server.go:211 running hook tool "juju-log" for ceph-osd/2-config-changed-6219614913432438912
2021-04-09 12:47:28 INFO juju-log new_version: mimic
2021-04-09 12:47:28 DEBUG jujuc server.go:211 running hook tool "juju-log" for ceph-osd/2-config-changed-6219614913432438912
2021-04-09 12:47:28 WARNING juju-log Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
2021-04-09 12:47:29 DEBUG jujuc server.go:211 running hook tool "juju-log" for ceph-osd/2-config-changed-6219614913432438912
2021-04-09 12:47:29 ERROR juju-log Invalid upgrade path from mimic to mimic. Valid paths are: ['firefly -> hammer', 'hammer -> jewel', 'jewel -> luminous', 'luminous -> mimic', 'mimic -> nautilus', 'nautilus -> octopus']

root@obayifo:~# juju-run -u ceph-osd/2 -- config-get
aa-profile-mode: complain
autotune: false
bdev-enable-discard: auto
bluestore: true
bluestore-block-db-size: 0
bluestore-block-wal-size: 0
bluestore-compression-algorithm: lz4
customize-failure-domain: true
ignore-device-errors: false
loglevel: 1
max-sectors-kb: 1.048576e+06
nagios_context: juju
nagios_servicegroups: ""
osd-devices: /dev/disk/by-dname/sdb
osd-encrypt: false
osd-encrypt-keymanager: ceph
osd-format: xfs
osd-journal-size: 1024
prefer-ipv6: false
source: cloud:bionic-rocky

# Unfortunately, the timestamp gets updated as hook executions pass so we don't have the original one anymore.
root@obayifo:/var/lib/juju/agents/unit-ceph-osd-2/charm# stat .juju-persistent-config
  File: .juju-persistent-config
  Size: 1392 Blocks: 8 IO Block: 4096 regular file
Device: 804h/2052d Inode: 23072467 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-04-09 14:59:40.833800139 +0000
Modify: 2021-04-09 14:57:45.776090185 +0000
Change: 2021-04-09 14:57:45.776090185 +0000
 Birth: -

root@obayifo:/var/lib/juju/agents/unit-ceph-osd-2/charm# cat .juju-persistent-config
{"aa-profile-mode": "complain", "autotune": false, "availability_zone": null, "bdev-enable-discard": "auto", "bluestore": true, "bluestore-block-db-size": 0, "bluestore-block-wal-size": 0, "bluestore-compression-algorithm": "lz4", "bluestore-compression-max-blob-size": null, "bluestore-compression-max-blob-size-hdd": null, "bluestore-compression-max-blob-size-ssd": null, "bluestore-compression-min-blob-size": null, "bluestore-compression-min-blob-size-hdd": null, "bluestore-compression-min-blob-size-ssd": null, "bluestore-compression-mode": null, "bluestore-compression-required-ratio": null, "bluestore-db": null, "bluestore-wal": null, "ceph-cluster-network": null, "ceph-public-network": null, "config-flags": null, "crush-initial-weight": null, "customize-failure-domain": true, "ephemeral-unmount": null, "harden": null, "ignore-device-errors": false, "key": null, "loglevel": 1, "max-sectors-kb": 1048576, "nagios_context": "juju", "nagios_servicegroups": "", "osd-devices": "/dev/disk/by-dname/sdb", "osd-encrypt": false, "osd-encrypt-keymanager": "ceph", "osd-format": "xfs", "osd-journal": null, "osd-journal-size": 1024, "osd-max-backfills": null, "osd-recovery-max-active": null, "prefer-ipv6": false, "source": "cloud:bionic-rocky", "sysctl": "{ kernel.pid_max : 2097152, vm.max_map_count : 524288, kernel.threads-max: 2097152 }", "use-direct-io": true, "use-syslog": false}

juju show-status-log ceph-osd/2
Time Type Status Message
07 Apr 2021 21:00:55Z juju-unit idle
07 Apr 2021 21:01:28Z juju-unit executing running mon-relation-changed hook for ceph-mon/1
07 Apr 2021 21:01:47Z juju-unit idle
08 Apr 2021 12:08:56Z juju-unit executing running leader-settings-changed hook
08 Apr 2021 12:08:56Z juju-unit idle
09 Apr 2021 12:47:28Z juju-unit executing running config-changed hook
09 Apr 2021 12:47:37Z juju-unit idle