I got this too. #6 works in my case too after reboot and resolved for ceph-mon/0. The ceph cluster is still working and operation. zlmao@p14s:~/gzgz/upgrade$ juju status ceph-mon Model Controller Cloud/Region Version SLA Timestamp openstack maas maas/default 2.9.45 unsupported 16:21:09+08:00 App Version Status Scale Charm Channel Rev Exposed Message ceph-mon 15.2.17 waiting 3 ceph-mon pacific/stable 189 no Waiting on juju-e5b537-0-lxd-0 to finish upgrading Unit Workload Agent Machine Public address Ports Message ceph-mon/0 maintenance executing 0/lxd/0 192.168.222.69 (config-changed) Finishing upgrade ceph-mon/1* waiting executing 1/lxd/0 192.168.222.65 (config-changed) Waiting on juju-e5b537-0-lxd-0 to finish upgrading ceph-mon/2 waiting executing 2/lxd/0 192.168.222.96 (config-changed) Waiting on juju-e5b537-1-lxd-0 to finish upgrading root@juju-e5b537-0-lxd-0:~# tail -n10 /var/log/juju/unit-ceph-mon-0.log 2024-01-04 07:39:21 INFO unit.ceph-mon/0.juju-log server.go:316 Upgrading to: pacific 2024-01-04 07:39:21 WARNING unit.ceph-mon/0.config-changed logger.go:60 set mon_juju-e5b537-0-lxd-0_pacific_alive 2024-01-04 07:39:21 INFO unit.ceph-mon/0.juju-log server.go:316 Installing [] with options: ['--option=Dpkg::Options::=--force-confold'] 2024-01-04 07:39:24 INFO unit.ceph-mon/0.juju-log server.go:316 Installing ['ceph', 'gdisk', 'radosgw', 'xfsprogs', 'lvm2', 'parted', 'smartmontools', 'btrfs-progs'] with options: ['--option=Dpkg::Options::=--force-confold'] 2024-01-04 07:39:25 INFO unit.ceph-mon/0.juju-log server.go:316 restarting ceph-mgr.target maybe: True 2024-01-04 07:39:25 INFO unit.ceph-mon/0.juju-log server.go:316 Making dir /var/lib/ceph/mon/ceph-juju-e5b537-0-lxd-0 ceph:ceph 755 2024-01-04 07:39:25 INFO unit.ceph-mon/0.juju-log server.go:316 starting ceph-mgr.target maybe: True 2024-01-04 07:39:25 INFO unit.ceph-mon/0.juju-log server.go:316 Done 2024-01-04 07:39:25 INFO unit.ceph-mon/0.juju-log server.go:316 monitor_key_set mon_juju-e5b537-0-lxd-0_pacific_done 1704353965.4807012 2024-01-04 07:39:25 WARNING unit.ceph-mon/0.config-changed logger.go:60 set mon_juju-e5b537-0-lxd-0_pacific_done root@juju-e5b537-0-lxd-0:~# ubuntu@juju-e5b537-1-lxd-0:~$ sudo tail -n10 /var/log/juju/unit-ceph-mon-1.log 2024-01-04 07:38:29 INFO unit.ceph-mon/1.juju-log server.go:316 roll_monitor_cluster called with pacific 2024-01-04 07:38:29 INFO unit.ceph-mon/1.juju-log server.go:316 monitor_list: ['juju-e5b537-1-lxd-0', 'juju-e5b537-2-lxd-0', 'juju-e5b537-0-lxd-0'] 2024-01-04 07:38:29 WARNING unit.ceph-mon/1.juju-log server.go:316 DEPRECATION WARNING: Function one_shot_log is being removed : Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25 2024-01-04 07:38:29 INFO unit.ceph-mon/1.juju-log server.go:316 Current Ceph version is 15.2 2024-01-04 07:38:29 INFO unit.ceph-mon/1.juju-log server.go:316 Upgrading to: pacific 2024-01-04 07:38:30 INFO unit.ceph-mon/1.juju-log server.go:316 Installing ['ubuntu-cloud-keyring'] with options: ['--option=Dpkg::Options::=--force-confold'] 2024-01-04 07:38:35 INFO unit.ceph-mon/1.juju-log server.go:316 Installing ['ceph', 'gdisk', 'radosgw', 'xfsprogs', 'lvm2', 'parted', 'smartmontools', 'btrfs-progs'] with options: ['--option=Dpkg::Options::=--force-confold'] 2024-01-04 07:39:31 INFO unit.ceph-mon/1.juju-log server.go:316 Packages upgraded but not restarting daemons yet. 2024-01-04 07:39:31 INFO unit.ceph-mon/1.juju-log server.go:316 upgrade position: 1 2024-01-04 07:39:32 INFO unit.ceph-mon/1.juju-log server.go:316 Previous node is: juju-e5b537-0-lxd-0 ubuntu@juju-e5b537-1-lxd-0:~$ ubuntu@juju-e5b537-2-lxd-0:~$ sudo tail -n10 /var/log/juju/unit-ceph-mon-2.log 2024-01-04 07:38:29 INFO unit.ceph-mon/2.juju-log server.go:316 Upgrading to: pacific 2024-01-04 07:38:30 INFO unit.ceph-mon/2.juju-log server.go:316 Installing ['ubuntu-cloud-keyring'] with options: ['--option=Dpkg::Options::=--force-confold'] 2024-01-04 07:38:35 INFO unit.ceph-mon/2.juju-log server.go:316 Installing ['ceph', 'gdisk', 'radosgw', 'xfsprogs', 'lvm2', 'parted', 'smartmontools', 'btrfs-progs'] with options: ['--option=Dpkg::Options::=--force-confold'] 2024-01-04 07:39:08 INFO unit.ceph-mon/2.juju-log server.go:316 Packages upgraded but not restarting daemons yet. 2024-01-04 07:39:08 INFO unit.ceph-mon/2.juju-log server.go:316 upgrade position: 2 2024-01-04 07:39:08 INFO unit.ceph-mon/2.juju-log server.go:316 Previous node is: juju-e5b537-1-lxd-0 2024-01-04 07:39:08 WARNING unit.ceph-mon/2.config-changed logger.go:60 Error ENOENT: key 'mon_juju-e5b537-1-lxd-0_pacific_start' doesn't exist 2024-01-04 07:39:08 INFO unit.ceph-mon/2.juju-log server.go:316 wait_until: waiting for 8 seconds 2024-01-04 07:39:17 WARNING unit.ceph-mon/2.config-changed logger.go:60 Error ENOENT: key 'mon_juju-e5b537-1-lxd-0_pacific_start' doesn't exist 2024-01-04 07:39:17 INFO unit.ceph-mon/2.juju-log server.go:316 wait_until: waiting for 19 seconds ubuntu@juju-e5b537-2-lxd-0:~$