Ceph OSDs require manual restarts after Octopuc -> Pacific upgrade after Victoria -> Wallaby openstack upgrade

[Aliaksandr Vasiuk]

Hi,

Environment is:
* Focal-Victoria cloud during upgrade to Focal-Wallaby
* Juju 2.9.45
* All charms were refreshed to their victoria/latest on the 29th of May.
* All control plane and all Nova were Wallaby already.
* Somewhat outdated(because I took it before Nova upgrades) juju status --relations output: https://pastebin.ubuntu.com/p/p4SNTnnNpk/
* Somewhat outdated(because I took it before Nova upgrades) sanitized bundle is here: https://pastebin.canonical.com/p/4fJ3JXrkTR/

How I reproduced it once:
* All OpenStack services were on Wallaby, all Ceph services except ceph-osd were on Pacific. Ceph osd were still Octopus
* Upgraded ceph-osd.
```
juju refresh ceph-osd --channel="pacific/stable"

# wait until settles

juju config ceph-osd source=cloud:focal-wallaby

# wait until settles
```

After that I saw that all my osds are still running octopus
```
$ sudo ceph tell osd.* version
... skipping ...
    "version": "15.2.17",
    "release": "octopus",
... skipping ...

```

I restarted all `systemctl restart ceph-osd.target` manually and all versions changed to
```
    "version": "16.2.15",
    "release": "pacific",
```

After that ceph complained about `require_osd_release < pacific`
```
[WRN] OSD_UPGRADE_FINISHED: all OSDs are running pacific or later but require_osd_release < pacific
    all OSDs are running pacific or later but require_osd_release < pacific
```
I had manually switch it to pacific
```
$ sudo ceph osd require-osd-release pacific
```

After that ceph cluster became healthy. Seems like charm haven't done much except for package refresh.

The juju log for one of ceph-osd units during the upgrade: https://pastebin.ubuntu.com/p/s4kcbXzFsp/

What I think important is that 6 out of 9 Nova hosts were affected by https://bugs.launchpad.net/charm-nova-compute/+bug/2068109 during Victoria -> Wallaby. And seems like there is some mess with dependencies between nova and ceph during this upgrade, however for me it is still not clear.