Comment 11 for bug 2039604

Hi,

I faced the same issue today during Control Plane upgrades from Victoria to Wallaby on a list of services:
- aodh
- barbican
- ceilometer
- designate
- placement

Environment is:
* Focal-Victoria cloud during upgrade to Focal-Wallaby
* Juju 2.9.45
* All charms were refreshed to their victoria/latest on the 29th of May.
* The juju status --relations output: https://pastebin.ubuntu.com/p/p4SNTnnNpk/
* The sanized bundle is here: https://pastebin.canonical.com/p/4fJ3JXrkTR/

Steps to reproduce:
1. I don't know if that matters, but the cloud was upgraded from Ussuri to Victoria the last week. So maybe you will not be able to reproduce just deploying Victoria cloud and upgrading to Wallaby.
2. Upgrading all at once. action-managed-upgrade for all services is "false". Auto restarts for rabbit and ovn-central are enabled.
3. For every service:
3.1 Refresh charm from 'victoria/stable' to the new channel: 'wallaby/stable'
3.2 Change charm config 'openstack-origin' to 'cloud:focal-wallaby'

Note, that from Ussuri to Wallaby I upgraded the same way and all services were upgraded without any issue.

What I faced:

### aodh ###

aodh/0, aodh/1 succesfully upgraded.
aodh/2 stayed on Victoria.
Worked around with:
juju run-action aodh/2 openstack-upgrade

### barbican ###

barbican/1, barbican/2 succesfully upgraded.
barbican/0 stayed on Victoria.
Worked around with:
juju run-action barbican/0 openstack-upgrade

### ceilometer ###
ceilometer/0, ceilometer/1 succesfully upgraded.
ceilometer/2 stayed on Victoria
Worked around didn't work:
```
$ juju run-action --wait ceilometer/2 openstack-upgrade
unit-ceilometer-2:
  UnitId: ceilometer/2
  id: "6455"
  results:
    outcome: no upgrade available.
  status: completed
  timing:
    completed: 2024-06-03 10:05:37 +0000 UTC
    enqueued: 2024-06-03 10:05:36 +0000 UTC
    started: 2024-06-03 10:05:36 +0000 UTC
```
Packages on the unit still was Victoria's
```
# dpkg -l | grep ceilometer
ii ceilometer-agent-central 1:15.1.0-0ubuntu1~cloud0 all ceilometer central agent
ii ceilometer-agent-notification 1:15.1.0-0ubuntu1~cloud0 all ceilometer notification agent
ii ceilometer-common 1:15.1.0-0ubuntu1~cloud0 all ceilometer common files
ii python3-ceilometer 1:15.1.0-0ubuntu1~cloud0 all ceilometer python libraries

# apt policy openstack-release
openstack-release:
  Installed: 1.0ubuntu1~cloud0
  Candidate: 1.0ubuntu1~cloud0
  Version table:
 *** 1.0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby/main amd64 Packages
        100 /var/lib/dpkg/status

# apt policy ceilometer-common
ceilometer-common:
  Installed: 1:15.1.0-0ubuntu1~cloud0
  Candidate: 2:16.0.1-0ubuntu1~cloud0
  Version table:
     2:16.0.1-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby/main amd64 Packages
 *** 1:15.1.0-0ubuntu1~cloud0 100
        100 /var/lib/dpkg/status
     1:14.1.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     1:14.0.0~b3~git2020041012.75b9d4e9-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
```
What I finally did is just upgraded apt packages
```
# apt upgrade
# systemctl restart ceilometer-agent-central.service
# systemctl restart ceilometer-agent-notification.service
```
After that juju picked new workload version and upgrade seems to be completed (however, I will see how it will upgrade to Xena).

### designate ###
designate/0, designate/1, designate/2 stayed on Victoria
Worked around with:
juju run-action designate/0 openstack-upgrade
juju run-action designate/1 openstack-upgrade
juju run-action designate/2 openstack-upgrade

### placement ###
placement/0, placement/1 succesfully upgraded.
placement/2 stayed on Victoria.
Worked around with:
juju run-action placement/2 openstack-upgrade

### octavia ###
octavia/0, octavia/1, octavia/2 stayed on Victoria
Worked around with:
juju run-action octavia/0 openstack-upgrade
juju run-action octavia/1 openstack-upgrade
juju run-action octavia/2 openstack-upgrade

I pulled juju logs for every unit which failed to upgrade, and also for aodh, barbican, ceilometer, placement I pulled some random unit logs which succeeded to upgrade after origin config change. The logs I attached to the case as wallaby-logs.zip
I don't see anything obvoius in the logs, but you may notice that before every upgrade to Wallaby, there is `config-changed` hook run, this in me changing `openstack-origin` to `cloud:focal-wallaby`.

Please tell if there is anything useful I can provide.