unit(s) not upgrading packages after changing openstack-origin

[Gabriel Cocenza]

I'm in a cloud upgrading from victoria to wallaby and I noticed that one unit didn't get upgrade after changing the openstack-origin to cloud:focal-wallaby (action-managed-upgrade was set to False).

The unit that didn't upgrade is active idle for some hours and the packages version didn't change:

https://pastebin.canonical.com/p/jf293y3dYV/

If I run `juju run-action barbican/2 openstack-upgrade` then after the unit is stable, the package is upgraded.

The same pattern happened on charm-aodh, so maybe it's a problem with reactive openstack-charms

[Gabriel Cocenza (gabrielcocenza)]

The workaround for ceph-radosgw was manually changing /etc/apt/sources.list.d/cloud-archive.list to wallaby in the leader unit since there is no openstack-upgrade action and pausing and resume was not enough to get the packages upgrade.

[Gabriel Cocenza (gabrielcocenza)]

all units on charm-octavia didn't get upgrade

[Alex Kavanagh (ajkavanagh)]

Hi Gabriel

Did you observe this behaviour on all the charms that you've added to this bug?

* Please could you add charm logs and juju machine logs for all the units of the applications/charms affected, and the rough time *when* the "juju config <app> openstack-origin=cloud:focal-wallaby" was run. This is so we can see when the config-changed hook ran for the units in question.
* Please could provide the version of Juju being used and/or a juju status output

Thanks

[Gabriel Cocenza (gabrielcocenza)]

Hi Alex.

Yes, I've observed this behavior on all charms that I've added.

Isn't enough the juju-status and the sos-report attached of barbican in the bug description?

I didn't collect the logs for other charms because the behavior was very similar.

[Alex Kavanagh (ajkavanagh)]

Hi Gabriel

No, to understand what's going on the units' debug logs and the units' juju machine logs along with when the config command is run *is* needed. Anyone debugging this needs to be able to match up when the config command was made, to link it to the machine log which will indicate when the config-changed hook ran to link it to the debug log for that config changed hook. Then it would be possible to rule out whether config-changed didn't run, but whether the value changed from the perspective of the charm, or whether the charm took no action on the changed value.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Placement Charm because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for OpenStack Octavia Charm because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for Ceph RADOS Gateway Charm because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for OpenStack Designate Charm because there has been no activity for 60 days.]

[Aliaksandr Vasiuk (valexby)]

Hi,

I faced the same issue today during Control Plane upgrades from Victoria to Wallaby on a list of services:
- aodh
- barbican
- ceilometer
- designate
- placement

Environment is:
* Focal-Victoria cloud during upgrade to Focal-Wallaby
* Juju 2.9.45
* All charms were refreshed to their victoria/latest on the 29th of May.
* The juju status --relations output: https://pastebin.ubuntu.com/p/p4SNTnnNpk/
* The sanized bundle is here: https://pastebin.canonical.com/p/4fJ3JXrkTR/

Steps to reproduce:
1. I don't know if that matters, but the cloud was upgraded from Ussuri to Victoria the last week. So maybe you will not be able to reproduce just deploying Victoria cloud and upgrading to Wallaby.
2. Upgrading all at once. action-managed-upgrade for all services is "false". Auto restarts for rabbit and ovn-central are enabled.
3. For every service:
3.1 Refresh charm from 'victoria/stable' to the new channel: 'wallaby/stable'
3.2 Change charm config 'openstack-origin' to 'cloud:focal-wallaby'

Note, that from Ussuri to Wallaby I upgraded the same way and all services were upgraded without any issue.

What I faced:

### aodh ###

aodh/0, aodh/1 succesfully upgraded.
aodh/2 stayed on Victoria.
Worked around with:
juju run-action aodh/2 openstack-upgrade

### barbican ###

barbican/1, barbican/2 succesfully upgraded.
barbican/0 stayed on Victoria.
Worked around with:
juju run-action barbican/0 openstack-upgrade

### ceilometer ###
ceilometer/0, ceilometer/1 succesfully upgraded.
ceilometer/2 stayed on Victoria
Worked around didn't work:
```
$ juju run-action --wait ceilometer/2 openstack-upgrade
unit-ceilometer-2:
  UnitId: ceilometer/2
  id: "6455"
  results:
    outcome: no upgrade available.
  status: completed
  timing:
    completed: 2024-06-03 10:05:37 +0000 UTC
  ...

[Andrea Ieri (aieri)]

Marking this bug back to new for the services mentioned by Alex. The attached logs will hopefully be sufficient for evaluating this bug.

[Alex Kavanagh (ajkavanagh)]

I've set the charms' bug status to confirmed as more than one person has experienced the bug. However, I've not triaged it (yet) as it's not clear whether it's Juju or a strange charms.reactive/charms.openstack upgrade bug. It could be due to:

* charms.openstack at the stable/<version> not having the correct metadata to determine whether a charm needs to upgrade openstack.
* a bug in the logic for openstack upgrades in charms.openstack.
* Juju not running the config-changed hook?
* <something else>