Series upgrade stateless API services; Clarify procedure

[Wesley Hershberger]

I'm looking for clarification on the series upgrade guidance provided in [1], specifically regarding the stateless API services (keystone, glance, etc).

For context, the document is describing a series upgrade where the version of the API service doesn't change (Keystone stays at 13.0.4, base OS goes xenial -> bionic). It recommends a concurrent upgrade for these services: "A sequential approach will not reduce downtime as the HA services will still need to be brought down during the upgrade associated with the application leader."

I don't think this is correct. As far as I can tell, it is possible to perform a series upgrade with negligible downtime by doing HA failover between the units, e.g. pause and upgrade the two followers; once those are restored, pause and upgrade the leader.

Were keystone itself being upgraded (ussuri -> victoria or similar), there might be an argument for doing it concurrently due to backwards-incompatible database migrations, but that's not the case here.

I'm working on upgrades in my lab environment and will try to add more context here if I discover it.

Thanks!

[1] https://docs.openstack.org/charm-guide/latest/admin/upgrades/series-openstack.html#api-applications

-----------------------------------
Release: 0.0.1.dev795 on 2023-09-27 11:56:11
SHA: ba48a8642ad67b1a239d0ca25473329109f83ddc
Source: https://opendev.org/openstack/charm-guide/src/doc/source/admin/upgrades/series-openstack.rst
URL: https://docs.openstack.org/charm-guide/latest/admin/upgrades/series-openstack.html

[Alex Kavanagh (ajkavanagh)]

> I don't think this is correct. As far as I can tell, it is possible to
perform a series upgrade with negligible downtime by doing HA failover
between the units, e.g. pause and upgrade the two followers; once those
are restored, pause and upgrade the leader.

Unfortunately, with bionic -> focal, the series upgrade for corosync and pacemaker (the hacluster charm's payloads) can't do a rolling upgrade and keep the cluster up due to major version bumps. See https://wiki.ubuntu.com/FocalFossa/ReleaseNotes

Thus for the docs we can't assert that the series upgrade of corosync/pacemaker won't affect a rolling upgrade. It's best to assume that there will be a short outage of the control plane service for that upgrade.