On HA overcloud, there are three different ways a pacemaker-managed container may need to be restarted:
1. a container image update
2. a tripleo service config change (in /var/lib/config-data/puppet-genenrated/<service>)
3. a pacemaker resource config update (i.e. pcs resource update <...>)
Case 1. has to take place during a minor update workflow (i.e. not during a stack redeploy/update), because
it requires a coordinated action. In this workflow, various ansible tasks are executed, sequentially, one controller after the other. The sequentiality of the workflow ensure that the image update is coordinated across the entire cluster.
Case 2. and 3. are handled a bit differently depending on the action (stack update or minor update workflow):
Stack update:
docker-puppet regenerates the service configs on all the controller nodes. Then:
. a special transient container <service>_init_bundle is run on a single controller, and restarts the pacemaker resource on all nodes if the bundle config has changed.
. another special container <service>_restart_bundle is run on a single controller, and it can also restart the pacemaker resource on all nodes if the tripleo service config has changed since last run.
Minor update:
The pacemaker cluster is restarted on each controller node, sequentially. This guarantees that all pacemaker-managed containers are being restarted unconditionally, and without service disruption (services restart on one node at a time, so there are always two controller nodes available).
Thanks to the unconditional restart, we avoid running container <service>_restart_bundle during minor upgrade because 1) we know that the service will get restarted anyway, and because 2) running restart_bundle during the minor update would restart the service on _all_ nodes at once, which would break the service availability.
However, the approach of "skipping <service>_restart_bundle in minor update" makes the underlying assumption that when the pacemaker cluster restarts on a node, the configs have already been regenerated by docker-puppet. But this assumption is false [1], so a restarted container won't pick any config update that may happen later during the minor update.
[1] https://review.opendev.org/#/c/635725/5/common/deploy-steps.j2@494 : ansible run update task _before_ deploy tasks, and one controller at a time. Which means that when pacemaker is restarted during the update tasks, the config hasn't been regenerated yet.
Fix proposed to branch: master
Review: https:/