Removing a unit from cluster doesn't fully remove the unit from Pacemaker

[Xav Paice]

I deployed a cluster of 3 percona-cluster units, then removed and added a unit twice.

This resulted in the pacemaker/corosync cluster being configured with 5 machines, rather than only 3. The two that were removed from Juju were not removed from the Pacemaker configs, and even after hand editing the configs, running the config-changed hook put them back there.

Xenial, Queens, and 18.08 charms.

[James Page (james-page)]

tl;dr they get removed from the configuration file on disk, however they persist in the corosync messaging state until manually purged right now. I've seen this in our QA deployment as well when we've been moving services around for the control plane. Functionally there should be no impact however it is a bit ugly.

[James Page (james-page)]

The charm hook code should only be writing out the current nodes in the node list into /etc/corosync/corosync.conf.

[Trent Lloyd (lathiat)]

For future travellers, the workaround for this situation may be simply to run "corosync-cfgtool -R" to reload the configuration file from disk. Detailed such a case here:
https://bugs.launchpad.net/charms/+source/hacluster/+bug/1400481/comments/15