Comment 7 for bug 1929234

I think I got at the root of the problem regarding the systemd services and Pacemaker. When those resources run independently of Pacemaker, the file always-restart does a great job of keeping then alive. OTOH, Pacemaker needs more configurations to run as expected. The first thing that I noticed is that if the configuration `start-failure-is-fatal` is not set to false, Pacemaker will give up of trying to restart the service and that is why you could have the problem described in the bug. The second thing is that apiserver should be running before starting controller-manager in the same node.

Right now pacemaker starts apiserver on one node and tries to start controller-manager in another node that does not have apiserver.

Those constraints can be achieved using colocation and order, but I found out that charm-hacluster doesn't wait to settle all configurations before starting allocating the services in the nodes. I've opened a bug [1] for this.

Another issue that I found is that right now charm-hacluster doesn't group the VIP with other resources. This means that if a service starts to fail, Pacemaker won't try to move the VIP to another node. I've also opened a bug for this [2].

The approach now it will be removing the master services from Pacemaker and leave it to systemd. At every hook execution it will check the healthy of those services (already implemented) and in case of a failed service, it will force the migration of the VIP to a node where services are healthy.

[1] https://bugs.launchpad.net/charm-hacluster/+bug/1952492
[2] https://bugs.launchpad.net/charm-hacluster/+bug/1952753