Resource adding cl_res_masakari_haproxy is added to compute nodes

[Diko Parvanov]

Masakari revision 10, cl_res_masakari_haproxy added as part of the compute nodes. This resource should only be scheduled on the masakari nodes:

ubuntu@juju-7de5e9-24-lxd-11:~$ sudo crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: juju-7de5e9-22-lxd-12 (version 2.0.3-4b1f869f0f) - partition with quorum
  * Last updated: Tue Jun 22 07:00:08 2021
  * Last change: Mon Jun 21 13:12:20 2021 by root via crm_node on juju-7de5e9-26-lxd-11
  * 9 nodes configured
  * 14 resource instances configured

Node List:
  * Online: [ juju-7de5e9-22-lxd-12 juju-7de5e9-24-lxd-11 juju-7de5e9-26-lxd-11 ]
  * RemoteOnline: [ compute-server-1 compute-server-2 compute-server-3 compute-server-4 compute-server-5 compute-server-6 ]

Full List of Resources:
  * Resource Group: grp_masakari_vips:
    * res_masakari_117daa6_vip (ocf::heartbeat:IPaddr2): Started juju-7de5e9-22-lxd-12
    * res_masakari_c365489_vip (ocf::heartbeat:IPaddr2): Started juju-7de5e9-22-lxd-12
  * Clone Set: cl_res_masakari_haproxy [res_masakari_haproxy]:
    * Started: [ juju-7de5e9-22-lxd-12 juju-7de5e9-24-lxd-11 juju-7de5e9-26-lxd-11 ]
    * Stopped: [ compute-server-1 compute-server-2 compute-server-3 compute-server-4 compute-server-5 compute-server-6 ]
  * compute-server-4 (ocf::pacemaker:remote): Started juju-7de5e9-22-lxd-12
  * compute-server-6 (ocf::pacemaker:remote): Started juju-7de5e9-24-lxd-11
  * compute-server-3 (ocf::pacemaker:remote): Started juju-7de5e9-26-lxd-11
  * compute-server-2 (ocf::pacemaker:remote): Started juju-7de5e9-24-lxd-11
  * compute-server-5 (ocf::pacemaker:remote): Started juju-7de5e9-22-lxd-12
  * compute-server-1 (ocf::pacemaker:remote): Started juju-7de5e9-22-lxd-12
  * st-maas (stonith:external/maas): Started juju-7de5e9-22-lxd-12
  * st-null (stonith:null): Started juju-7de5e9-22-lxd-12

Workaround was to run:

juju run --unit hacluster-masakari/0 "hooks/pacemaker-remote-relation-changed"

[DUFOUR Olivier (odufourc)]

To be more precise, the issue is not exactly related to an association with the compute nodes.
It is indirectly related to issue #1874719 where at the initialization of Corosync takes in account the non-existing node "node1"

Looking at the code here : https://github.com/openstack/charm-hacluster/blob/bbe25222e0ad91ba83f2d5f5c72ac8889482575c/hooks/utils.py#L981

It sets the option clone-max to 4 at initialization which correspond to :
- node1 + 3 units of Masakari.
However since node1 is unavailable, it tries to run the resource on any other available node.

Another quicker workaround is to run this command (and in fact what the mentioned hook above does) :
juju ssh masakari/leader \
  sudo crm configure set cl_res_masakari_haproxy.clone-max 3

Replace the number 3 by any number of Masakari units there are in the deployment.

An improvement eventually may be to set a location constraint eventually to ensure it doesn't try to run specific resources such as haproxy and vIP on the remote nodes.

[Billy Olsen (billy-olsen)]

I was already suspicious of the node1 setting, but the comment from Olivier makes it clear in my mind that this is related to bug #1874719.

The hacluster charm has a fix to address this, which should resolve this for new deployments. However, existing deployments will need to be cleaned up using the above workaround from comment #1.

I'll add charm-hacluster and mark it as invalid for charm-masakari as it is due to functionality in the hacluster charm. This may be marked as a duplicate to the above bug as well, but I'll leave that momentarily for later while we determine if clones should be set to the number of nodes determined in pacemaker (my default answer is yes) or the number of units that exist in the deployment.