Pacemaker remote resources marked as unclean and not moved.

[Liam Young]

ocf:pacemaker:remote resources are marked as unclean and do
not move in the event of the node hosting them powering off.

This causes the pacemaker_remote.service on the remote nodes
to fail to connect:

# systemctl status pacemaker_remote.service
● pacemaker_remote.service - Pacemaker Remote Service
   Loaded: loaded (/lib/systemd/system/pacemaker_remote.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-07-27 11:00:02 UTC; 1h 5min ago
     Docs: man:pacemaker_remoted
           http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Pacemaker_Remote/index.html
 Main PID: 76263 (pacemaker_remot)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/pacemaker_remote.service
           └─76263 /usr/sbin/pacemaker_remoted

Jul 27 11:00:02 node2 systemd[1]: Started Pacemaker Remote Service.
Jul 27 11:00:02 node2 pacemaker_remoted[76263]: notice: Additional logging available in /var/log/pacemaker.log
Jul 27 11:00:05 node2 pacemaker_remoted[76263]: error: Error in connection setup (76263-76947-14): Remote I/O error (121)
Jul 27 11:52:32 node2 pacemaker_remoted[76263]: error: Connection terminated: The specified session has been invalidated for some reason.
Jul 27 11:52:32 node2 pacemaker_remoted[76263]: error: Connection terminated: The specified session has been invalidated for some reason.
Jul 27 11:52:32 node2 pacemaker_remoted[76263]: error: Could not send remote message: Software caused connection abort

This in turn causes masakari host monitors to hang when checking
the state of their peers which prevents a host down notification
being sent to masakari.

The cause of this seems to be that the cluster has the global
stonith-enabled=true set. The charms create a stonith
resource for the pacemaker remotes but there is no
stonith resource for the lxd container. In the event of a host
being lost, the cluster tries to power off the compute
node, which does have a stonith resource. It then tries to power
off the lxd container but can find no corresponding stonith
resource. This causes the container to marked as unclean
and the resources are not moved.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743742
Committed: https://git.openstack.org/cgit/openstack/charm-hacluster/commit/?id=b40a6754b0256058213afcde80174ca7e730a403
Submitter: Zuul
Branch: master

commit b40a6754b0256058213afcde80174ca7e730a403
Author: Liam Young <email address hidden>
Date: Wed Jul 29 11:59:43 2020 +0000

    Create null stonith resource for lxd containers.

    If stonith is enabled then when a compute node is detected as failed
    it is powered down. This can include a lxd container which is also
    part of the cluster. In this case because stonith is enabled at a
    global level, pacemaker will try and power off the lxd container
    too. But the container does not have a stonith device and this causes
    the container to be marked as unclean (but not down). This running
    unclean state prevents resources being moved and causes any
    pacemaker-remotes that are associated with the lost container from
    losing their connection which prevents masakari hostmonitor from
    ascertaining the cluster health.

    The way to work around this is to create a dummy stonith device for
    the lxd containers. This allows the cluster to properly mark the lost
    container as down and resources are relocated.

    Change-Id: Ic45dbdd9d8581f25549580c7e98a8d6e0bf8c3e7
    Partial-Bug: #1889094

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/749685
Committed: https://git.openstack.org/cgit/openstack/charm-hacluster/commit/?id=527fd2c704f1925ec3bfd59f7d062c849e3f4f3a
Submitter: Zuul
Branch: master

commit 527fd2c704f1925ec3bfd59f7d062c849e3f4f3a
Author: Liam Young <email address hidden>
Date: Thu Jul 30 06:25:36 2020 +0000

    Spread pacemaker remote resources across cluster.

    Use location directives to spread pacemaker remote resources across
    cluster. This is to prevent multiple resources being taken down in
    the event of a single node failure. This would usually not be a
    problem but if the node is being queried by masakari host
    monitors at the time the node goes down then the query can hang.

    Change-Id: Ib8a667d0d82ef3dcd4da27e62460b4f0ce32ee43
    Partial-Bug: #1889094
    Depends-On: Ic45dbdd9d8581f25549580c7e98a8d6e0bf8c3e7