(Before I start... confession ... I currently know very little about pacemaker or the inner workings of charms... so apologies if I'm barking up the wrong tree)
It looks like it fails to set up the resources (e.g. the vip):
ubuntu@juju-a55f84-2-lxd-8:~$ sudo crm config show
node 1000: juju-a55f84-2-lxd-8
node 1001: juju-a55f84-0-lxd-8
node 1002: juju-a55f84-1-lxd-8
property cib-bootstrap-options: \ have-watchdog=false \ dc-version=1.1.18-2b07d5c5a9 \ cluster-infrastructure=corosync \ cluster-name=debian \ no-quorum-policy=stop \ cluster-recheck-interval=60 \ stonith-enabled=false
rsc_defaults rsc-options: \ resource-stickiness=100 \ failure-timeout=0
Relevant bit of /var/log/juju/unit-manila-ha-0.log (manila-ha/0 is the leader):
2020-03-26 15:47:43 INFO juju-log hanode:108: Setting cluster symmetry
2020-03-26 15:47:43 WARNING juju-log hanode:108: Inconsistent or absent enable-resources setting []
2020-03-26 15:47:43 WARNING juju-log hanode:108: Unable to calculated desired symmetric-cluster setting
2020-03-26 15:47:43 DEBUG juju-log hanode:108: Deleting Resources
2020-03-26 15:47:43 DEBUG juju-log hanode:108: Configuring Resources: {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Groups: {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Master/Slave (ms): {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Orders: {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Clones: {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Colocations: {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Locations: {}
2020-03-26 15:47:44 INFO juju-log hanode:108: Configuring any remote nodes
2020-03-26 15:47:45 DEBUG juju-log hanode:108: Checking for pacemaker-remote nodes
2020-03-26 15:47:45 WARNING juju-log hanode:108: Inconsistent or absent enable-resources setting []
2020-03-26 15:47:45 WARNING juju-log hanode:108: Unable to calculate whether resources should run on remotes
2020-03-26 15:47:56 DEBUG juju-log hanode:108: Pacemaker is ready
2020-03-26 15:49:34 DEBUG juju-log Pacemaker is ready
Following the code...
unit-manila-ha-0/charm/hooks/hooks.py:ha_relation_changed()
unit-manila-ha-0/charm/hooks/utils.py:set_cluster_symmetry()
unit-manila-ha-0/charm/hooks/utils.py:need_resources_on_remotes()
... it seems like this:
for relid in relation_ids('pacemaker-remote'):
for unit in related_units(relid):
data = parse_data(relid, unit, 'enable-resources')
# parse_data returns {} if key is absent.
if type(data) is bool: responses.append(data)
... doesn't result in a clean response.
I'll keep digging but in the meantime I'm hoping this triggers an "ah ha" moment in someone more knowledgeable.
(Before I start... confession ... I currently know very little about pacemaker or the inner workings of charms... so apologies if I'm barking up the wrong tree)
It looks like it fails to set up the resources (e.g. the vip): juju-a55f84- 2-lxd-8: ~$ sudo crm config show options: \
have-watchdog= false \
dc-version= 1.1.18- 2b07d5c5a9 \
cluster- infrastructure= corosync \
cluster- name=debian \
no-quorum- policy= stop \
cluster- recheck- interval= 60 \
stonith- enabled= false
resource- stickiness= 100 \
failure- timeout= 0
ubuntu@
node 1000: juju-a55f84-2-lxd-8
node 1001: juju-a55f84-0-lxd-8
node 1002: juju-a55f84-1-lxd-8
property cib-bootstrap-
rsc_defaults rsc-options: \
Relevant bit of /var/log/ juju/unit- manila- ha-0.log (manila-ha/0 is the leader):
2020-03-26 15:47:43 INFO juju-log hanode:108: Setting cluster symmetry
2020-03-26 15:47:43 WARNING juju-log hanode:108: Inconsistent or absent enable-resources setting []
2020-03-26 15:47:43 WARNING juju-log hanode:108: Unable to calculated desired symmetric-cluster setting
2020-03-26 15:47:43 DEBUG juju-log hanode:108: Deleting Resources
2020-03-26 15:47:43 DEBUG juju-log hanode:108: Configuring Resources: {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Groups: {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Master/Slave (ms): {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Orders: {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Clones: {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Colocations: {}
2020-03-26 15:47:44 DEBUG juju-log hanode:108: Configuring Locations: {}
2020-03-26 15:47:44 INFO juju-log hanode:108: Configuring any remote nodes
2020-03-26 15:47:45 DEBUG juju-log hanode:108: Checking for pacemaker-remote nodes
2020-03-26 15:47:45 WARNING juju-log hanode:108: Inconsistent or absent enable-resources setting []
2020-03-26 15:47:45 WARNING juju-log hanode:108: Unable to calculate whether resources should run on remotes
2020-03-26 15:47:56 DEBUG juju-log hanode:108: Pacemaker is ready
2020-03-26 15:49:34 DEBUG juju-log Pacemaker is ready
Following the code... ha-0/charm/ hooks/hooks. py:ha_relation_ changed( ) ha-0/charm/ hooks/utils. py:set_ cluster_ symmetry( ) ha-0/charm/ hooks/utils. py:need_ resources_ on_remotes( ) ids('pacemaker- remote' ): units(relid) :
responses. append( data)
unit-manila-
unit-manila-
unit-manila-
... it seems like this:
for relid in relation_
for unit in related_
data = parse_data(relid, unit, 'enable-resources')
# parse_data returns {} if key is absent.
if type(data) is bool:
... doesn't result in a clean response.
I'll keep digging but in the meantime I'm hoping this triggers an "ah ha" moment in someone more knowledgeable.