ERROR: could not patch cib (rc=203)

[Marian Gasparovic]

During hanode-relation-changed this error occurred

2020-06-22 18:16:42 DEBUG hanode-relation-changed ERROR: could not patch cib (rc=203)
2020-06-22 18:16:42 DEBUG hanode-relation-changed INFO: offending xml diff: <diff format="2">
2020-06-22 18:16:42 DEBUG hanode-relation-changed <change operation="create" path="/cib/configuration/resources" position="0">
2020-06-22 18:16:42 DEBUG hanode-relation-changed <primitive id="st-maas-a2282ef" class="stonith" type="external/maas">
2020-06-22 18:16:42 DEBUG hanode-relation-changed <instance_attributes id="st-maas-a2282ef-instance_attributes">
2020-06-22 18:16:42 DEBUG hanode-relation-changed <nvpair name="url" value="http://10.244.40.33/MAAS/" id="st-maas-a2282ef-instance_attributes-url"/>
2020-06-22 18:16:42 DEBUG hanode-relation-changed <nvpair name="apikey" value="*censored*" id="st-maas-a2282ef-instance_attributes-apikey"/>
2020-06-22 18:16:42 DEBUG hanode-relation-changed <nvpair name="hostnames" value="armaldo armaldo.production.solutionsqa elgyem elgyem.production.solutionsqa quilava quilava.production.solutionsqa spearow spearow.production.solutionsqa" id="st-maas-a2282ef-instance_attributes-hostnames"/>
2020-06-22 18:16:42 DEBUG hanode-relation-changed </instance_attributes>
2020-06-22 18:16:42 DEBUG hanode-relation-changed <operations>
2020-06-22 18:16:42 DEBUG hanode-relation-changed <op name="monitor" interval="25" start-delay="25" timeout="25" id="st-maas-a2282ef-monitor-25"/>
2020-06-22 18:16:42 DEBUG hanode-relation-changed </operations>
2020-06-22 18:16:42 DEBUG hanode-relation-changed </primitive>
2020-06-22 18:16:42 DEBUG hanode-relation-changed </change>
2020-06-22 18:16:42 DEBUG hanode-relation-changed </diff>

I will add crashdump

[Marian Gasparovic (marosg)]

2020-06-22 18:16:42 DEBUG hanode-relation-changed Traceback (most recent call last):
2020-06-22 18:16:42 DEBUG hanode-relation-changed File "/var/lib/juju/agents/unit-hacluster-masakari-1/charm/hooks/hanode-relation-changed", line 660, in <module>
2020-06-22 18:16:42 DEBUG hanode-relation-changed hooks.execute(sys.argv)
2020-06-22 18:16:42 DEBUG hanode-relation-changed File "/var/lib/juju/agents/unit-hacluster-masakari-1/charm/charmhelpers/core/hookenv.py", line 943, in execute
2020-06-22 18:16:42 DEBUG hanode-relation-changed self._hooks[hook_name]()
2020-06-22 18:16:42 DEBUG hanode-relation-changed File "/var/lib/juju/agents/unit-hacluster-masakari-1/charm/hooks/hanode-relation-changed", line 270, in hanode_relation_changed
2020-06-22 18:16:42 DEBUG hanode-relation-changed ha_relation_changed()
2020-06-22 18:16:42 DEBUG hanode-relation-changed File "/var/lib/juju/agents/unit-hacluster-masakari-1/charm/hooks/hanode-relation-changed", line 367, in ha_relation_changed
2020-06-22 18:16:42 DEBUG hanode-relation-changed configure_stonith()
2020-06-22 18:16:42 DEBUG hanode-relation-changed File "/var/lib/juju/agents/unit-hacluster-masakari-1/charm/hooks/utils.py", line 509, in configure_stonith
2020-06-22 18:16:42 DEBUG hanode-relation-changed if configure_pacemaker_remote_stonith_resource():
2020-06-22 18:16:42 DEBUG hanode-relation-changed File "/var/lib/juju/agents/unit-hacluster-masakari-1/charm/hooks/utils.py", line 799, in configure_pacemaker_remote_stonith_resource
2020-06-22 18:16:42 DEBUG hanode-relation-changed stonith_resource = configure_maas_stonith_resource(hostnames)
2020-06-22 18:16:42 DEBUG hanode-relation-changed File "/var/lib/juju/agents/unit-hacluster-masakari-1/charm/hooks/utils.py", line 653, in configure_maas_stonith_resource
2020-06-22 18:16:42 DEBUG hanode-relation-changed pcmk.commit(cmd, failure_is_fatal=True)
2020-06-22 18:16:42 DEBUG hanode-relation-changed File "/var/lib/juju/agents/unit-hacluster-masakari-1/charm/hooks/pcmk.py", line 68, in commit
2020-06-22 18:16:42 DEBUG hanode-relation-changed return subprocess.check_call(cmd.split())
2020-06-22 18:16:42 DEBUG hanode-relation-changed File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
2020-06-22 18:16:42 DEBUG hanode-relation-changed raise CalledProcessError(retcode, cmd)
2020-06-22 18:16:42 DEBUG hanode-relation-changed subprocess.CalledProcessError: Command '['crm', 'configure', 'primitive', 'st-maas-a2282ef', 'stonith:external/maas', 'params', "url='http://10.244.40.33/MAAS/'", "apikey='*censored*'", "hostnames='armaldo", 'armaldo.production.solutionsqa', 'elgyem', 'elgyem.production.solutionsqa', 'quilava', 'quilava.production.solutionsqa', 'spearow', "spearow.production.solutionsqa'", 'op', 'monitor', 'interval=25', 'start-delay=25', 'timeout=25']' returned non-zero exit status 1.
2020-06-22 18:16:42 ERROR juju.worker.uniter.operation runhook.go:132 hook "hanode-relation-changed" failed: exit status 1

[Marian Gasparovic (marosg)]

crashdump in artifacts
https://oil-jenkins.canonical.com/artifacts/78ac71a5-5358-4cf7-81d0-82b38775e18b/index.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741991
Committed: https://git.openstack.org/cgit/openstack/charm-hacluster/commit/?id=ca345745923d26da480531976a1ec7bcbd474386
Submitter: Zuul
Branch: master

commit ca345745923d26da480531976a1ec7bcbd474386
Author: Liam Young <email address hidden>
Date: Mon Jul 20 17:04:57 2020 +0000

    Ensure setup is only run on leader.

    configure_cluster_global, configure_monitor_host and configure_stonith should
    only be run by the leader, otherwise there is the risk of the updates
    happening simultaneously and failing.

    Change-Id: I495ee093a8395433412d890396cd991c6acd97f3
    Closes-Bug: #1884797