Comment 0 for bug 1994160

The masakari change for the zed batch is hitting this:

https://review.opendev.org/c/openstack/charm-masakari/+/850201

Charm deployment fails with:

2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 Traceback (most recent call last):
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/hanode-relation-changed", line 767, in <module>
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 hooks.execute(sys.argv)
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/charmhelpers/core/hookenv.py", line 962, in execute
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 self._hooks[hook_name]()
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/hanode-relation-changed", line 310, in hanode_relation_changed
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 ha_relation_changed()
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/hanode-relation-changed", line 584, in ha_relation_changed
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 configure_resources_on_remotes(
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/utils.py", line 969, in configure_resources_on_remotes
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 add_location_rules_for_local_nodes(cl_name)
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/utils.py", line 804, in add_location_rules_for_local_nodes
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 pcmk.commit(cmd, failure_is_fatal=True)
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/pcmk.py", line 94, in commit
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 return subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 524, in run
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 raise CalledProcessError(retcode, process.args,
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 subprocess.CalledProcessError: Command '['crm', '-w', '-F', 'configure', 'location', 'loc-cl_res_masakari_haproxy-juju-7484ce-zaza-b229c134f7f3-10.project.serverstack:', 'member', 'cl_res_masakari_haproxy', '0:', 'juju-7484ce-zaza-b229c134f7f3-10.project.serverstack:', 'member']' returned non-zero exit status 1.

This is due to 'member' being included in the command.

More details:

ubuntu@juju-951736-zaza-bfe11080faa5-12:/var/lib/juju/agents/unit-hacluster-0/charm$ sudo crm node show
juju-951736-zaza-bfe11080faa5-10.project.serverstack: member
        remote_node: true
        in_ccm: true
        crm-debug-origin: do_update_resource
        node_fenced: 0
juju-951736-zaza-bfe11080faa5-12(1000): member
juju-951736-zaza-bfe11080faa5-13(1001): member
juju-951736-zaza-bfe11080faa5-14(1002): member
juju-951736-zaza-bfe11080faa5-8.project.serverstack: member
        remote_node: true
        in_ccm: true
        crm-debug-origin: do_update_resource
        node_fenced: 0
juju-951736-zaza-bfe11080faa5-9.project.serverstack: member
        remote_node: true
        in_ccm: true
        crm-debug-origin: do_update_resource
        node_fenced: 0

pcmk.list_nodes()=['juju-951736-zaza-bfe11080faa5-10.project.serverstack: member', 'juju-951736-zaza-bfe11080faa5-12', 'juju-951736-zaza-bfe11080faa5-13', 'juju-951736-zaza-bfe11080faa5-14', 'juju-951736-zaza-bfe11080faa5-8.project.serverstack: member', 'juju-951736-zaza-bfe11080faa5-9.project.serverstack: member'

I was also able to recreate this on a focal (ussuri) deploy with the latest/stable charms.

It seems that we need a fix similar to the following in order to trim ": member":
https://opendev.org/openstack/charm-hacluster/commit/4e53bea0769d6c8c37e1e2a612c191961e738222