'crm -w -F configure location' fails due to list_nodes() including ': member' in node names output

[Corey Bryant]

The masakari change for the zed batch is hitting this:

https://review.opendev.org/c/openstack/charm-masakari/+/850201

Charm deployment fails with:

2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 Traceback (most recent call last):
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/hanode-relation-changed", line 767, in <module>
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 hooks.execute(sys.argv)
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/charmhelpers/core/hookenv.py", line 962, in execute
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 self._hooks[hook_name]()
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/hanode-relation-changed", line 310, in hanode_relation_changed
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 ha_relation_changed()
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/hanode-relation-changed", line 584, in ha_relation_changed
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 configure_resources_on_remotes(
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/utils.py", line 969, in configure_resources_on_remotes
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 add_location_rules_for_local_nodes(cl_name)
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/utils.py", line 804, in add_location_rules_for_local_nodes
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 pcmk.commit(cmd, failure_is_fatal=True)
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/pcmk.py", line 94, in commit
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 return subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 524, in run
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 raise CalledProcessError(retcode, process.args,
2022-10-11 13:35:11 WARNING unit.hacluster/0.hanode-relation-changed logger.go:60 subprocess.CalledProcessError: Command '['crm', '-w', '-F', 'configure', 'location', 'loc-cl_res_masakari_haproxy-juju-7484ce-zaza-b229c134f7f3-10.project.serverstack:', 'member', 'cl_res_masakari_haproxy', '0:', 'juju-7484ce-zaza-b229c134f7f3-10.project.serverstack:', 'member']' returned non-zero exit status 1.

This is due to 'member' being included in the command.

More details:

ubuntu@juju-951736-zaza-bfe11080faa5-12:/var/lib/juju/agents/unit-hacluster-0/charm$ sudo crm node show
juju-951736-zaza-bfe11080faa5-10.project.serverstack: member
        remote_node: true
        in_ccm: true
        crm-debug-origin: do_update_resource
        node_fenced: 0
juju-951736-zaza-bfe11080faa5-12(1000): member
juju-951736-zaza-bfe11080faa5-13(1001): member
juju-951736-zaza-bfe11080faa5-14(1002): member
juju-951736-zaza-bfe11080faa5-8.project.serverstack: member
        remote_node: true
        in_ccm: true
        crm-debug-origin: do_update_resource
        node_fenced: 0
juju-951736-zaza-bfe11080faa5-9.project.serverstack: member
        remote_node: true
        in_ccm: true
        crm-debug-origin: do_update_resource
        node_fenced: 0

pcmk.list_nodes()=['juju-951736-zaza-bfe11080faa5-10.project.serverstack: member', 'juju-951736-zaza-bfe11080faa5-12', 'juju-951736-zaza-bfe11080faa5-13', 'juju-951736-zaza-bfe11080faa5-14', 'juju-951736-zaza-bfe11080faa5-8.project.serverstack: member', 'juju-951736-zaza-bfe11080faa5-9.project.serverstack: member'

I also deployed on focal (ussuri) with the latest/stable charms and 'crm node show' has the same behavior.

It seems that we need a fix similar to the following in order to trim ": member":
https://opendev.org/openstack/charm-hacluster/commit/4e53bea0769d6c8c37e1e2a612c191961e738222

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-hacluster/+/862627

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-hacluster/+/862627
Committed: https://opendev.org/openstack/charm-hacluster/commit/f73ca4d52fdc530531366d65e412a17877c88c63
Submitter: "Zuul (22348)"
Branch: master

commit f73ca4d52fdc530531366d65e412a17877c88c63
Author: Corey Bryant <email address hidden>
Date: Tue Oct 25 16:48:21 2022 +0000

    Update 'crm node show' parsing to trim ': member'

    The command 'crm node show' is used on jammy to retrieve the list of
    nodes defined in a cluster. The output for nodes includes ': member'
    which breaks ensuing commands that are using list_nodes() output.

    For example:
    juju-3f6cb6-zaza-4135aa8b2509-8.project.serverstack: member

    This change trims everything including and after the ':' from the
    output.

    Closes-Bug: #1994160
    Change-Id: I54a4f854f3e293503ec97d99a49b6dc51ee50c87

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/jammy
Review: https://review.opendev.org/c/openstack/charm-hacluster/+/862719

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-hacluster/+/862719
Committed: https://opendev.org/openstack/charm-hacluster/commit/6dc05401c65f5be684913a25d9f8ad7c114e0a70
Submitter: "Zuul (22348)"
Branch: stable/jammy

commit 6dc05401c65f5be684913a25d9f8ad7c114e0a70
Author: Corey Bryant <email address hidden>
Date: Tue Oct 25 16:48:21 2022 +0000

    Update 'crm node show' parsing to trim ': member'

    The command 'crm node show' is used on jammy to retrieve the list of
    nodes defined in a cluster. The output for nodes includes ': member'
    which breaks ensuing commands that are using list_nodes() output.

    For example:
    juju-3f6cb6-zaza-4135aa8b2509-8.project.serverstack: member

    This change trims everything including and after the ':' from the
    output.

    Closes-Bug: #1994160
    Change-Id: I54a4f854f3e293503ec97d99a49b6dc51ee50c87
    (cherry picked from commit f73ca4d52fdc530531366d65e412a17877c88c63)