[jammy] ActionFailed: Run of action "update-ring" with parameters "{'i-really-mean-it': True}" on "hacluster/0" failed

[Felipe Reyes]

Running the action 'update-ring' fails when one (or more) nodes are in maintenance mode, this is because on jammy the list of nodes is retrieved using "crm node show" which includes the properties set on a node, making the function pcmk.list_nodes() fail to return the correct information due to a failure in the parsing of the output.

See https://opendev.org/openstack/charm-hacluster/commit/1347ea00c1627566d5c9ed3efadb098a52d42c23

$ juju run-action --wait hacluster/0 update-ring i-really-mean-it=True
unit-hacluster-0:
  UnitId: hacluster/0
  id: "76"
  message: "Removing \tstandby=off maintenance=true from the cluster failed. Command
    '['crm', '-w', '-F', 'node', 'maintenance', 'standby=off', 'maintenance=true']'
    returned non-zero exit status 1. output=b'ERROR: node.maintenance: Expected ([node]),
    takes at most 1 arguments (2 given)\\n'"
  results: {}
  status: failed
  timing:
    completed: 2022-08-25 14:59:49 +0000 UTC
    enqueued: 2022-08-25 14:59:47 +0000 UTC
    started: 2022-08-25 14:59:48 +0000 UTC

2022-08-24 18:04:24 [INFO] ======================================================================
2022-08-24 18:04:24 [INFO] ERROR: test_930_scaleback (zaza.openstack.charm_tests.hacluster.tests.HaclusterScaleBackAndForthTest)
2022-08-24 18:04:24 [INFO] Remove one unit, recalculate quorum and re-add one unit.
2022-08-24 18:04:24 [INFO] ----------------------------------------------------------------------
2022-08-24 18:04:24 [INFO] Traceback (most recent call last):
2022-08-24 18:04:24 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-hacluster/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/hacluster/tests.py", line 137, in test_930_scaleback
2022-08-24 18:04:24 [INFO] zaza.model.run_action_on_leader(
2022-08-24 18:04:24 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-hacluster/.tox/func-target/lib/python3.8/site-packages/zaza/__init__.py", line 108, in _wrapper
2022-08-24 18:04:24 [INFO] return run(_run_it())
2022-08-24 18:04:24 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-hacluster/.tox/func-target/lib/python3.8/site-packages/zaza/__init__.py", line 93, in run
2022-08-24 18:04:24 [INFO] return task.result()
2022-08-24 18:04:24 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-hacluster/.tox/func-target/lib/python3.8/site-packages/zaza/__init__.py", line 107, in _run_it
2022-08-24 18:04:24 [INFO] return await f(*args, **kwargs)
2022-08-24 18:04:24 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-hacluster/.tox/func-target/lib/python3.8/site-packages/zaza/model.py", line 1027, in async_run_action_on_leader
2022-08-24 18:04:24 [INFO] raise ActionFailed(action_obj, output=output)
2022-08-24 18:04:24 [INFO] zaza.model.ActionFailed: Run of action "update-ring" with parameters "{'i-really-mean-it': True}" on "hacluster/0" failed with "Removing standby=off from the cluster failed. Command '['crm', '-w', '-F', 'node', 'maintenance', 'standby=off']' returned non-zero exit status 1. output=b'ERROR: CIB is not valid!\nERROR: Failed to update node attributes for standby=off\n'" (id=72 status=failed enqueued=2022-08-24T18:04:18Z started=2022-08-24T18:04:19Z completed=2022-08-24T18:04:24Z output={'Code': '0'})
2022-08-24 18:04:24 [INFO] ----------------------------------------------------------------------

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-hacluster/+/854651

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/jammy
Review: https://review.opendev.org/c/openstack/charm-hacluster/+/854652

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-hacluster/+/854651
Committed: https://opendev.org/openstack/charm-hacluster/commit/4e53bea0769d6c8c37e1e2a612c191961e738222
Submitter: "Zuul (22348)"
Branch: master

commit 4e53bea0769d6c8c37e1e2a612c191961e738222
Author: Felipe Reyes <email address hidden>
Date: Thu Aug 25 12:13:33 2022 -0400

    Fix 'crm node show' parsing to get list of nodes.

    The command 'crm node show' is used on jammy to retrieve the list of
    nodes defined in a cluster, although this command also includes the
    properties set on a node (e.g. standby=off) which breaks the current
    logic parsing.

    This change uses a regular expresion to filter out all the lines from
    the output that don't start with a non-white character (^\S+).

    Change-Id: I3e00daa1b877a7faae1370f08b2d9c5bd7795c5f
    Closes-Bug: #1987685
    Related-Bug: #1972022

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-hacluster/+/854652
Committed: https://opendev.org/openstack/charm-hacluster/commit/a2ed86b295cb445f5f1d1a4eee0127f34d6073a1
Submitter: "Zuul (22348)"
Branch: stable/jammy

commit a2ed86b295cb445f5f1d1a4eee0127f34d6073a1
Author: Felipe Reyes <email address hidden>
Date: Thu Aug 25 12:13:33 2022 -0400

    Fix 'crm node show' parsing to get list of nodes.

    The command 'crm node show' is used on jammy to retrieve the list of
    nodes defined in a cluster, although this command also includes the
    properties set on a node (e.g. standby=off) which breaks the current
    logic parsing.

    This change uses a regular expresion to filter out all the lines from
    the output that don't start with a non-white character (^\S+).

    Change-Id: I3e00daa1b877a7faae1370f08b2d9c5bd7795c5f
    Closes-Bug: #1987685
    Related-Bug: #1972022
    (cherry picked from commit 4e53bea0769d6c8c37e1e2a612c191961e738222)