Seen twice on hirsute in our test gate [0][1]
## Running Test zaza.openstack.charm_tests.hacluster.tests.HaclusterScaleBackAndForthTest ##
looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}}
Using keystone API V3 (or later) for overcloud auth
looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}}
test_930_scaleback (zaza.openstack.charm_tests.hacluster.tests.HaclusterScaleBackAndForthTest)
Remove one unit, recalculate quorum and re-add one unit.
...
Pausing unit keystone-hacluster/1
Removing keystone/1
Waiting for model to settle
Checking that corosync considers at least one node to be offline
Updating corosync ring
ERROR
======================================================================
ERROR: test_930_scaleback (zaza.openstack.charm_tests.hacluster.tests.HaclusterScaleBackAndForthTest)
Remove one unit, recalculate quorum and re-add one unit.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-router/build/builds/mysql-router/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/charm_tests/hacluster/tests.py", line 141, in test_930_scaleback
raise_on_failure=True)
File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-router/build/builds/mysql-router/.tox/func-target/lib/python3.6/site-packages/zaza/__init__.py", line 48, in _wrapper
return run(_run_it())
File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-router/build/builds/mysql-router/.tox/func-target/lib/python3.6/site-packages/zaza/__init__.py", line 36, in run
return task.result()
File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-router/build/builds/mysql-router/.tox/func-target/lib/python3.6/site-packages/zaza/__init__.py", line 47, in _run_it
return await f(*args, **kwargs)
File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-router/build/builds/mysql-router/.tox/func-target/lib/python3.6/site-packages/zaza/model.py", line 901, in async_run_action_on_leader
raise ActionFailed(action_obj, output=output)
zaza.model.ActionFailed: Run of action "update-ring" with parameters "{'i-really-mean-it': True}" on "keystone-hacluster/0" failed with "Removing node1 from the cluster failed. Command '['crm', '-w', '-F', 'node', 'delete', 'node1']' returned non-zero exit status 1. output=b'Could not remove node1[0] from nodes: Transport endpoint is not connectedCould not remove node1[0] from status: Transport endpoint is not connectedWARNING: "crm_node --force -R node1" failed, rc=1\n'" (id=256 status=failed enqueued=2021-06-10T13:32:21Z started=2021-06-10T13:32:21Z completed=2021-06-10T13:32:24Z output={'Code': '0'})
Note that `node1` is wrongly created by pacemaker because of another bug [2] and the update-ring action is supposed to successfully work around that by removing `node1`.
[0] https://review.opendev.org/c/openstack/charm-mysql-router/+/795226
[1] https://openstack-ci-reports.ubuntu.com/artifacts/74b/795226/3/check/full_model_ha-hirsute-full-ha/74b6249/job-output.txt
[2] lp:1874719
Issue also visible in this gate:
https:/