OpenStack RabbitMQ Server Charm

Leader-settings-changed hook failed, cluster_status exit code 64

Bug #1976523 reported by Alexander Balderson
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack RabbitMQ Server Charm
Confirmed
Undecided
Unassigned

Bug Description

Running a deployment of Focal Yoga on the stable channels (yoga/stable) and rabbit on the channel 3.9/stable one rabbit unit, a non leader, failed on the leader-setting-changed hook.

The juju logs from the unit show:
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'.
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed Arguments given:
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed cluster_status --formatter=json
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed Usage
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed rabbitmqctl [--node <node>] [--longnames] [--quiet] cluster_status [--timeout <timeout>]
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed Traceback (most recent call last):
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 1062, in <module>
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed hooks.execute(sys.argv)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/charmhelpers/core/hookenv.py", line 963, in execute
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed self._hooks[hook_name]()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1474, in wrapped_f
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed f(*args, **kwargs)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 992, in leader_settings_changed
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed check_coordinated_functions()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 217, in check_coordinated_functions
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed coordinated_cluster()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 205, in coordinated_cluster
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed rabbit.join_leader()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1024, in join_leader
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed if clustered_with_leader():
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 990, in clustered_with_leader
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed return node in running_nodes()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/charmhelpers/core/hookenv.py", line 85, in wrapper
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed res = func(*args, **kwargs)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1345, in running_nodes
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed return nodes(get_running=True)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/charmhelpers/core/hookenv.py", line 85, in wrapper
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed res = func(*args, **kwargs)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1311, in nodes
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed status = get_cluster_status()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1296, in get_cluster_status
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed output = subprocess.check_output(
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/usr/lib/python3.8/subprocess.py", line 516, in run
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed raise CalledProcessError(retcode, process.args,
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed subprocess.CalledProcessError: Command '['/usr/sbin/rabbitmqctl', 'cluster_status', '--formatter=json']' returned non-zero exit status 64.
unit-rabbitmq-server-2: 08:48:12 ERROR juju.worker.uniter.operation hook "leader-settings-changed" (via explicit, bespoke hook script) failed: exit status 1

Nothing else stands out to me in the rabbit logs on that unit, but rabbit looks like it was asked to stop about 10 min earlier, and then this call required rabbit, so maybe it was not running?

Crashdump can be found at: https://oil-jenkins.canonical.com/artifacts/3b62889d-bc01-4974-b96e-3cc137e7781a/generated/generated/openstack/juju-crashdump-openstack-2022-06-01-08.50.21.tar.gz
and the unit in question is rabbitmq-server 2.

See original description
Alexander Balderson (asbalderson)
description: updated
Revision history for this message
Moises Emilio Benzan Mora (moisesbenzan) wrote :

Seen again on: https://solutions.qa.canonical.com/v2/testruns/43d81b3b-6dc0-4d12-a5f3-0682983f813e

Artifacts: https://oil-jenkins.canonical.com/artifacts/43d81b3b-6dc0-4d12-a5f3-0682983f813e/index.html

Chris Johnston (cjohnston)
Changed in charm-rabbitmq-server:
status: New → Confirmed
Revision history for this message
Abdullah (motjuste) wrote :

More identified occurrences of this bug can be found here: https://solutions.qa.canonical.com/bugs/1976523

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.