| 2022-06-01 15:09:29 |
Alexander Balderson |
description |
Running a deployment of Focal Yoga on the stable channels (yoga/stable) and rabbit on the channel 3.9/stable one rabbit unit, a non leader, failed on the leader-setting-changed hook.
The juju logs from the unit show:
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'.
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed Arguments given:
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed cluster_status --formatter=json
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed [1mUsage[0m
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed rabbitmqctl [--node <node>] [--longnames] [--quiet] cluster_status [--timeout <timeout>]
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed Traceback (most recent call last):
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 1062, in <module>
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed hooks.execute(sys.argv)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/charmhelpers/core/hookenv.py", line 963, in execute
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed self._hooks[hook_name]()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1474, in wrapped_f
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed f(*args, **kwargs)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 992, in leader_settings_changed
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed check_coordinated_functions()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 217, in check_coordinated_functions
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed coordinated_cluster()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 205, in coordinated_cluster
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed rabbit.join_leader()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1024, in join_leader
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed if clustered_with_leader():
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 990, in clustered_with_leader
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed return node in running_nodes()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/charmhelpers/core/hookenv.py", line 85, in wrapper
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed res = func(*args, **kwargs)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1345, in running_nodes
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed return nodes(get_running=True)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/charmhelpers/core/hookenv.py", line 85, in wrapper
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed res = func(*args, **kwargs)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1311, in nodes
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed status = get_cluster_status()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1296, in get_cluster_status
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed output = subprocess.check_output(
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/usr/lib/python3.8/subprocess.py", line 516, in run
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed raise CalledProcessError(retcode, process.args,
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed subprocess.CalledProcessError: Command '['/usr/sbin/rabbitmqctl', 'cluster_status', '--formatter=json']' returned non-zero exit status 64.
unit-rabbitmq-server-2: 08:48:12 ERROR juju.worker.uniter.operation hook "leader-settings-changed" (via explicit, bespoke hook script) failed: exit status 1
Nothing else stands out to me in the rabbit logs on that unit, but rabbit looks like it was asked to stop about 10 min earlier, and then this call required rabbit, so maybe it was not running? |
Running a deployment of Focal Yoga on the stable channels (yoga/stable) and rabbit on the channel 3.9/stable one rabbit unit, a non leader, failed on the leader-setting-changed hook.
The juju logs from the unit show:
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'.
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed Arguments given:
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed cluster_status --formatter=json
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed [1mUsage[0m
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed rabbitmqctl [--node <node>] [--longnames] [--quiet] cluster_status [--timeout <timeout>]
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed Traceback (most recent call last):
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 1062, in <module>
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed hooks.execute(sys.argv)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/charmhelpers/core/hookenv.py", line 963, in execute
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed self._hooks[hook_name]()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1474, in wrapped_f
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed f(*args, **kwargs)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 992, in leader_settings_changed
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed check_coordinated_functions()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 217, in check_coordinated_functions
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed coordinated_cluster()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 205, in coordinated_cluster
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed rabbit.join_leader()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1024, in join_leader
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed if clustered_with_leader():
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 990, in clustered_with_leader
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed return node in running_nodes()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/charmhelpers/core/hookenv.py", line 85, in wrapper
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed res = func(*args, **kwargs)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1345, in running_nodes
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed return nodes(get_running=True)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/charmhelpers/core/hookenv.py", line 85, in wrapper
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed res = func(*args, **kwargs)
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1311, in nodes
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed status = get_cluster_status()
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabbit_utils.py", line 1296, in get_cluster_status
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed output = subprocess.check_output(
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed File "/usr/lib/python3.8/subprocess.py", line 516, in run
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed raise CalledProcessError(retcode, process.args,
unit-rabbitmq-server-2: 08:48:11 WARNING unit.rabbitmq-server/2.leader-settings-changed subprocess.CalledProcessError: Command '['/usr/sbin/rabbitmqctl', 'cluster_status', '--formatter=json']' returned non-zero exit status 64.
unit-rabbitmq-server-2: 08:48:12 ERROR juju.worker.uniter.operation hook "leader-settings-changed" (via explicit, bespoke hook script) failed: exit status 1
Nothing else stands out to me in the rabbit logs on that unit, but rabbit looks like it was asked to stop about 10 min earlier, and then this call required rabbit, so maybe it was not running?
Crashdump can be found at: https://oil-jenkins.canonical.com/artifacts/3b62889d-bc01-4974-b96e-3cc137e7781a/generated/generated/openstack/juju-crashdump-openstack-2022-06-01-08.50.21.tar.gz
and the unit in question is rabbitmq-server 2. |
|
