Solutions QA team has a failed run, where while trying to deploy Openstack Yoga on top of MAAS (3.3.4) using Juju 3.1.6, rabbitmq-server/0 was in an error state with the message hook failed: "amqp-relation-changed".
From the logs:
unit-rabbitmq-server-0: 07:04:29 DEBUG juju.machinelock machine lock "machine-lock" acquired for rabbitmq-server/0 uniter (run relation-changed (222; unit: gnocchi/2) hook)
unit-rabbitmq-server-0: 07:04:29 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (222; unit: gnocchi/2) hook" for rabbitmq-server/0
unit-rabbitmq-server-0: 07:04:29 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (222; unit: gnocchi/2) hook" for rabbitmq-server/0
unit-rabbitmq-server-0: 07:04:29 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running amqp-relation-changed hook for gnocchi/2
unit-rabbitmq-server-0: 07:04:29 DEBUG juju.worker.uniter.runner starting jujuc server {unix @/var/lib/juju/agents/unit-rabbitmq-server-0/agent.socket <nil>}
unit-rabbitmq-server-0: 07:04:29 INFO unit.rabbitmq-server/0.juju-log amqp:222: coordinator.Serial Loading state
unit-rabbitmq-server-0: 07:04:29 INFO unit.rabbitmq-server/0.juju-log amqp:222: coordinator.Serial Granted restart
unit-rabbitmq-server-0: 07:04:30 INFO unit.rabbitmq-server/0.juju-log amqp:222: coordinator.Serial Released local restart lock
unit-rabbitmq-server-0: 07:04:30 INFO unit.rabbitmq-server/0.juju-log amqp:222: coordinator.Serial Publishing state
unit-rabbitmq-server-0: 07:04:30 DEBUG unit.rabbitmq-server/0.juju-log amqp:222: Rerunning update_clients as initial update not yet performed
unit-rabbitmq-server-0: 07:04:30 DEBUG unit.rabbitmq-server/0.juju-log amqp:222: Not updating clients: leader node is ready:False, client node is ready:False
unit-rabbitmq-server-0: 07:04:31 DEBUG unit.rabbitmq-server/0.juju-log amqp:222: Calculating erl vm io thread pool size based on num_cpus=20 and multiplier=24
unit-rabbitmq-server-0: 07:04:31 DEBUG unit.rabbitmq-server/0.amqp-relation-changed lxc
unit-rabbitmq-server-0: 07:04:31 DEBUG unit.rabbitmq-server/0.amqp-relation-changed lxc
unit-rabbitmq-server-0: 07:04:31 DEBUG unit.rabbitmq-server/0.juju-log amqp:222: erl vm io thread pool size = 48 (capped=True)
unit-rabbitmq-server-0: 07:04:31 DEBUG unit.rabbitmq-server/0.juju-log amqp:222: Checking for minimum of 3 peer units
unit-rabbitmq-server-0: 07:04:31 DEBUG unit.rabbitmq-server/0.juju-log amqp:222: Sufficient number of peer units to form cluster 3
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'.
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed Arguments given:
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed cluster_status --formatter=json
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed [1mUsage[0m
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed rabbitmqctl [--node <node>] [--longnames] [--quiet] cluster_status [--timeout <timeout>]
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed Traceback (most recent call last):
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/hooks/amqp-relation-changed", line 1243, in <module>
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed rabbit.assess_status(rabbit.ConfigRenderer(rabbit.CONFIG_FILES()))
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/hooks/rabbit_utils.py", line 1611, in assess_status
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed assess_status_func(configs)()
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/hooks/rabbit_utils.py", line 1632, in _assess_status_func
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed state, message = _determine_os_workload_status(
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/charmhelpers/contrib/openstack/utils.py", line 1036, in _determine_os_workload_status
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed state, message = _ows_check_charm_func(
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/charmhelpers/contrib/openstack/utils.py", line 1200, in _ows_check_charm_func
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed charm_state, charm_message = charm_func_with_configs()
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/charmhelpers/contrib/openstack/utils.py", line 1037, in <lambda>
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed state, message, lambda: charm_func(configs))
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/hooks/rabbit_utils.py", line 1510, in assess_cluster_status
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed if not clustered():
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/charmhelpers/core/hookenv.py", line 85, in wrapper
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed res = func(*args, **kwargs)
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/hooks/rabbit_utils.py", line 1487, in clustered
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed if len(running_nodes()) > 1:
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/charmhelpers/core/hookenv.py", line 85, in wrapper
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed res = func(*args, **kwargs)
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/hooks/rabbit_utils.py", line 1429, in running_nodes
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed _nodes = nodes(get_running=True)
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/charmhelpers/core/hookenv.py", line 85, in wrapper
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed res = func(*args, **kwargs)
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/hooks/rabbit_utils.py", line 1395, in nodes
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed status = get_cluster_status()
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/hooks/rabbit_utils.py", line 1380, in get_cluster_status
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed output = subprocess.check_output(
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed File "/usr/lib/python3.10/subprocess.py", line 526, in run
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed raise CalledProcessError(retcode, process.args,
unit-rabbitmq-server-0: 07:04:32 WARNING unit.rabbitmq-server/0.amqp-relation-changed subprocess.CalledProcessError: Command '['/usr/sbin/rabbitmqctl', 'cluster_status', '--formatter=json']' returned non-zero exit status 64.
unit-rabbitmq-server-0: 07:04:32 ERROR juju.worker.uniter.operation hook "amqp-relation-changed" (via explicit, bespoke hook script) failed: exit status 1
unit-rabbitmq-server-0: 07:04:32 DEBUG juju.machinelock created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
unit-rabbitmq-server-0: 07:04:32 DEBUG juju.machinelock machine lock "machine-lock" released for rabbitmq-server/0 uniter (run relation-changed (222; unit: gnocchi/2) hook)
unit-rabbitmq-server-0: 07:04:32 DEBUG juju.worker.uniter.operation lock released for rabbitmq-server/0
unit-rabbitmq-server-0: 07:04:32 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-rabbitmq-server-0: 07:04:32 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "amqp-relation-changed"
unit-rabbitmq-server-0: 07:08:32 DEBUG juju.worker.uniter.remotestate update status timer triggered for rabbitmq-server/0
unit-rabbitmq-server-0: 07:08:32 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
Failed run:
https://solutions.qa.canonical.com/testruns/9f57ba93-eac5-482d-bd92-1030c0b99722
Logs:
https://oil-jenkins.canonical.com/artifacts/9f57ba93-eac5-482d-bd92-1030c0b99722/index.html
