Something's rotten in the state of Rabbit... any action taken by Kolla-Ansible for RabbitMQ fails: ``` TASK [service-rabbitmq : nova | Ensure RabbitMQ users exist] ************************************************************************************************************************************************************************************************** skipping: [ctl01] => (item=None) skipping: [ctl01] skipping: [ctl02] => (item=None) skipping: [ctl02] FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (5 retries left). FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (4 retries left). FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (3 retries left). FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (2 retries left). FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (1 retries left). failed: [ctl00] (item=None) => {"attempts": 5, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} fatal: [ctl00 -> {{ service_rabbitmq_delegate_host }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ``` while running `list_users` on the host returns: ``` docker exec -ti rabbitmq /usr/sbin/rabbitmqctl -q -n rabbit list_users user tags openstack [administrator] ``` Kolla-ansible cannot reconfigure RabbitMQ either: ``` TASK [rabbitmq : Put RabbitMQ node into maintenance mode] ***************************************************************************************************************************************************************************************************** fatal: [ctl00]: FAILED! => {"action": "community.rabbitmq.rabbitmq_upgrade", "changed": false, "cmd": "/usr/sbin/rabbitmqctl list_feature_flags -q", "msg": "Error: {:badrpc, :nodedown}\nArguments given:\n\tlist_feature_flags -q\n\n\u001b[1mUsage\u001b[0m\n\nrabbitmqctl [--node ] [--longnames] [--quiet] list_feature_flags [ ...] [--timeout ]", "rc": 64, "stderr": "Error: {:badrpc, :nodedown}\nArguments given:\n\tlist_feature_flags -q\n\n\u001b[1mUsage\u001b[0m\n\nrabbitmqctl [--node ] [--longnames] [--quiet] list_feature_flags [ ...] [--timeout ]\n", "stderr_lines": ["Error: {:badrpc, :nodedown}", "Arguments given:", "\tlist_feature_flags -q", "", "\u001b[1mUsage\u001b[0m", "", "rabbitmqctl [--node ] [--longnames] [--quiet] list_feature_flags [ ...] [--timeout ]"], "stdout": "", "stdout_lines": []} ``` leaving the cloud in a somewhat strange and very down state while the cluster reports up and not in maintenance/showing alarms: ``` Basics Cluster name: rabbit@ctl01 Total CPU cores available cluster-wide: 96 Disk Nodes rabbit@ctl00 rabbit@ctl01 rabbit@ctl02 Running Nodes rabbit@ctl00 rabbit@ctl01 rabbit@ctl02 Versions rabbit@ctl00: RabbitMQ 3.10.24 on Erlang 25.3.2.3 rabbit@ctl01: RabbitMQ 3.10.24 on Erlang 25.3.2.3 rabbit@ctl02: RabbitMQ 3.10.24 on Erlang 25.3.2.3 CPU Cores Node: rabbit@ctl00, available CPU cores: 32 Node: rabbit@ctl01, available CPU cores: 32 Node: rabbit@ctl02, available CPU cores: 32 Maintenance status Node: rabbit@ctl00, status: not under maintenance Node: rabbit@ctl01, status: not under maintenance Node: rabbit@ctl02, status: not under maintenance Alarms (none) Network Partitions (none) Listeners Node: rabbit@ctl00, interface: [::], port: 15672, protocol: http, purpose: HTTP API Node: rabbit@ctl00, interface: x.y.122.10, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Node: rabbit@ctl00, interface: x.y.122.10, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Node: rabbit@ctl01, interface: [::], port: 15672, protocol: http, purpose: HTTP API Node: rabbit@ctl01, interface: x.y.122.11, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Node: rabbit@ctl01, interface: x.y.122.11, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Node: rabbit@ctl02, interface: [::], port: 15672, protocol: http, purpose: HTTP API Node: rabbit@ctl02, interface: x.y.122.12, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Node: rabbit@ctl02, interface: x.y.122.12, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Feature flags Flag: classic_mirrored_queue_version, state: enabled Flag: classic_queue_type_delivery_support, state: enabled Flag: drop_unroutable_metric, state: enabled Flag: empty_basic_get_metric, state: enabled Flag: implicit_default_bindings, state: enabled Flag: maintenance_mode_status, state: enabled Flag: quorum_queue, state: enabled Flag: stream_queue, state: enabled Flag: user_limits, state: enabled Flag: virtual_host_metadata, state: enabled ```