Something's rotten in the state of Rabbit... any action taken by Kolla-Ansible for RabbitMQ fails:
```
TASK [service-rabbitmq : nova | Ensure RabbitMQ users exist] **************************************************************************************************************************************************************************************************
skipping: [ctl01] => (item=None)
skipping: [ctl01]
skipping: [ctl02] => (item=None)
skipping: [ctl02]
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (5 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (4 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (3 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (2 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (1 retries left).
failed: [ctl00] (item=None) => {"attempts": 5, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
fatal: [ctl00 -> {{ service_rabbitmq_delegate_host }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
```
while running `list_users` on the host returns:
```
docker exec -ti rabbitmq /usr/sbin/rabbitmqctl -q -n rabbit list_users
user tags
openstack [administrator]
```
Kolla-ansible cannot reconfigure RabbitMQ either:
```
TASK [rabbitmq : Put RabbitMQ node into maintenance mode] *****************************************************************************************************************************************************************************************************
fatal: [ctl00]: FAILED! => {"action": "community.rabbitmq.rabbitmq_upgrade", "changed": false, "cmd": "/usr/sbin/rabbitmqctl list_feature_flags -q", "msg": "Error: {:badrpc, :nodedown}\nArguments given:\n\tlist_feature_flags -q\n\n\u001b[1mUsage\u001b[0m\n\nrabbitmqctl [--node <node>] [--longnames] [--quiet] list_feature_flags [<column> ...] [--timeout <timeout>]", "rc": 64, "stderr": "Error: {:badrpc, :nodedown}\nArguments given:\n\tlist_feature_flags -q\n\n\u001b[1mUsage\u001b[0m\n\nrabbitmqctl [--node <node>] [--longnames] [--quiet] list_feature_flags [<column> ...] [--timeout <timeout>]\n", "stderr_lines": ["Error: {:badrpc, :nodedown}", "Arguments given:", "\tlist_feature_flags -q", "", "\u001b[1mUsage\u001b[0m", "", "rabbitmqctl [--node <node>] [--longnames] [--quiet] list_feature_flags [<column> ...] [--timeout <timeout>]"], "stdout": "", "stdout_lines": []}
```
leaving the cloud in a somewhat strange and very down state while the cluster reports up and not in maintenance/showing alarms:
```
Basics
Cluster name: rabbit@ctl01
Total CPU cores available cluster-wide: 96
Disk Nodes
rabbit@ctl00
rabbit@ctl01
rabbit@ctl02
Running Nodes
rabbit@ctl00
rabbit@ctl01
rabbit@ctl02
Versions
rabbit@ctl00: RabbitMQ 3.10.24 on Erlang 25.3.2.3
rabbit@ctl01: RabbitMQ 3.10.24 on Erlang 25.3.2.3
rabbit@ctl02: RabbitMQ 3.10.24 on Erlang 25.3.2.3
CPU Cores
Node: rabbit@ctl00, available CPU cores: 32
Node: rabbit@ctl01, available CPU cores: 32
Node: rabbit@ctl02, available CPU cores: 32
Maintenance status
Node: rabbit@ctl00, status: not under maintenance
Node: rabbit@ctl01, status: not under maintenance
Node: rabbit@ctl02, status: not under maintenance
Alarms
(none)
Network Partitions
(none)
Listeners
Node: rabbit@ctl00, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@ctl00, interface: x.y.122.10, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@ctl00, interface: x.y.122.10, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@ctl01, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@ctl01, interface: x.y.122.11, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@ctl01, interface: x.y.122.11, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@ctl02, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@ctl02, interface: x.y.122.12, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@ctl02, interface: x.y.122.12, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Something's rotten in the state of Rabbit... any action taken by Kolla-Ansible for RabbitMQ fails: ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ***** rabbitmq_ delegate_ host }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} rabbitmqctl -q -n rabbit list_users
```
TASK [service-rabbitmq : nova | Ensure RabbitMQ users exist] *******
skipping: [ctl01] => (item=None)
skipping: [ctl01]
skipping: [ctl02] => (item=None)
skipping: [ctl02]
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (5 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (4 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (3 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (2 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (1 retries left).
failed: [ctl00] (item=None) => {"attempts": 5, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
fatal: [ctl00 -> {{ service_
```
while running `list_users` on the host returns:
```
docker exec -ti rabbitmq /usr/sbin/
user tags
openstack [administrator]
```
Kolla-ansible cannot reconfigure RabbitMQ either: ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* * rabbitmq. rabbitmq_ upgrade" , "changed": false, "cmd": "/usr/sbin/ rabbitmqctl list_feature_flags -q", "msg": "Error: {:badrpc, :nodedown} \nArguments given:\ n\tlist_ feature_ flags -q\n\n\ u001b[1mUsage\ u001b[0m\ n\nrabbitmqctl [--node <node>] [--longnames] [--quiet] list_feature_flags [<column> ...] [--timeout <timeout>]", "rc": 64, "stderr": "Error: {:badrpc, :nodedown} \nArguments given:\ n\tlist_ feature_ flags -q\n\n\ u001b[1mUsage\ u001b[0m\ n\nrabbitmqctl [--node <node>] [--longnames] [--quiet] list_feature_flags [<column> ...] [--timeout <timeout>]\n", "stderr_lines": ["Error: {:badrpc, :nodedown}", "Arguments given:", "\tlist_ feature_ flags -q", "", "\u001b[ 1mUsage\ u001b[0m" , "", "rabbitmqctl [--node <node>] [--longnames] [--quiet] list_feature_flags [<column> ...] [--timeout <timeout>]"], "stdout": "", "stdout_lines": []}
```
TASK [rabbitmq : Put RabbitMQ node into maintenance mode] *******
fatal: [ctl00]: FAILED! => {"action": "community.
```
leaving the cloud in a somewhat strange and very down state while the cluster reports up and not in maintenance/showing alarms:
```
Basics
Cluster name: rabbit@ctl01
Total CPU cores available cluster-wide: 96
Disk Nodes
rabbit@ctl00
rabbit@ctl01
rabbit@ctl02
Running Nodes
rabbit@ctl00
rabbit@ctl01
rabbit@ctl02
Versions
rabbit@ctl00: RabbitMQ 3.10.24 on Erlang 25.3.2.3
rabbit@ctl01: RabbitMQ 3.10.24 on Erlang 25.3.2.3
rabbit@ctl02: RabbitMQ 3.10.24 on Erlang 25.3.2.3
CPU Cores
Node: rabbit@ctl00, available CPU cores: 32
Node: rabbit@ctl01, available CPU cores: 32
Node: rabbit@ctl02, available CPU cores: 32
Maintenance status
Node: rabbit@ctl00, status: not under maintenance
Node: rabbit@ctl01, status: not under maintenance
Node: rabbit@ctl02, status: not under maintenance
Alarms
(none)
Network Partitions
(none)
Listeners
Node: rabbit@ctl00, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@ctl00, interface: x.y.122.10, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@ctl00, interface: x.y.122.10, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@ctl01, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@ctl01, interface: x.y.122.11, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@ctl01, interface: x.y.122.11, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@ctl02, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@ctl02, interface: x.y.122.12, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@ctl02, interface: x.y.122.12, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Feature flags
Flag: classic_ mirrored_ queue_version, state: enabled queue_type_ delivery_ support, state: enabled _metric, state: enabled get_metric, state: enabled default_ bindings, state: enabled mode_status, state: enabled host_metadata, state: enabled
Flag: classic_
Flag: drop_unroutable
Flag: empty_basic_
Flag: implicit_
Flag: maintenance_
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_
```