Comment 1 for bug 2047297

Revision history for this message
Boris Lukashev (rageltman) wrote (last edit ):

Something's rotten in the state of Rabbit... any action taken by Kolla-Ansible for RabbitMQ fails:
```
TASK [service-rabbitmq : nova | Ensure RabbitMQ users exist] **************************************************************************************************************************************************************************************************
skipping: [ctl01] => (item=None)
skipping: [ctl01]
skipping: [ctl02] => (item=None)
skipping: [ctl02]
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (5 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (4 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (3 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (2 retries left).
FAILED - RETRYING: [ctl00]: nova | Ensure RabbitMQ users exist (1 retries left).
failed: [ctl00] (item=None) => {"attempts": 5, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
fatal: [ctl00 -> {{ service_rabbitmq_delegate_host }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
```
while running `list_users` on the host returns:
```
docker exec -ti rabbitmq /usr/sbin/rabbitmqctl -q -n rabbit list_users
user tags
openstack [administrator]
```

Kolla-ansible cannot reconfigure RabbitMQ either:
```
TASK [rabbitmq : Put RabbitMQ node into maintenance mode] *****************************************************************************************************************************************************************************************************
fatal: [ctl00]: FAILED! => {"action": "community.rabbitmq.rabbitmq_upgrade", "changed": false, "cmd": "/usr/sbin/rabbitmqctl list_feature_flags -q", "msg": "Error: {:badrpc, :nodedown}\nArguments given:\n\tlist_feature_flags -q\n\n\u001b[1mUsage\u001b[0m\n\nrabbitmqctl [--node <node>] [--longnames] [--quiet] list_feature_flags [<column> ...] [--timeout <timeout>]", "rc": 64, "stderr": "Error: {:badrpc, :nodedown}\nArguments given:\n\tlist_feature_flags -q\n\n\u001b[1mUsage\u001b[0m\n\nrabbitmqctl [--node <node>] [--longnames] [--quiet] list_feature_flags [<column> ...] [--timeout <timeout>]\n", "stderr_lines": ["Error: {:badrpc, :nodedown}", "Arguments given:", "\tlist_feature_flags -q", "", "\u001b[1mUsage\u001b[0m", "", "rabbitmqctl [--node <node>] [--longnames] [--quiet] list_feature_flags [<column> ...] [--timeout <timeout>]"], "stdout": "", "stdout_lines": []}
```
leaving the cloud in a somewhat strange and very down state while the cluster reports up and not in maintenance/showing alarms:
```
Basics

Cluster name: rabbit@ctl01
Total CPU cores available cluster-wide: 96

Disk Nodes

rabbit@ctl00
rabbit@ctl01
rabbit@ctl02

Running Nodes

rabbit@ctl00
rabbit@ctl01
rabbit@ctl02

Versions

rabbit@ctl00: RabbitMQ 3.10.24 on Erlang 25.3.2.3
rabbit@ctl01: RabbitMQ 3.10.24 on Erlang 25.3.2.3
rabbit@ctl02: RabbitMQ 3.10.24 on Erlang 25.3.2.3

CPU Cores

Node: rabbit@ctl00, available CPU cores: 32
Node: rabbit@ctl01, available CPU cores: 32
Node: rabbit@ctl02, available CPU cores: 32

Maintenance status

Node: rabbit@ctl00, status: not under maintenance
Node: rabbit@ctl01, status: not under maintenance
Node: rabbit@ctl02, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@ctl00, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@ctl00, interface: x.y.122.10, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@ctl00, interface: x.y.122.10, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@ctl01, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@ctl01, interface: x.y.122.11, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@ctl01, interface: x.y.122.11, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@ctl02, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@ctl02, interface: x.y.122.12, port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@ctl02, interface: x.y.122.12, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: classic_queue_type_delivery_support, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled
```