Charmed rabbitmq does not configure queue_master_locator and this results in the majority of queue masters being put on the node where the queues are declared. A lot of the times the node where the queues are defined is simply the first node listed in rabbit client configurations such as nova.conf.
This leads to often unbalanced clusters with queue masters concentrated on one node. That node would have to handle most of the connections and have high load in cpu and memory, and at the time when that node is down, a majority of queue masters would have to be moved, putting even more pressure on the cluster.
Configuring queue_master_locator to something like "min-master" can relieviate the problem. Otherwise, load balancer is another option.
The rabbitmqctl_report file is attached for reference.
# Queue masters are mostly on 11-lxd-8
> cat rabbitmqctl_report | awk -F'\t' '/^Queues on openstack/ {a=1}; a && /rabbit/ {split($1,s,"."); b[s[1]]++}; NF==0 {a=0}; END {for(i in b) {print i, b[i]}}'
<rabbit@juju-0aa49a-9-lxd-9 15
<rabbit@juju-0aa49a-11-lxd-8 15426
<rabbit@juju-0aa49a-10-lxd-8 309
# Most connections are made to 11-lxd-8
> cat rabbitmqctl_report| awk -F'\t' '/^Connections/ {a=1}; a && /rabbit/ {split($1,s,"."); b[s[1]]++}; NF==0 {a=0}; END {for(i in b) {print i, b[i]}}'
<rabbit@juju-0aa49a-9-lxd-9 8763
<rabbit@juju-0aa49a-11-lxd-8 14843
<rabbit@juju-0aa49a-10-lxd-8 6413
# 11-lxd-8 uses the most memory
> cat rabbitmqctl_report| awk '/^Status/ {a=1; print}; a && /total,/; !NF {a=0}'
Status of node 'rabbit@juju-0aa49a-10-lxd-8'
[{total,3536412448},
Status of node 'rabbit@juju-0aa49a-11-lxd-8'
[{total,7030459176},
Status of node 'rabbit@juju-0aa49a-9-lxd-9'
[{total,4256377400},
Patch: https:/