RMQ doesn't honor min-cluster-size when a second unit is added

[Alvaro Uria]

In a trusty environment, using the charmstore (so, 18.11):
"""
juju deploy cs:trusty/rabbitmq-server
juju config rabbitmq-server min-cluster-size=3
juju add-relation neutron-api rabbitmq-server
juju add-relation neutron-api percona-cluster
juju add-relation neutron-api keystone
juju add-unit rabbitmq-server
"""

A message that the second unit is waiting for 60 seconds is seen, but both RMQ units eventually settle and join the same cluster (seen via rabbitmqctl cluster_status).

Furthermore, "egrep rabbit /etc/neutron/neutron.conf" shows that the second unit is also configured on rabbit_hosts.

The expectation is similar to Percona-Cluster, when min-cluster-size=3 and there are 2 units of the same application: the second unit is not bootstrapped until 3 units are discovered via the "cluster" relation. I would expect that RMQ does nothing until 3 units are seen, time the 2 units would join the cluster and their IPs be configured on the rabbit_hosts parameter of each API amqp relation.

[Alvaro Uria (aluria)]

For a complete information on applications used to reproduce this issue:
"""
juju deploy cs:trusty/keystone # vip=XX.XX.XX.10
juju deploy cs:trusty/percona-cluster mysql # vip=XX.XX.XX.11
juju deploy cs:trusty/neutron-api # vip=XX.XX.XX.12 neutron-security-groups=True
juju deploy cs:trusty/hacluster keystone-hacluster # cluster_count=1
juju deploy cs:trusty/hacluster mysql-hacluster # cluster_count=1
juju deploy cs:trusty/hacluster neutron-hacluster # cluster_count=1
juju add-relation keystone keystone-hacluster
juju add-relation mysql mysql-hacluster
juju add-relation neutron-api neutron-hacluster
juju add-relation keystone mysql
juju add-relation neutron-api mysql
juju add-relation neutron-api keystone

juju deploy cs:trusty/rabbitmq-server
juju config rabbitmq-server min-cluster-size=3
juju add-relation neutron-api rabbitmq-server
"""

openstack catalog list shows both Identity and Network services with their expected VIP.
"""
juju add-unit rabbitmq-server
"""

Second RMQ unit will join the RMQ cluster and be added to rabbit_hosts in neutron-api, while "juju status" will still show that a 3rd RMQ unit is needed.

[Frode Nordahl (fnordahl)]

The original [0] implementation details that guarded against this has indeed weathered by subsequent changes to the charm. It would be pertinent to restore this behavior.

The gate for re-run of client relations appears to be intact, but relation data is set as clients joins, instead of abstaining and doing the update when all cluster peers are present.

The fix would be to restore these checks in the ``amqp_changed`` function.

0: https://review.openstack.org/#/c/422318/