MongoDB charm deploys broken replicaset

[Mario Splivalo]

When deploying replicaset using juju one ends up with broken replicaset very often:

Deploying 3 replicasets (mongo-a, mongo-b, mongo-c):

for i in {a..c}; do echo $i; juju deploy mongodb mongo-${i}; done

After verifying all the units are up and mongodb is running on them, adding another unit:

for i in {a..c}; do echo $i; juju add-unit -n 1 mongo-${i}; done

And another:

for i in {a..c}; do echo $i; juju add-unit -n 1 mongo-${i}; done

After waiting some time for relations to settle down, check shows that neither replicasets are healthy:

mario@buntor ~> for i in {a..c}; do echo $i; juju ssh mongo-${i}/0 -- mongo --eval "'printjson(rs.status())'" | grep stateStr | sort | uniq -c; done
a
Warning: Permanently added '10.0.3.176' (ECDSA) to the list of known hosts.
Connection to 10.0.3.176 closed.
      2 "stateStr" : "PRIMARY",
      1 "stateStr" : "SECONDARY",
b
Warning: Permanently added '10.0.3.72' (ECDSA) to the list of known hosts.
Connection to 10.0.3.72 closed.
      1 "stateStr" : "FATAL",
      2 "stateStr" : "PRIMARY",
c
Warning: Permanently added '10.0.3.226' (ECDSA) to the list of known hosts.
Connection to 10.0.3.226 closed.
      2 "stateStr" : "PRIMARY",
      1 "stateStr" : "SECONDARY",

Each replicaset consists of two primaries, which is not allowed.

[Mario Splivalo (mariosplivalo)]

This is now fixed so that charm avoids most of the race conditions and in most cases deploying a 3-unit replicaset will work.
However, in rare cases issues can arise so one should firstly deploy one or two units, and then add remaining one(s).