MongoDB charm deploys broken replicaset

Bug #1403698 reported by Mario Splivalo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
mongodb (Juju Charms Collection)
High
Mario Splivalo

Bug Description

When deploying replicaset using juju one ends up with broken replicaset very often:

Deploying 3 replicasets (mongo-a, mongo-b, mongo-c):

for i in {a..c}; do echo $i; juju deploy mongodb mongo-${i}; done

After verifying all the units are up and mongodb is running on them, adding another unit:

for i in {a..c}; do echo $i; juju add-unit -n 1 mongo-${i}; done

And another:

for i in {a..c}; do echo $i; juju add-unit -n 1 mongo-${i}; done

After waiting some time for relations to settle down, check shows that neither replicasets are healthy:

mario@buntor ~> for i in {a..c}; do echo $i; juju ssh mongo-${i}/0 -- mongo --eval "'printjson(rs.status())'" | grep stateStr | sort | uniq -c; done
a
Warning: Permanently added '10.0.3.176' (ECDSA) to the list of known hosts.
Connection to 10.0.3.176 closed.
      2 "stateStr" : "PRIMARY",
      1 "stateStr" : "SECONDARY",
b
Warning: Permanently added '10.0.3.72' (ECDSA) to the list of known hosts.
Connection to 10.0.3.72 closed.
      1 "stateStr" : "FATAL",
      2 "stateStr" : "PRIMARY",
c
Warning: Permanently added '10.0.3.226' (ECDSA) to the list of known hosts.
Connection to 10.0.3.226 closed.
      2 "stateStr" : "PRIMARY",
      1 "stateStr" : "SECONDARY",

Each replicaset consists of two primaries, which is not allowed.

Tags: cts Edit Tag help

Related branches

Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

This is now fixed so that charm avoids most of the race conditions and in most cases deploying a 3-unit replicaset will work.
However, in rare cases issues can arise so one should firstly deploy one or two units, and then add remaining one(s).

tags: added: cts
Changed in mongodb (Juju Charms Collection):
status: New → Fix Released
assignee: nobody → Mario Splivalo (mariosplivalo)
Changed in mongodb (Juju Charms Collection):
status: Fix Released → Fix Committed
Changed in mongodb (Juju Charms Collection):
importance: Undecided → High
Changed in mongodb (Juju Charms Collection):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers