MongoDB charm deploys broken replicaset
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
mongodb (Juju Charms Collection) |
Fix Released
|
High
|
Mario Splivalo |
Bug Description
When deploying replicaset using juju one ends up with broken replicaset very often:
Deploying 3 replicasets (mongo-a, mongo-b, mongo-c):
for i in {a..c}; do echo $i; juju deploy mongodb mongo-${i}; done
After verifying all the units are up and mongodb is running on them, adding another unit:
for i in {a..c}; do echo $i; juju add-unit -n 1 mongo-${i}; done
And another:
for i in {a..c}; do echo $i; juju add-unit -n 1 mongo-${i}; done
After waiting some time for relations to settle down, check shows that neither replicasets are healthy:
mario@buntor ~> for i in {a..c}; do echo $i; juju ssh mongo-${i}/0 -- mongo --eval "'printjson(
a
Warning: Permanently added '10.0.3.176' (ECDSA) to the list of known hosts.
Connection to 10.0.3.176 closed.
2 "stateStr" : "PRIMARY",
1 "stateStr" : "SECONDARY",
b
Warning: Permanently added '10.0.3.72' (ECDSA) to the list of known hosts.
Connection to 10.0.3.72 closed.
1 "stateStr" : "FATAL",
2 "stateStr" : "PRIMARY",
c
Warning: Permanently added '10.0.3.226' (ECDSA) to the list of known hosts.
Connection to 10.0.3.226 closed.
2 "stateStr" : "PRIMARY",
1 "stateStr" : "SECONDARY",
Each replicaset consists of two primaries, which is not allowed.
Related branches
- Felipe Reyes (community): Approve
- Jorge Niedbalski (community): Approve
- James Page: Pending requested
- Edward Hope-Morley: Pending requested
- Ryan Beisner: Pending requested
-
Diff: 1639 lines (+1138/-145)18 files modified.bzrignore (+5/-0)
README.md (+62/-23)
charm-helpers-sync.yaml (+1/-0)
config.yaml (+1/-1)
hooks/charmhelpers/contrib/hahelpers/cluster.py (+26/-12)
hooks/charmhelpers/contrib/python/packages.py (+80/-0)
hooks/charmhelpers/core/decorators.py (+41/-0)
hooks/charmhelpers/core/host.py (+16/-5)
hooks/charmhelpers/core/templating.py (+1/-1)
hooks/charmhelpers/fetch/__init__.py (+8/-1)
hooks/hooks.py (+322/-92)
setup.cfg (+6/-0)
test_requirements.txt (+1/-1)
tests/00_setup.sh (+0/-9)
tests/03_deploy_replicaset.py (+114/-0)
unit_tests/__init__.py (+2/-0)
unit_tests/test_hooks.py (+341/-0)
unit_tests/test_utils.py (+111/-0)
Changed in mongodb (Juju Charms Collection): | |
status: | Fix Released → Fix Committed |
Changed in mongodb (Juju Charms Collection): | |
importance: | Undecided → High |
Changed in mongodb (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
This is now fixed so that charm avoids most of the race conditions and in most cases deploying a 3-unit replicaset will work.
However, in rare cases issues can arise so one should firstly deploy one or two units, and then add remaining one(s).