Comment 24 for bug 1701275

Revision history for this message
Alvaro Uria (aluria) wrote :

In my case, I was not able to reproduce the issue in a staging environment (using an openstack endpoint as backend). Prod env uses MAAS as endpoint.

1)
Prod: juju bootstrap prod-cloud --constraints tags=juju
Staging: juju bootstrap canonistack --constraints "cores=2 mem=4G"

2) juju enable-ha

3)
Prod: node#0 has-vote; node#1,2 "adding-vote". PRIMARY mongo does not show replicasets, but jujud-machine-{1,2} are started and mongo installed (although no machine-{1,2} users are created in juju-db).
Staging:
3.1) iptables --dport 37017 -j DROP
PRIMARY shows replicasets created, but in STARTUP state (no SECONDARY).
Stopping jujud-machine-{1,2}, "apt remove juju-mongodb3.2" and restarting jujud-machine-{1,2} makes juju-mongodb3.2 being reinstalled.

3.2) iptables --dport 17070 -j DROP (no other DROPs)
PRIMARY does not show new replicasets, only the primary mongo.
juju-mongodb3.2 is installed and started, but no machine-{1,2} user exists.
"juju status" shows nodes#1,#2 as "pending" and "juju ssh 1" (or "2") does not work because pub key has not been imported (no access to jujud-machine-0 to gather data).

At any time, flushing iptables rules recovers the environment (juju-db might need a restart on node#1,#2 to complete replication).

TL;DR:
The diversion between Prod and Staging is that Prod's jujud-machine-{1,2} connects to jujud-machine-0 but replicasets are not created in PRIMARY juju-db (needed to step for the other mongos to be able to connect to the primary instance).