juju status fails, mongo replication error

[Alexander Balderson]

During a standard deployment with an HA controller, juju status started reporting that "ERROR:root:ERROR not master and slaveOk=false"
This first caused juju wait to fail, but then the end of the test run we then run juju status against the models, which also failed
Looking down the line in the test log collection it took about 20 min before a juju status call succeeded.

some short googling shows that this error can result when trying to read from a mongo replica that is not the master, but the replica isnt set with read permissions. the controller logs dont seem like they show the replicas being in a weird state though. The logs are attached.

Testrun can be found at:
https://solutions.qa.canonical.com/testruns/testRun/7355daa0-d965-4d10-9a9b-e319631eda01

and all logs can be found at:
https://oil-jenkins.canonical.com/artifacts/7355daa0-d965-4d10-9a9b-e319631eda01/index.html

[John A Meinel (jameinel)]

We generally never run reading from the secondaries, so the real question is why we didn't notice that the primary had changed and switched our reads over to the primary.