Canonical IS recently encountered problems with a Juju 2.0.3 environment rapidly changeing which node was primary. With assistance from Team Juju, we upgraded to mongodb 3.2.12 and juju 2.1. However, we are still having problems with mongodb. Some recent events:
Mar 8 21:28:28 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-0 mongod.37017[24506]: [ReplicationExecutor] transition to PRIMARY
Mar 8 21:28:29 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-2 mongod.37017[24282]: [replExecDBWorker-0] transition to SECONDARY
Mar 8 21:28:34 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-0 mongod.37017[24506]: [rsSync] transition to primary complete; database writes are now permitted
Mar 8 21:28:41 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-2 mongod.37017[24282]: [ReplicationExecutor] transition to ROLLBACK
Mar 8 21:28:53 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-2 mongod.37017[24282]: [ReplicationExecutor] transition to RECOVERING
Mar 8 21:28:53 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-2 mongod.37017[24282]: [ReplicationExecutor] transition to ROLLBACK
Mar 8 21:28:54 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-2 mongod.37017[24282]: [ReplicationExecutor] transition to RECOVERING
Mar 8 21:28:56 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-2 mongod.37017[24282]: [ReplicationExecutor] transition to SECONDARY
Mar 8 21:28:58 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-1 mongod.37017[13540]: [ReplicationExecutor] transition to PRIMARY
Mar 8 21:28:59 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-1 mongod.37017[13540]: [rsSync] transition to primary complete; database writes are now permitted
Mar 8 21:29:06 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-0 mongod.37017[32554]: [ReplicationExecutor] transition to RECOVERING
Mar 8 21:29:06 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-0 mongod.37017[32554]: [ReplicationExecutor] transition to SECONDARY
Mar 8 21:29:06 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-0 mongod.37017[32554]: [ReplicationExecutor] transition to STARTUP2
Mar 8 21:31:06 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-2 mongod.37017[24282]: [ReplicationExecutor] transition to PRIMARY
Mar 8 21:31:15 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-1 mongod.37017[13540]: [replExecDBWorker-0] transition to SECONDARY
Mar 8 21:31:16 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-2 mongod.37017[24282]: [rsSync] transition to primary complete; database writes are now permitted
Mar 8 21:31:25 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-1 mongod.37017[13540]: [ReplicationExecutor] transition to ROLLBACK
Mar 8 21:31:26 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-1 mongod.37017[13540]: [ReplicationExecutor] transition to RECOVERING
Mar 8 21:31:26 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-1 mongod.37017[13540]: [ReplicationExecutor] transition to SECONDARY
Mar 8 21:31:34 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-0 mongod.37017[575]: [ReplicationExecutor] transition to RECOVERING
Mar 8 21:31:34 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-0 mongod.37017[575]: [ReplicationExecutor] transition to SECONDARY
Mar 8 21:31:34 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-0 mongod.37017[575]: [ReplicationExecutor] transition to STARTUP2
Mar 8 21:31:36 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-1 mongod.37017[11744]: [ReplicationExecutor] transition to RECOVERING
Mar 8 21:31:36 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-1 mongod.37017[11744]: [ReplicationExecutor] transition to SECONDARY
Mar 8 21:31:36 juju-4da59b22-9710-4e69-840a-be49ee864a97-machine-1 mongod.37017[11744]: [ReplicationExecutor] transition to STARTUP2
mongodb logging from 21:00 onwards is at https:/ /private- fileshare. canonical. com/~pjdc/ bug1671258/
The primary logs tend to be very large due to the default slow query logging at 100ms.