cannot get replset config: not authorized for query on local.system.replset

[David Britton]

Long log, paste here: http://paste.ubuntu.com/7832774/

2014-07-21 22:05:19 DEBUG juju.worker.peergrouper initiate.go:41 Initiating mongo replicaset; dialInfo &mgo.DialInfo{Addrs:[]string{"127.0.0.1:37017"}, Direct:false, Timeout:300000000000, FailFast:false, Database:"", Source:"", Service:"", Mechanism:"", Username:"", Password:"", DialServer:(func(*mgo.ServerAddr) (net.Conn, error))(nil), Dial:(func(net.Addr) (net.Conn, error))(0x5bbd70)}; memberHostport "10.1.3.229:37017"; user ""; password ""
2014-07-21 22:05:19 DEBUG juju.mongo open.go:87 connection failed, will retry: dial tcp 127.0.0.1:37017: connection refused
2014-07-21 22:05:19 DEBUG juju.mongo open.go:87 connection failed, will retry: dial tcp 127.0.0.1:37017: connection refused
2014-07-21 22:05:19 INFO juju.mongo open.go:95 dialled mongo successfully
2014-07-21 22:05:19 INFO juju.worker.peergrouper initiate.go:94 finished MaybeInitiateMongoServer
2014-07-21 22:05:19 ERROR juju.cmd supercommand.go:323 cannot get replica set configuration: cannot get replset config: not authorized for query on local.system.replset
ERROR bootstrap failed: subprocess encountered error code 1
Stopping instance...
Bootstrap failed, destroying environment
ERROR subprocess encountered error code 1

Will add the rest to a log file.

[David Britton (davidpbritton)]

MAAS environment was dirty due to another bug that we are tracking down and will file. If anyone else gets this on maas, make sure your machines were in fact *off* before the bootstrap was allocated one of them.

[Ryan Harper (raharper)]

What's the other bug?

[David Britton (davidpbritton)]

Ryan -- IIRC, either this one, or one like it (we added tags to a couple of these types of bugs):

https://bugs.launchpad.net/maas/+bug/1012954

Basically, our system was left on, even though the power script had run from the previous attempt to shut them down. The juju bootstrap came around and grabbed a machine that was already booted as a juju bootstrap node, leading to this error (mongo was up, but the credentials were all wrong).

As for why the power down command didn't work. I don't remember specifically, but if I didn't file a bug it was probably a network glitch.

[Jason Hobbs (jason-hobbs)]

This is showing up in OIL with juju-core 1.22-beta4.

[Jason Hobbs (jason-hobbs)]

Some more logs for it:

https://pastebin.canonical.com/126730/

[Wayne Witzel III (wwitzel3)]

Working with the OIL installation scripts to determine the point during the deployment process when the error starts to occur. I was unable reproduce on juju-1.22 trunk using just a simple HA deployment on MAAS.

[Wayne Witzel III (wwitzel3)]

Still not able to successfully reproduce this error using my local MAAS. Can you provide the exact commands that were issued for the bootstrap and adding units?

[Curtis Hovey (sinzui)]

My reading of the log from comment 6 (https://pastebin.canonical.com/126730/) is that cloud-init updates the the machine machines packages from the precise archives, then tries to start the trusty jujud (tool/agent). This is a mismatch between the series of provisioned machine and the series of the agent juju instructed cloud init to install. I suspect that this is mass provisioning issue.

[Curtis Hovey (sinzui)]

We are removing this bug from the 1.22-beta5 milestone to unblock its release.

[Tyson Malchow (tyson-malchow)]

i get this from taking the following steps:

- launch vagrant machine using trusty-server-cloudimg-amd64-juju-vagrant-disk1.box
- destroy-environment
- try to bootstrap again

literally cannot get juju to work again on that machine after this point. uninstalling everything via apt-get purge to no avail. have to start a new machine entirely. :'(

[Alexis Bruemmer (alexis-bruemmer)]

There needs to either be valid reproduction steps or access to an environment that can reproduce this issue.

[Launchpad Janitor (janitor)]

[Expired for juju-core 1.24 because there has been no activity for 60 days.]

[James Tunnicliffe (dooferlad)]

Not sure if we want to count this as the same issue, but if you set up MAAS 1.9 (maybe other versions) to give out addresses over DHCP and try to bootstrap a node with 1.25.x (not tested with 2.0) the bootstrap will fail:

2016-05-25 11:21:42 WARNING juju.replicaset replicaset.go:98 Initiate: fetching replication status failed: cannot get replica set status: can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2016-05-25 11:21:42 WARNING juju.replicaset replicaset.go:98 Initiate: fetching replication status failed: cannot get replica set status: can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2016-05-25 11:21:43 WARNING juju.replicaset replicaset.go:98 Initiate: fetching replication status failed: cannot get replica set status: can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2016-05-25 11:21:43 WARNING juju.replicaset replicaset.go:98 Initiate: fetching replication status failed: cannot get replica set status: can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2016-05-25 11:21:44 INFO juju.worker.peergrouper initiate.go:77 finished MaybeInitiateMongoServer
2016-05-25 11:21:44 ERROR juju.cmd supercommand.go:429 cannot initiate replica set: cannot get replica set status: can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2016-05-25 11:21:55 ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: subprocess encountered error code 1

The same set up with addressing changed to auto assign works fine.

[Anastasia (anastasia-macmood)]

We have not seen this in 2.0. We need a reproducible scenario to tackle it.

[Nate Finch (natefinch)]

I'm getting this when I bring up a GCE machine to use as a bootstrap machine for manual provider:

WARNING juju.replicaset replicaset.go:98 Initiate: fetching replication status failed: cannot get replica set status: can't get local.system.replset config from self or any seed (EMPTYCONFIG)

notably, when I'm ssh'd into that machine, I can telnet to localhost 37017 (the port mongod is listening on), but I can't telnet into <externalIP> 37017, which is what is stored in the replicaset config (you can check the replicaset config easily by doing a grep replSetInitiate /var/log/syslog)

Mine says:

command admin.$cmd command: { replSetInitiate: { _id: "juju", version: 1, members: [ { _id: 1, host: "104.155.13.238:37017", tags: { juju-machine-id: "0" } } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:33 reslen:122 6001ms

The way replicasets work is that juju connects to mongo on localhost, and mongo will tell it to connect to one of the machines listed in the replicaset config (in this case, just the one with the local machine's external IP address), and then juju drops the original connection and tries to connect to that machine... which in this case just fails... the error it fails with is that it can't get the replicaset config from any members of the replicaset (because it can't connect to them). This even though it read the replicaset in order to get the list of addresses to try :/

[Nate Finch (natefinch)]

So, it's likely to be a firewall issue with GCE - there's a firewall external to the machine that is configurable via the web GCE console. I haven't yet twiddled with it enough to get it to work, though.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]