juju 1.20 consistently fails to bootstrap a MAAS environment (regression from 1.18)

[James Troup]

I'm using juju 1.20.0-0ubuntu1~14.04.1~juju1 from the Juju stable PPA
on Ubuntu 14.04 with MAAS 1.5.2+bzr2282-0ubuntu0.2 and I can no longer
bootstrap an environment, it fails reproducibly (3 times and across different
hardware) with the following message:

2014-07-08 17:16:51 INFO juju.worker.peergrouper initiate.go:94 finished MaybeInitiateMongoServer
2014-07-08 17:16:51 ERROR juju.cmd supercommand.go:323 cannot initiate replica set: Closed explicitly

Full log:

  https://pastebin.canonical.com/113157/

I was previously able to successfully bootstrap an environment with
this MAAS with juju 1.18.1-0ubuntu1 from trusty directly.

[James Troup (elmo)]

So, just to confirm, I ran this a fourth time with 1.20 and it failed again. I downgraded to 1.18 and it bootstrapped just fine.

[Michael Foord (mfoord)]

1.18 does not pass the "--replSet" argument to mongo when starting, this was added (along with the HA work) between 1.18 and 1.20. In agent/mongo/mongo.go we see:

                // TODO(Nate): uncomment when we commit HA stuff
                // +
                // " --replSet juju",

This is relevant because we saw a very similar problem when we attempted to enable replica sets for the local provider. (Leading us to believe that it is the use of replicasets in 1.20 causing this problem.)

"Closed Explicitly" is a message from the mgo driver we use to talk to mongo. We're looking at our use of sessions to fix this issue here - creating a new session / refreshing the sockets instead of using a session that may have a closed connection.

[Andrew Wilkins (axwalk)]

Potential fix here: https://github.com/juju/juju/pull/271

[James Troup (elmo)]

This works for me - https://pastebin.canonical.com/113168/

[Greg Lutostanski (lutostag)]

Causing all bootstrapping in OIL to fail. Will see if we can get an updated package sooner than 1.20.1 just for OIL -- will be poking via irc.

[Andrew Wilkins (axwalk)]

@lutostag: the test binary that elmo used can be found at https://chinstrap.canonical.com/~axw/