support mongo 3.6

[John A Meinel]

There are a few failures that I'm seeing trying to run the test suite with Mongo 3.6

1) --nohttpinterface

diff --git a/mgo.go b/mgo.go
index c128747..cde1be0 100644
--- a/mgo.go
+++ b/mgo.go
@@ -230,7 +230,6 @@ func (inst *MgoInstance) run() error {
                "--nssize", "1",
                "--noprealloc",
                "--smallfiles",
- "--nohttpinterface",
                "--oplogSize", "10",
                "--ipv6",
                "--setParameter", "enableTestCommands=1",

Looking at "mongod --help" I can see that mongo 2.6, 3.2 and 3.4 (and presumably 2.4 and 3.0) support passing '--httpinterface'. Which doesn't appear to be present in 3.6. Presumably because they have an "--httpinterface" we are passing "--nohttpinterface" to make sure it isn't enabled, but it doesn't look like it would be active unless we actually were passing "--httpinterface", so I think we can just always drop it.

2) 'cannot drop admin':
----------------------------------------------------------------------
FAIL: <autogenerated>:1: ActionSuite.TearDownSuite

dev/go/src/github.com/juju/testing/mgo.go:544:
    c.Assert(err, jc.ErrorIsNil)
... value *errors.Err = &errors.Err{message:"cannot drop MongoDB database admin", cause:(*mgo.QueryError)(0xc042d65c00), previous:(*mgo.QueryError)(0xc042d65c00), file:"github.com/juju/testing/mgo.go", line:753} ("cannot drop MongoDB database admin: Dropping the 'admin' database is prohibited.")
... error stack:
        Dropping the 'admin' database is prohibited.
        github.com/juju/testing/mgo.go:753: cannot drop MongoDB database admin

PASS: action_test.go:1090: ActionPruningSuite.TestDoNotPruneIncompleteActions 0.247s
PASS: action_test.go:1059: ActionPruningSuite.TestPruneActionByAge 0.298s
PASS: action_test.go:987: ActionPruningSuite.TestPruneActionsBySize 0.330s
PASS: action_test.go:1018: ActionPruningSuite.TestPruneActionsBySizeOldestFirst 0.254s

We generally avoid dropping 'system.' tables. But this seems we are getting confused and trying to destroy the entire database.
Regardless, we need to avoid doing that.

3) Eventually we'll need to start passing --bind_ip_all on the command line for 3.6, but we don't need to do that for the test suite.

[John A Meinel (jameinel)]

https://github.com/juju/testing/pull/136

[John A Meinel (jameinel)]

https://github.com/juju/juju/pull/8553

[John A Meinel (jameinel)]

Note that those associated patches get the unit tests to run with Mongo 3.6 (if you have it locally), but does not provide everything needed to support 3.6 for "juju bootstrap". (We know we at least need to be passing --bind_ip_all to mongod startup when in HA mode.)

We are currently waiting for a 3.6 to be available in some sort of PPA or the Bionic archive to finish testing bootstrap.

See also bugs that were about upgrading to mongo 3.4 like:
https://bugs.launchpad.net/juju/+bug/1672967
https://bugs.launchpad.net/bugs/1756432

And
https://github.com/jameinel/juju/tree/2.3-bionic-mongodb-server-core

which is a spike to have Juju 2.3 support installing 'mongodb-server-core' on Bionic. Which might evolve to be Mongo 3.6 at which point we'll need to change the command line parameters we are passing.

There may be other issues supporting 3.6 but at least the test suite passes.

[John A Meinel (jameinel)]

I did a bootstrap on bionic with the updated hacked branch mentioned in the
previous comment.
As expected 'juju enable-ha' failed with the mongo peers not being
healthy. (since all replicas were only binding to localhost, none of them
could talk to each other.)

While they were running, I used "lxc exec ... bash" to get into each
machine, and then edited:
/var/lib/juju/init/juju-db/juju-db.service
to set it to:
ExecStart=/usr/bin/mongod --dbpath '/var/lib/juju/db' --sslPEMKeyFile
'/var/lib/juju/server.pem' --sslPEMKeyPassword=ignored --port 37017
--syslog --journal --replSet juju --quiet --oplogSize 512 --ipv6 --auth
--keyFile '/var/lib/juju/shared-secret' --sslMode requireSSL
--storageEngine wiredTiger --wiredTigerCacheSizeGB 1 --bind_ip_all

(I just appended --bind_ip_all)

I then ran
systemctl daemon-reload
systemctl restart juju-db

After doing that on all 3 units, the replicaset became happy, and juju
status
Machine State DNS Inst id Series AZ Message
0 started 10.16.17.198 juju-160f73-0 bionic Running
1 started 10.16.17.138 juju-160f73-1 bionic Running
2 started 10.16.17.238 juju-160f73-2 bionic Running

juju show-controller
...
  controller-machines:
    "0":
      instance-id: juju-160f73-0
      ha-status: ha-enabled
    "1":
      instance-id: juju-160f73-1
      ha-status: ha-enabled
    "2":
      instance-id: juju-160f73-2
      ha-status: ha-enabled

So everything seems to be happy.
John
=:->

On Mon, Apr 2, 2018 at 5:32 PM, John A Meinel <email address hidden>
wrote:

> Note that those associated patches get the unit tests to run with Mongo
> 3.6 (if you have it locally), but does not provide everything needed to
> support 3.6 for "juju bootstrap". (We know we at least need to be
> passing --bind_ip_all to mongod startup when in HA mode.)
>
> We are currently waiting for a 3.6 to be available in some sort of PPA
> or the Bionic archive to finish testing bootstrap.
>
> See also bugs that were about upgrading to mongo 3.4 like:
> https://bugs.launchpad.net/juju/+bug/1672967
> https://bugs.launchpad.net/bugs/1756432
>
> And
> https://github.com/jameinel/juju/tree/2.3-bionic-mongodb-server-core
>
> which is a spike to have Juju 2.3 support installing 'mongodb-server-
> core' on Bionic. Which might evolve to be Mongo 3.6 at which point we'll
> need to change the command line parameters we are passing.
>
> There may be other issues supporting 3.6 but at least the test suite
> passes.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1760390
>
> Title:
> support mongo 3.6
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1760390/+subscriptions
>

[Nicholas Skaggs (nskaggs)]

I confirm John's findings. In addition, I tested the proposed deb against our unit tests:

spawn lxd bionic container
install mongod from ppa
mkdir -p /usr/lib/juju/mongo3.2/bin/
ln -s /usr/bin/mongod /usr/lib/juju/mongo3.2/bin/mongod
# without this ^^, the tests look for and fail to find mongo. We don't look in /usr/bin currently
go get juju
make add-patches
make check

There were some spurious failures but nothing jumped out at me as mongo related or specific.

[John A Meinel (jameinel)]

https://github.com/juju/juju/pull/8589

[John A Meinel (jameinel)]

https://github.com/juju/juju/pull/8606 merge into 2.4

[Roger Peppe (rogpeppe)]

FWIW when I run the juju tests with mongod 3.6, I see a problem when running the featuretest tests - MongoDB appears to crash and then the tests do not complete.

I've attached a log of the tests running ("go test -check.vv -check.f syslogSuite" in the featuretests directory).

Search for "aborting after invariant" in the log output to see the mongod crash.
This also happens with mongod 3.7.

I was testing with juju commit adc21b400d457e28bae5e93dd104e615d4e0a4a7 and Go 1.9.