controller leaking mongodb connections

[Martin Hilton]

Version: jujud [2.0-beta6.1 gc go1.6]

We have a controller being used by a number of users to create models. After some time new models could no longer be created. The error returned was "could not create state for new model: cannot connect to mongodb: no reachable servers" Investigation on the controller machine shows a large number of open connections from jujud to mongodb. (see https://pastebin.canonical.com/156345/)

[Roger Peppe (rogpeppe)]

Further analysis of the log shows that there are 449 connections to MongoDB,
64 connections to the API server of which 27 are local to the controller node, giving an average of 7 MongoDB sockets for each API connection.

[Menno Finlay-Smits (menno.smits)]

Please provide logs for the controller machines and the MongoDB logs in /var/log/syslog on the controller machines.

[Roger Peppe (rogpeppe)]

I suspect that the underlying reason here might be that the code
calls mgo.Session.Copy every time it does anything, which means
that every concurrent operation will use its own session.

That is, this issue may not be an actual leak at all - just
an excessive use of mongo sessions.

There's an associated issue with using Copy all the time
like that too, which is that consistency between sequential
operations is not guaranteed, as the copied session may
talk to a different server.

[William Reade (fwereade)]

I have a potential mitigation proposed at http://reviews.vapour.ws/r/4981/ -- but I'd be surprised if it made any serious difference.

I'm looking into what it'd take to associate sessions with api activity and/or the various long-running internal tasks; I have nothing very concrete yet; but we've actually built out quite a lot of the necessary infrastructure already, and the most critical step -- extracting infrastructure tasks from *State instances -- is evidently *large*: but relatively comprehensible, and I think not high-risk.

[Tim Penhey (thumper)]

We have landed something that we believe will have either fixed or improved the situation.

Can we please retest and let us know?

[Cheryl Jennings (cherylj)]

William's patch landed in beta9. Can you retest with the latest beta?

[Anastasia (anastasia-macmood)]

@Roger, @Martin,

Could you please re-test with newer beta - currently, 14?
We believe this to have been fixed couple of betas back :D

[Junien F (axino)]

Hi,

I believe we have hit this bug. We have a shared controller running juju 2.1.1, running around 40 models. The number of connections to mongodb was above 4000 for a while, it's now back to nearly 3000 :

juju:PRIMARY> db.serverStatus()["connections"]
{ "current" : 2929, "available" : 49071, "totalCreated" : NumberLong(6974) }

There are only a dozen or so queries in progress at any time (as reported by db.currentOp() ), so I believe these connections are unnecessarily wasting resources.

[Tim Penhey (thumper)]

Found and fixed the underlying last (that we know of) leak
  https://github.com/juju/juju/pull/7134