juju should pool connections to mongo
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Triaged
|
Low
|
Unassigned |
Bug Description
Hi,
On our juju 2.3.8 HA controllers (3 of them) with ~4k units and ~800 machines, there are around 7.3k connections to the mongodb primary. This seems excessive. I seem to remember that for each agent, a controller will create one or more mongodb connection. This doesn't scale very well, and leads to reconnect storms that can overload the mongodb server (for example after a full restart of the controllers), which certainly can't (and doesn't) process 7k queries simultaneously.
Instead, I believe the controllers should maintain a reasonable number of connections to the database, at all times, and use them whenever they need to make a database query to fulfill an agent request.
I see some mention of "PoolLimit" in the code, but can't figure out that the default value is, and if it's even used by default or not.
Thanks
tags: | added: mongodb scalability |
Changed in juju: | |
status: | New → Triaged |
importance: | Undecided → Medium |
We do pool connections. If the pool is in active use, we allow it to grow.
Likely lots of agents were doing concurrent activity and we don't put hard
caps on the pool.
Likely we *should* be putting a cap on the pool, though it means we'll also
have to handle when a thread is unable to make their request because they
can't get a slot in the pool.
It may just be a limiter/timeout sort of system (wait up to 50ms for a free
slot to be available before creating a new connection).
On Fri, Aug 10, 2018 at 3:17 AM, Tim Penhey <email address hidden>
wrote:
> ** Tags added: mongodb scalability /bugs.launchpad .net/bugs/ 1786258 /bugs.launchpad .net/juju/ +bug/1786258/ +subscriptions
>
> ** Changed in: juju
> Status: New => Triaged
>
> ** Changed in: juju
> Importance: Undecided => Medium
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https:/
>
> Title:
> juju should pool connections to mongo
>
> To manage notifications about this bug go to:
> https:/
>