juju should pool connections to mongo

[Junien F]

Hi,

On our juju 2.3.8 HA controllers (3 of them) with ~4k units and ~800 machines, there are around 7.3k connections to the mongodb primary. This seems excessive. I seem to remember that for each agent, a controller will create one or more mongodb connection. This doesn't scale very well, and leads to reconnect storms that can overload the mongodb server (for example after a full restart of the controllers), which certainly can't (and doesn't) process 7k queries simultaneously.

Instead, I believe the controllers should maintain a reasonable number of connections to the database, at all times, and use them whenever they need to make a database query to fulfill an agent request.

I see some mention of "PoolLimit" in the code, but can't figure out that the default value is, and if it's even used by default or not.

Thanks

[John A Meinel (jameinel)]

We do pool connections. If the pool is in active use, we allow it to grow.
Likely lots of agents were doing concurrent activity and we don't put hard
caps on the pool.

Likely we *should* be putting a cap on the pool, though it means we'll also
have to handle when a thread is unable to make their request because they
can't get a slot in the pool.

It may just be a limiter/timeout sort of system (wait up to 50ms for a free
slot to be available before creating a new connection).

On Fri, Aug 10, 2018 at 3:17 AM, Tim Penhey <email address hidden>
wrote:

> ** Tags added: mongodb scalability
>
> ** Changed in: juju
> Status: New => Triaged
>
> ** Changed in: juju
> Importance: Undecided => Medium
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1786258
>
> Title:
> juju should pool connections to mongo
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1786258/+subscriptions
>

[John A Meinel (jameinel)]

Default pool limit, is in the gopkg.in/mgo.v2 code base is 4096 per
controller. You can set a value in agent.conf:
if limitStr := agentConfig.Value("MONGO_SOCKET_POOL_LIMIT"); limitStr != ""
{

Likely it will cause queries to just fail. w/ 3 HA controllers, max cons
would be 3*4096=12,288 connections. So you aren't currently hitting that.

On Mon, Aug 13, 2018 at 8:46 AM, John Meinel <email address hidden> wrote:

> We do pool connections. If the pool is in active use, we allow it to grow.
> Likely lots of agents were doing concurrent activity and we don't put hard
> caps on the pool.
>
> Likely we *should* be putting a cap on the pool, though it means we'll
> also have to handle when a thread is unable to make their request because
> they can't get a slot in the pool.
>
> It may just be a limiter/timeout sort of system (wait up to 50ms for a
> free slot to be available before creating a new connection).
>
>
> On Fri, Aug 10, 2018 at 3:17 AM, Tim Penhey <email address hidden>
> wrote:
>
>> ** Tags added: mongodb scalability
>>
>> ** Changed in: juju
>> Status: New => Triaged
>>
>> ** Changed in: juju
>> Importance: Undecided => Medium
>>
>> --
>> You received this bug notification because you are subscribed to juju.
>> Matching subscriptions: juju bugs
>> https://bugs.launchpad.net/bugs/1786258
>>
>> Title:
>> juju should pool connections to mongo
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/juju/+bug/1786258/+subscriptions
>>
>
>

[John A Meinel (jameinel)]

I should also mention that I don't think the Pool is gardened. So it grows
to a max size, and then stays there. I would guess that without a policy of
"allow a request to not be satisfied for X ms", you'll run into workers
restarting because hitting the limits of the pool will just cause the query
to be considered failed and the worker thread (and possibly agent) will
bounce.

That might introduce more load than it saves, or it might leave you in a
healthier state because it forces your peak load to go down.

On Mon, Aug 13, 2018 at 8:51 AM, John Meinel <email address hidden> wrote:

> Default pool limit, is in the gopkg.in/mgo.v2 code base is 4096 per
> controller. You can set a value in agent.conf:
> if limitStr := agentConfig.Value("MONGO_SOCKET_POOL_LIMIT"); limitStr !=
> "" {
>
>
> Likely it will cause queries to just fail. w/ 3 HA controllers, max cons
> would be 3*4096=12,288 connections. So you aren't currently hitting that.
>
>
> On Mon, Aug 13, 2018 at 8:46 AM, John Meinel <email address hidden>
> wrote:
>
>> We do pool connections. If the pool is in active use, we allow it to
>> grow. Likely lots of agents were doing concurrent activity and we don't put
>> hard caps on the pool.
>>
>> Likely we *should* be putting a cap on the pool, though it means we'll
>> also have to handle when a thread is unable to make their request because
>> they can't get a slot in the pool.
>>
>> It may just be a limiter/timeout sort of system (wait up to 50ms for a
>> free slot to be available before creating a new connection).
>>
>>
>> On Fri, Aug 10, 2018 at 3:17 AM, Tim Penhey <email address hidden>
>> wrote:
>>
>>> ** Tags added: mongodb scalability
>>>
>>> ** Changed in: juju
>>> Status: New => Triaged
>>>
>>> ** Changed in: juju
>>> Importance: Undecided => Medium
>>>
>>> --
>>> You received this bug notification because you are subscribed to juju.
>>> Matching subscriptions: juju bugs
>>> https://bugs.launchpad.net/bugs/1786258
>>>
>>> Title:
>>> juju should pool connections to mongo
>>>
>>> To manage notifications about this bug go to:
>>> https://bugs.launchpad.net/juju/+bug/1786258/+subscriptions
>>>
>>
>>
>

[Junien F (axino)]

So apparently I read in the code that you can change the max pool size in the agent config, correct ?

Are you saying that the agents will start failing if the controllers wants to run more parallel queries than the pool size ? (in other words : there's no queue mechanisms for requests "overflowing" the pool size ?)

[John A Meinel (jameinel)]

I believe the config is only read at startup. But otherwise you would be
able to set a max pool size. However, if you hit it, it will just fail the
query, triggering an error, which is probably only handled by restarting
the worker.

On Mon, Aug 13, 2018 at 11:10 AM, Junien Fridrick <
<email address hidden>> wrote:

> So apparently I read in the code that you can change the max pool size
> in the agent config, correct ?
>
> Are you saying that the agents will start failing if the controllers
> wants to run more parallel queries than the pool size ? (in other words
> : there's no queue mechanisms for requests "overflowing" the pool size
> ?)
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1786258
>
> Title:
> juju should pool connections to mongo
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1786258/+subscriptions
>

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.