Memory leak in jujud agents

[Haw Loeung]

Hi,

As seen with various units, both jujud machine and unit agents seems to continue to grow. That's fine if it's just one machine and one unit agent but on various services, we usually have quite a bundle (application/service, nrpe, livepatch, landscape-client, autocert, etc.). Can we please take a look into this to see what can be done to keep memory usage constant?

So far, it's all 2.6.10 agents. We're in the process of rolling out juju 2.7.4 across the board but that will still be a few weeks.

[Haw Loeung (hloeung)]

Before restarting:

| PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
| 5500 root 20 0 621004 208440 12692 S 0.0 10.2 186:08.96 /var/lib/juju/tools/unit-lpa-nrpe-3/jujud unit --data-d+
| 5499 root 20 0 621196 203388 7948 S 0.0 9.9 191:41.05 /var/lib/juju/tools/unit-lpa-logstash-forwarder-2/jujud+
| 5494 root 20 0 621004 200832 7516 S 0.0 9.8 188:36.01 /var/lib/juju/tools/unit-landscape-client-7/jujud unit +
| 5405 root 20 0 629200 200156 4156 S 0.0 9.8 193:37.70 /var/lib/juju/tools/unit-canonical-livepatch-7/jujud un+
| 5496 root 20 0 621068 198928 3376 S 0.0 9.7 203:56.50 /var/lib/juju/tools/unit-autocert-lpa-fe-1/jujud unit -+
| 5495 root 20 0 621004 197996 2260 S 0.0 9.7 193:13.57 /var/lib/juju/tools/unit-lpa-fe-0/jujud unit --data-dir+
| 5364 root 20 0 844604 15468 2136 S 0.0 0.8 52:05.13 /var/lib/juju/tools/machine-3/jujud machine --data-dir +
(machine agent was just restarted)

After:

| PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
| 32467 root 20 0 705080 72132 60516 S 0.0 3.5 0:00.44 /var/lib/juju/tools/machine-3/jujud machine --data-dir +
| 32526 root 20 0 541224 71344 60424 S 0.0 3.5 0:00.33 /var/lib/juju/tools/unit-autocert-lpa-fe-1/jujud unit -+
| 32514 root 20 0 475688 71288 60004 S 0.0 3.5 0:00.33 /var/lib/juju/tools/unit-lpa-fe-0/jujud unit --data-dir+
| 32505 root 20 0 541224 71216 60508 S 0.0 3.5 0:00.32 /var/lib/juju/tools/unit-canonical-livepatch-7/jujud un+
| 32508 root 20 0 475688 71080 60600 S 0.0 3.5 0:00.28 /var/lib/juju/tools/unit-landscape-client-7/jujud unit +
| 32510 root 20 0 483884 70928 60436 S 0.0 3.5 0:00.30 /var/lib/juju/tools/unit-lpa-nrpe-3/jujud unit --data-d+
| 32513 root 20 0 483884 70048 59804 S 0.0 3.4 0:00.26 /var/lib/juju/tools/unit-lpa-logstash-forwarder-2/jujud+

[Ian Booth (wallyworld)]

We've fixed a few leaks in 2.7 compared to 2.6, eg in 2.6 there's a goroutine leak that can be triggered under the right circumstances.

Just to set expectations, we don't plan on anymore 2.6 releases unfortunately.

[John A Meinel (jameinel)]

That said, the only one we've monitored closely and taken big iterations on
is the controller agent. I believe on prodstack we actively take goroutine
and heap dumps so we can look for memory leaks after the fact. We could
potentially do something like that for a couple of the unit agents,
(something like a cron that calls juju_heap_profile and saves the result
somewhere we can look at it afterward.

On Mon, Mar 23, 2020 at 9:15 AM Ian Booth <email address hidden> wrote:

> We've fixed a few leaks in 2.7 compared to 2.6, eg in 2.6 there's a
> goroutine leak that can be triggered under the right circumstances.
>
> Just to set expectations, we don't plan on anymore 2.6 releases
> unfortunately.
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1868362
>
> Title:
> Memory leak in jujud agents
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1868362/+subscriptions
>

[Tim Penhey (thumper)]

I was just typing in exactly what John had just said.

+1 to that.