jujud uses too much memory

[Jay R. Wren]

Using 1.5GB RSS is too much memory.

ps auxww output:
root 32754 1.4 18.9 3169796 1528172 ? Sl Aug25 440:42 /home/jrwren/.juju/local/tools/machine-0/jujud machine --data-dir /home/jrwren/.juju/local --machine-id 0 --debug

Version seems to be 1.24.5 judging from symbols in /proc/32754/exe

I've since upgraded juju on this system.

I will leave this process running for a couple of days. Please let me know how I can provide you with further information for debugging.

[Alexis Bruemmer (alexis-bruemmer)]

for clarity: This issue was found on 1.24.5 correct?

What version did you upgrade to?

[Jay R. Wren (evarlast)]

Upgraded to 1.25-alpha1-vivid-amd64. I do not know if the upgrade is
related to memory usage or not.

On Tue, Sep 15, 2015 at 11:51 AM, Alexis Bruemmer
<email address hidden> wrote:
> for clarity: This issue was found on 1.24.5 correct?
>
> What version did you upgrade to?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1496016
>
> Title:
> jujud uses too much memory
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1496016/+subscriptions

[Jay R. Wren (evarlast)]

Yes, as far as I know, this issue was found on 1.24.5.

On Tue, Sep 15, 2015 at 12:11 PM, Jay Wren <email address hidden> wrote:
> Upgraded to 1.25-alpha1-vivid-amd64. I do not know if the upgrade is
> related to memory usage or not.
>
> On Tue, Sep 15, 2015 at 11:51 AM, Alexis Bruemmer
> <email address hidden> wrote:
>> for clarity: This issue was found on 1.24.5 correct?
>>
>> What version did you upgrade to?
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1496016
>>
>> Title:
>> jujud uses too much memory
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/juju-core/+bug/1496016/+subscriptions

[Jay R. Wren (evarlast)]

It may be a leak? I've not used this juju at all in the past 24 hours. In fact, I've only ran `juju status` once since reporting this bug.

The RSS has grown to 2.002GB.

At this growth rate, I may have to kill this process sooner than I originally thought. Are there know triage commands which I could run to capture the state of this process before killing it?

[Jay R. Wren (evarlast)]

Something happened which triggered a GC. RSS is now down to 639816.

[Jay R. Wren (evarlast)]

I had to move on to get work done. I destroyed the environment. Please close this ticket.

[Junien F (axino)]

Hi,

This is happening to us as well. juju version 1.24.5, machine-0 jujud regularly eats up all the memory. machine-0.log even has :

2015-10-13 06:17:56 DEBUG juju.worker runner.go:203 "diskmanager" done: cannot list block devices: lsblk failed: fork/exec /bin/lsblk: cannot allocate memory

When it happened today, both jujud and mongo were taking 100% CPU as well, and there was ~20 load on the node.

jujud also logged ~300MB in about 45 min (machine-0.log).

Once restarted, jujud resumes processing operations, so a few config-changed hooks fired.

Additionally, all the services in this environment also get regularly in the "lost" state. Restarting machine-0 jujud brings them back to their normal "idle" status.

As evarlast asked, is there any known triage command we should run when this happen ? Should I run some commands to extract useful information from the logs ?

Thanks

[Matt Dirba (5qxm)]

Does jujud keep a separate copy of each charm in memory? When my charm directory was 400mB I saw high memory usage by jujud.

[Cheryl Jennings (cherylj)]

Bug 1382556 was opened for excessive memory consumption and is actively being investigated. Dup'ing this one to bug 1382556.