Extreme RAM and SWAP usage

Bug #1685885 reported by Thomas Ward on 2017-04-24
68
This bug affects 14 people
Affects Status Importance Assigned to Milestone
Landscape Client
High
Simon Poirier
landscape-client (Ubuntu)
High
Simon Poirier
Xenial
High
Simon Poirier
Bionic
High
Simon Poirier
Cosmic
High
Simon Poirier

Bug Description

[Impact]

 * Memory usage of landscape processes abnormally grows, over time,
   even for processes which are generally idle and have no reason to
   grow.

 * Most of that memory leak is due to uninitialized python-twisted
   logging facilities buffering logs and their context. This fix
   Initializes logging with a null handler to have those unused log
   entries discarded.

 * Upstream twisted has mitigated part of the issue in recent releases
   by reducing the default buffer size. Because of that, the impact
   of this bug is greater on xenial.

[Test Case]

 * register a landscape client and log the memory growth over a day

 * the effect is most noticeable on /usr/bin/landscape-client and less on
   the other processes

 * the memory difference on recent releases (bionic and up) is
   expected to be smaller (~10mb on bionic instead of 200mb on xenial)

[Regression Potential]

 * The fix initializes twisted logging, which is unused by landscape
   so it is unlikely to have an impact.

 * The fix comes from recommendations on the upstream bug:
   https://twistedmatrix.com/trac/ticket/8164

[Original description]

This has been noticed on several systems of mine, all 16.04.

Landscape Client takes up, over time, an *extreme* amount of system resources.

On a 1024MB RAM system, within 3 days to a month, Landscape Client and it's python scripts (possibly cached data?) are consuming massive quantities of memory - in the realm of 512MB RAM and over 400MB of the Swap on the system.

This is repeatably shown in other systems of mine as well. Is there a memory leak or something in landscape-client that would contribute to this issue?

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: landscape-client 16.03-0ubuntu2.16.04.1
ProcVersionSignature: Ubuntu 4.4.0-72.93-generic 4.4.49
Uname: Linux 4.4.0-72-generic x86_64
NonfreeKernelModules: talpa_vfshook talpa_pedconnector talpa_vcdevice talpa_core talpa_linux talpa_syscallhook talpa_pedevice
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
Date: Mon Apr 24 14:42:37 2017
InstallationDate: Installed on 2017-04-02 (21 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: landscape-client
UpgradeStatus: No upgrade log present (probably fresh install)
---
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
DistroRelease: Ubuntu 16.04
InstallationDate: Installed on 2017-04-02 (21 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
Package: landscape-client 16.03-0ubuntu2.16.04.1
PackageArchitecture: amd64
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 4.4.0-72.93-generic 4.4.49
Tags: third-party-packages xenial
Uname: Linux 4.4.0-72-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True

Related branches

apport information

tags: added: apport-collected
description: updated

apport information

Thomas Ward (teward) wrote :

Note that the kernel modules listed in the bug may be out of date - I don't have those modules listed, but i hadn't rebooted since they were installed, due to the antivirus in place on this server where I reported from (it's a mail server, so an antivirus engine with amavisd was necessary; note that the usage values for this bug report - for landscape client consuming lots of RAM - had the kernel modules 'off' and disabled with its standard software that correspond with the antivirus engine.)

Thomas Ward (teward) wrote :

I discussed this briefly in #ubuntu-hardened and #ubuntu-server and confirmed that the code hasn't been 'reviewed' lately.

I'm setting the importance here to "high" because I just got nipped with this again, and my primary mail server went down because of insanely high swap and RAM usage from Landscape Client. I'm going to test this with a new server to rule out headaches, but if landscape client is going to consume ungodly amounts of RAM, then it should be shelved until the memory leaks are solved.

Changed in landscape-client (Ubuntu):
importance: Undecided → High
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in landscape-client (Ubuntu):
status: New → Confirmed

Also affecting me -- landscape-client using 25% of RAM on a 1GB server.

Thomas Ward (teward) on 2017-07-14
Changed in landscape-client:
status: New → Confirmed
flindeberg (flindeberg) wrote :

This is actually so severe that I started to regularly restart my small instances *every week*. The instance below is with an uptime of nine (9) days on a heavily used file server, and as you can see the first none landscape related process is the ssh-session I'm in! Sorted on memory %.

  PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
 1410 root 20 0 302M 229M 2252 S 0.0 25.2 35:39.09 /usr/bin/python /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/landscape-client.p
 1426 root 20 0 205M 117M 3768 S 0.0 12.9 9:08.72 /usr/bin/python /usr/bin/landscape-manager --ignore-sigint --quiet
 1470 landscape 20 0 552M 67320 5136 S 0.0 7.2 2:19.10 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1474 landscape 20 0 552M 67320 5136 S 0.0 7.2 2:18.09 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1423 landscape 20 0 552M 67320 5136 S 0.0 7.2 15:38.76 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1472 landscape 20 0 355M 28884 3780 S 0.0 3.1 0:00.01 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
 1424 landscape 20 0 355M 28884 3780 S 0.0 3.1 16:14.51 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
65097 root 20 0 119M 7772 6516 S 0.0 0.8 0:00.10 sshd: flindeberg [priv]

Same instance after reboot (sorted on memory %):

  PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
  976 mongodb 20 0 959M 62340 31504 S 0.0 6.7 0:00.00 /usr/bin/mongod --unixSocketPrefix=/run/mongodb --config /etc/mongodb.conf
  -- some other db and file server processes --
 1269 landscape 20 0 307M 54100 12616 S 0.0 5.8 0:00.49 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1231 landscape 20 0 307M 54100 12616 S 0.0 5.8 0:01.49 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1271 landscape 20 0 253M 38316 14680 S 0.0 4.1 0:00.01 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
 1232 landscape 20 0 253M 38316 14680 S 0.0 4.1 0:01.16 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
 1233 root 20 0 104M 34472 11840 S 0.0 3.7 0:00.86 /usr/bin/python /usr/bin/landscape-manager --ignore-sigint --quiet
 1224 root 20 0 94364 24656 3236 S 1.3 2.6 0:00.32 /usr/bin/python /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/landscape-client.p

If it is of any use, all my instances are virtual, and I'd like to keep them really small.

Changed in landscape-client:
importance: Undecided → High
Jan Brodda (janxb) wrote :

I am still able to reproduce this issue with the following versions:

landscape-client: 16.03-0ubuntu2.16.04.3
landscape-server: 17.03.4-0ubuntu1
OS version: Ubuntu 16.04.4 LTS

Attached you can find a graph of roughly 24 hours. The container runs just the landscape client, otherwise it's the default ubuntu:lts image from LXD with only the snapd package removed.

Memory usage started with 180MB and raised up to around 480MB after 20 hours. See the attached image for details.

Simon Poirier (simpoir) on 2018-04-12
Changed in landscape-client:
assignee: nobody → Simon Poirier (simpoir)
status: Confirmed → In Progress
Jeff Zamanski (jzamanski) wrote :

+1 for a fix here ... been forced to basically abandon landscape due to this bug .... glad to provide any info necessary for troubleshooting

Simon Poirier (simpoir) on 2018-06-19
Changed in landscape-client:
status: In Progress → Fix Committed
flindeberg (flindeberg) wrote :

When will the fix be available and for which distros?

Jan Brodda (janxb) wrote :

Is there an ETA for this fix? I really want to start using landscape again..

GhostSeven (blakedrayson) wrote :

Can we please get an ETA for this, it is getting a little silly now, how you can possibly have your own management client have a bug fixed of this magnitude and it not be released is beyond me.

David Britton (davidpbritton) wrote :

Since people continue to ask; We don't give ETAs on bug reports. Please mark this bug as affecting you if it is.

See: https://help.ubuntu.com/community/ReportingBugs, Etiquite Section, #9

Contribution in the form of testing help is always welcome (you can find the fix in the attached link in comment 12).

Thanks!

Eric Desrochers (slashd) on 2019-02-12
Changed in landscape-client (Ubuntu Cosmic):
assignee: nobody → Simon Poirier (simpoir)
Changed in landscape-client (Ubuntu Bionic):
assignee: nobody → Simon Poirier (simpoir)
Changed in landscape-client (Ubuntu Xenial):
assignee: nobody → Simon Poirier (simpoir)
Changed in landscape-client (Ubuntu Trusty):
assignee: nobody → Simon Poirier (simpoir)
importance: Undecided → Medium
Changed in landscape-client (Ubuntu Xenial):
importance: Undecided → Medium
Changed in landscape-client (Ubuntu Bionic):
importance: Undecided → Medium
Changed in landscape-client (Ubuntu Cosmic):
importance: Undecided → Medium
status: New → In Progress
Changed in landscape-client (Ubuntu Bionic):
status: New → In Progress
Changed in landscape-client (Ubuntu Xenial):
status: New → In Progress
Changed in landscape-client (Ubuntu Trusty):
status: New → In Progress
Changed in landscape-client (Ubuntu):
assignee: nobody → Simon Poirier (simpoir)
status: Confirmed → In Progress
Changed in landscape-client (Ubuntu Trusty):
importance: Medium → High
Changed in landscape-client (Ubuntu Xenial):
importance: Medium → High
Changed in landscape-client (Ubuntu Bionic):
importance: Medium → High
Changed in landscape-client (Ubuntu Cosmic):
importance: Medium → High
Simon Poirier (simpoir) on 2019-02-12
no longer affects: landscape-client (Ubuntu Trusty)
Simon Poirier (simpoir) on 2019-02-12
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 18.01-0ubuntu7

---------------
landscape-client (18.01-0ubuntu7) disco; urgency=medium

  * debian/patches/mem-usage-1685885.patch: Reduce memory usage by
    unused twisted logs. (LP: #1685885)

 -- Simon Poirier <email address hidden> Fri, 08 Feb 2019 12:02:17 -0500

Changed in landscape-client (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers