Extreme RAM and SWAP usage

[Thomas Ward]

[Impact]

 * Memory usage of landscape processes abnormally grows, over time,
   even for processes which are generally idle and have no reason to
   grow.

 * Most of that memory leak is due to uninitialized python-twisted
   logging facilities buffering logs and their context. This fix
   Initializes logging with a null handler to have those unused log
   entries discarded.

 * Upstream twisted has mitigated part of the issue in recent releases
   by reducing the default buffer size. Because of that, the impact
   of this bug is greater on xenial.

[Test Case]

 * register a landscape client and log the memory growth over a day

 * the effect is most noticeable on /usr/bin/landscape-client and less on
   the other processes

 * the memory difference on recent releases (bionic and up) is
   expected to be smaller (~10mb on bionic instead of 200mb on xenial)

[Regression Potential]

 * The fix initializes twisted logging, which is unused by landscape
   so it is unlikely to have an impact.

 * The fix comes from recommendations on the upstream bug:
   https://twistedmatrix.com/trac/ticket/8164

[Original description]

This has been noticed on several systems of mine, all 16.04.

Landscape Client takes up, over time, an *extreme* amount of system resources.

On a 1024MB RAM system, within 3 days to a month, Landscape Client and it's python scripts (possibly cached data?) are consuming massive quantities of memory - in the realm of 512MB RAM and over 400MB of the Swap on the system.

This is repeatably shown in other systems of mine as well. Is there a memory leak or something in landscape-client that would contribute to this issue?

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: landscape-client 16.03-0ubuntu2.16.04.1
ProcVersionSignature: Ubuntu 4.4.0-72.93-generic 4.4.49
Uname: Linux 4.4.0-72-generic x86_64
NonfreeKernelModules: talpa_vfshook talpa_pedconnector talpa_vcdevice talpa_core talpa_linux talpa_syscallhook talpa_pedevice
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
Date: Mon Apr 24 14:42:37 2017
InstallationDate: Installed on 2017-04-02 (21 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: landscape-client
UpgradeStatus: No upgrade log present (probably fresh install)
---
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
DistroRelease: Ubuntu 16.04
InstallationDate: Installed on 2017-04-02 (21 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
Package: landscape-client 16.03-0ubuntu2.16.04.1
PackageArchitecture: amd64
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 4.4.0-72.93-generic 4.4.49
Tags: third-party-packages xenial
Uname: Linux 4.4.0-72-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True

[Thomas Ward (teward)]

apport information

[Thomas Ward (teward)]

apport information

[Thomas Ward (teward)]

Note that the kernel modules listed in the bug may be out of date - I don't have those modules listed, but i hadn't rebooted since they were installed, due to the antivirus in place on this server where I reported from (it's a mail server, so an antivirus engine with amavisd was necessary; note that the usage values for this bug report - for landscape client consuming lots of RAM - had the kernel modules 'off' and disabled with its standard software that correspond with the antivirus engine.)

[Thomas Ward (teward)]

I discussed this briefly in #ubuntu-hardened and #ubuntu-server and confirmed that the code hasn't been 'reviewed' lately.

I'm setting the importance here to "high" because I just got nipped with this again, and my primary mail server went down because of insanely high swap and RAM usage from Landscape Client. I'm going to test this with a new server to rule out headaches, but if landscape client is going to consume ungodly amounts of RAM, then it should be shelved until the memory leaks are solved.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Captain Skyhawk (captain-skyhawk)]

Also affecting me -- landscape-client using 25% of RAM on a 1GB server.

[flindeberg (flindeberg)]

This is actually so severe that I started to regularly restart my small instances *every week*. The instance below is with an uptime of nine (9) days on a heavily used file server, and as you can see the first none landscape related process is the ssh-session I'm in! Sorted on memory %.

  PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
 1410 root 20 0 302M 229M 2252 S 0.0 25.2 35:39.09 /usr/bin/python /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/landscape-client.p
 1426 root 20 0 205M 117M 3768 S 0.0 12.9 9:08.72 /usr/bin/python /usr/bin/landscape-manager --ignore-sigint --quiet
 1470 landscape 20 0 552M 67320 5136 S 0.0 7.2 2:19.10 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1474 landscape 20 0 552M 67320 5136 S 0.0 7.2 2:18.09 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1423 landscape 20 0 552M 67320 5136 S 0.0 7.2 15:38.76 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1472 landscape 20 0 355M 28884 3780 S 0.0 3.1 0:00.01 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
 1424 landscape 20 0 355M 28884 3780 S 0.0 3.1 16:14.51 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
65097 root 20 0 119M 7772 6516 S 0.0 0.8 0:00.10 sshd: flindeberg [priv]

Same instance after reboot (sorted on memory %):

  PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
  976 mongodb 20 0 959M 62340 31504 S 0.0 6.7 0:00.00 /usr/bin/mongod --unixSocketPrefix=/run/mongodb --config /etc/mongodb.conf
  -- some other db and file server processes --
 1269 landscape 20 0 307M 54100 12616 S 0.0 5.8 0:00.49 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1231 landscape 20 0 307M 54100 12616 S 0.0 5.8 0:01.49 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1271 landscape 20 0 253M 38316 14680 S 0.0 4.1 0:00.01 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
 1232 landscape 20 0 253M 38316 14680 S 0.0 4.1 0:01.16 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
 1233 root 20 0 104M 34472 11840 S 0.0 3.7 0:00.86 /usr/bin/python /usr/bin/landscape-manager --ignore-sigint --quiet
 1224 root 20 0 94364 24656 3236 S 1.3 2.6 0:00.32 /usr/bin/python /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/landscape-client.p

If it is of any use, all my instances are virtual, and I'd like to keep them really small.

[Jan Brodda (janxb)]

I am still able to reproduce this issue with the following versions:

landscape-client: 16.03-0ubuntu2.16.04.3
landscape-server: 17.03.4-0ubuntu1
OS version: Ubuntu 16.04.4 LTS

Attached you can find a graph of roughly 24 hours. The container runs just the landscape client, otherwise it's the default ubuntu:lts image from LXD with only the snapd package removed.

Memory usage started with 180MB and raised up to around 480MB after 20 hours. See the attached image for details.

[Jeff Zamanski (jzamanski)]

+1 for a fix here ... been forced to basically abandon landscape due to this bug .... glad to provide any info necessary for troubleshooting

[flindeberg (flindeberg)]

When will the fix be available and for which distros?

[Mathew Hodson (mhodson)]

https://github.com/CanonicalLtd/landscape-client/commit/7f98306e3159e956798b70d2b5fc8c0773373fdd

[Jan Brodda (janxb)]

Is there an ETA for this fix? I really want to start using landscape again..

[GhostSeven (blakedrayson)]

Can we please get an ETA for this, it is getting a little silly now, how you can possibly have your own management client have a bug fixed of this magnitude and it not be released is beyond me.

[David Britton (dpb)]

Since people continue to ask; We don't give ETAs on bug reports. Please mark this bug as affecting you if it is.

See: https://help.ubuntu.com/community/ReportingBugs, Etiquite Section, #9

Contribution in the form of testing help is always welcome (you can find the fix in the attached link in comment 12).

Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 18.01-0ubuntu7

---------------
landscape-client (18.01-0ubuntu7) disco; urgency=medium

  * debian/patches/mem-usage-1685885.patch: Reduce memory usage by
    unused twisted logs. (LP: #1685885)

 -- Simon Poirier <email address hidden> Fri, 08 Feb 2019 12:02:17 -0500

[Brian Murray (brian-murray)]

Hello Thomas, or anyone else affected,

Accepted landscape-client into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/18.01-0ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Thomas, or anyone else affected,

Accepted landscape-client into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/18.01-0ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Thomas, or anyone else affected,

Accepted landscape-client into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/16.03-0ubuntu2.16.04.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Simon Poirier (simpoir)]

I tested this on xenial, bionic and cosmic by launching 2 identical lxd containers of each, one of which using the landscape-client/common from -proposed packages. Clients were registered in against the same account and left idle for a few hours before taking a reading:

  ps o rss,args `pgrep landscape-cli`

Here are the results which, as per test case, show the most noticeable improvement on xenial.

xenial-updates
  RSS COMMAND
254704 /usr/bin/python /usr/bin/landscape-client --daemon --pid-file /var/run/

xenial-proposed
  RSS COMMAND
46040 /usr/bin/python /usr/bin/landscape-client --daemon --pid-file /var/run/l

bionic-updates
  RSS COMMAND
33744 /usr/bin/python3 /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/

bionic-proposed
  RSS COMMAND
30960 /usr/bin/python3 /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/

cosmic-updates
  RSS COMMAND
33660 /usr/bin/python3 /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/la

cosmic-proposed
  RSS COMMAND
30912 /usr/bin/python3 /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/la

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for landscape-client has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 18.01-0ubuntu4.2

---------------
landscape-client (18.01-0ubuntu4.2) cosmic; urgency=medium

  * debian/patches/mem-usage-1685885.patch: Reduce memory usage by
    unused twisted logs. (LP: #1685885)

 -- Simon Poirier <email address hidden> Fri, 08 Feb 2019 12:02:17 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 18.01-0ubuntu3.3

---------------
landscape-client (18.01-0ubuntu3.3) bionic; urgency=medium

  * debian/patches/mem-usage-1685885.patch: Reduce memory usage by
    unused twisted logs. (LP: #1685885)

 -- Simon Poirier <email address hidden> Fri, 08 Feb 2019 12:02:17 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 16.03-0ubuntu2.16.04.6

---------------
landscape-client (16.03-0ubuntu2.16.04.6) xenial; urgency=medium

  * debian/patches/mem-usage-1685885.patch: Reduce memory usage by
    unused twisted logs. (LP: #1685885)

 -- Simon Poirier <email address hidden> Fri, 08 Feb 2019 12:02:17 -0500