landscape-client consumes too much memory and CPU time

[Andreas Hasenack]

A customer complained that, compared to MySQL and Apache, landscape-client consumes too much resources, particularly memory. Here is the "top" output the customer sent:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 6253 landscap 20 0 33780 12m 3780 S 0 1.3 380:10.31 landscape-monit
 6246 root 20 0 13960 3648 1312 S 1 0.4 272:38.24 landscape-clien
 6251 landscap 20 0 124m 53m 4408 S 0 5.2 266:37.10 landscape-broke
 6256 root 20 0 30796 6816 3700 S 0 0.7 171:29.33 landscape-manag
 3403 root 15 -5 0 0 0 S 0 0.0 102:56.88 kjournald
 5705 messageb 20 0 2968 1536 912 S 0 0.1 69:50.66 dbus-daemon
 5964 mysql 20 0 124m 5056 4064 S 0 0.5 53:37.92 mysqld
 6068 snmp 20 0 8584 2452 1684 S 0 0.2 31:40.23 snmpd
 6137 haldaemo 20 0 30588 22m 2580 S 0 2.3 19:32.14 hald
 6140 root 20 0 8784 2768 1388 S 0 0.3 6:25.22 console-kit-dae
 5665 syslog 20 0 2036 796 648 S 0 0.1 3:54.30 syslogd
 1751 root 15 -5 0 0 0 S 0 0.0 3:49.43 scsi_eh_1
 6231 root 20 0 3304 924 844 S 0 0.1 3:29.79 hald-addon-stor
 5327 ntp 20 0 4124 868 756 S 0 0.1 3:24.69 ntpd
    6 root RT -5 0 0 0 S 0 0.0 2:58.15 migration/1
    3 root RT -5 0 0 0 S 0 0.0 2:55.40 migration/0
 9678 root 20 0 29156 14m 5868 S 0 1.5 2:27.08 apache2

This is on a hardy system.

[Jamu Kakar (jkakar)]

I wonder what memory use is like 4h into operation compared to after
many days of operation. In other words, could we improve the
situation by restarting the client every 6h. One thing that will
affect the broker is message size.

[Andreas Hasenack (ahasenack)]

I'm also wondering if this is a 64bit system.

I created some custom graphs in staging to track the process VSZ size of each one of the landscape-* processes, and I'm also switching our karmic test machine from 32bits to 64bits.

[Peter Matulis (petermatulis)]

The described machine had an uptime of 78 days.

[Peter Matulis (petermatulis)]

The total CPU time consumed by the 4 landscape processes is approximately 1089 minutes.

[Peter Matulis (petermatulis)]

No, this is not a 64-bit machine.

-------- lsb_release -d --------

        Ubuntu 8.04.3 LTS

-------- uname -a --------

Linux ????? 2.6.24-24-server #1 SMP Fri Jul 24 23:27:55 UTC 2009 i686 GNU/Linux

[Paul Elliott (omahn)]

Our site is also experiencing very high RAM usage by landscape, this is typical on our servers: (from top, sorted by swap usage)

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
 4660 landscap 20 0 333m 108m 5464 S 0.0 10.8 96:49.59 224m landscape-broke
 4663 landscap 20 0 181m 18m 5232 S 0.0 1.9 106:28.65 163m landscape-monit
 4665 root 20 0 176m 15m 5064 S 0.0 1.6 33:55.16 161m landscape-manag
 4658 root 20 0 122m 10m 1344 S 0.0 1.1 56:36.47 112m landscape-clien

As a workaround we plan to disable landscape on our servers but a long term solution of reduced memory usage by the client would be preferable.

[Paul Elliott (omahn)]

Sorry, forgot to include that these landscape processes have been running since May 18, uptime on the box is 97 days. The machine is running 8.04.4 and is a minimal VM, hence the low amount of RAM and high swap numbers.

[Andreas Hasenack (ahasenack)]

You can try this command to reduce the memory footprint of the package reporter, which landscape-client calls every hour. It's a forked process, so I don't see how it could help improve the situation of the long-running ones, but it's worth a shot:

sudo smart config --set psyco=0

Also, what version of landscape-client is that? At some version we did reduce the number of interrupts it was generating. That has no implications on memory usage, but on CPU and overall power.

[Sorin Sbarnea (ssbarnea)]

I think there is a serious issue with the CPU usage. On my system it has a 3.4 execution hours in 3 days, this translates to ~5% of cpu usage all the time.

[Andreas Hasenack (ahasenack)]

Sorin, what's the version of landscape-client you have installed?

[Sorin Sbarnea (ssbarnea)]

Version: 1.5.5.1-0ubuntu0.10.04.0

[Sorin Sbarnea (ssbarnea)]

I just upgraded my machines to 10.10 and reenabled teh landscape-client, I will be back with results in 2-3 days.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.